[00:00:23] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 4.82, 6.70, 7.58 [00:00:49] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.54, 4.19, 3.88 [00:01:47] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 26.14, 22.91, 22.05 [00:02:23] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 10.05, 8.19, 8.03 [00:03:32] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.99, 7.92, 7.86 [00:03:32] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2607:5300:201:3100::929a/cpweb [00:03:33] PROBLEM - mw101 Puppet on mw101 is CRITICAL: CRITICAL: Puppet has 163 failures. Last run 2 minutes ago with 163 failures. Failed resources (up to 3 shown): File[/etc/apt/trusted.gpg.d/puppetlabs.gpg],File[/usr/local/bin/puppet-enabled],File[authority certificates],File[/etc/apt/apt.conf.d/50unattended-upgrades] [00:03:46] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.52, 22.91, 22.15 [00:04:49] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.32, 3.84, 3.82 [00:05:14] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:05:27] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.00, 6.70, 7.88 [00:05:44] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:06:03] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:06:07] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:06:16] [02MirahezeMagic] 07dmehus synchronize pull request 03#311: Re-add global interface messages for default user groups - 13https://git.io/J9GqS [00:06:18] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:06:46] CosmicAlpha: you can probably merge now. I see Agent did the PR for the Special:MyLanguage links [00:07:11] * dmehus notes that his first time clicking the merge button on GitHub, although was only to my personal repository :P [00:07:32] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 51.195.220.68/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [00:07:57] [02MirahezeMagic] 07Universal-Omega reviewed pull request 03#311 commit - 13https://git.io/J9Zft [00:08:20] dmehus: ^ [00:09:22] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 9.682 second response time [00:09:27] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [00:09:44] [02MirahezeMagic] 07Universal-Omega reviewed pull request 03#311 commit - 13https://git.io/J9ZfG [00:09:44] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.38, 6.76, 7.74 [00:09:52] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:09:58] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23541 bytes in 6.811 second response time [00:10:04] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:10:07] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:10:08] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:10:16] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 6.013 second response time [00:10:20] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 5.654 second response time [00:10:27] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:10:29] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 5.216 second response time [00:11:27] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.78, 7.20, 7.66 [00:11:31] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.41, 7.40, 7.50 [00:11:40] [02MirahezeMagic] 07dmehus synchronize pull request 03#311: Re-add global interface messages for default user groups - 13https://git.io/J9GqS [00:11:44] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.70, 7.99, 8.08 [00:12:01] [02MirahezeMagic] 07Universal-Omega reviewed pull request 03#311 commit - 13https://git.io/J9Zfj [00:12:02] [02MirahezeMagic] 07Universal-Omega reviewed pull request 03#311 commit - 13https://git.io/J9ZJe [00:12:04] [02MirahezeMagic] 07Universal-Omega reviewed pull request 03#311 commit - 13https://git.io/J9ZJv [00:12:05] [02MirahezeMagic] 07Universal-Omega reviewed pull request 03#311 commit - 13https://git.io/J9ZJf [00:12:08] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 9.638 second response time [00:12:30] [02MirahezeMagic] 07dmehus synchronize pull request 03#311: Re-add global interface messages for default user groups - 13https://git.io/J9GqS [00:12:32] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 6.676 second response time [00:13:20] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 198.244.148.90/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [00:13:22] [02MirahezeMagic] 07dmehus reviewed pull request 03#311 commit - 13https://git.io/J9ZJk [00:13:29] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.31, 7.07, 7.54 [00:13:35] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.95, 7.10, 7.38 [00:13:42] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:13:55] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 0.350 second response time [00:14:04] [02MirahezeMagic] 07Universal-Omega reviewed pull request 03#311 commit - 13https://git.io/J9ZJq [00:14:13] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 4.037 second response time [00:14:16] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23538 bytes in 6.090 second response time [00:14:21] [02MirahezeMagic] 07dmehus reviewed pull request 03#311 commit - 13https://git.io/J9ZJm [00:14:24] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:14:44] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:14:46] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:14:49] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.33, 3.43, 3.52 [00:14:55] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:14:59] [02MirahezeMagic] 07Universal-Omega reviewed pull request 03#311 commit - 13https://git.io/J9ZJY [00:15:31] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.25, 7.49, 7.49 [00:15:33] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.20, 22.49, 22.18 [00:15:40] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:15:47] [02MirahezeMagic] 07dmehus synchronize pull request 03#311: Re-add global interface messages for default user groups - 13https://git.io/J9GqS [00:15:51] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 9.034 second response time [00:16:49] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.64, 3.33, 3.46 [00:16:50] miraheze/MirahezeMagic - dmehus the build passed. [00:17:27] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.60, 8.21, 7.87 [00:17:31] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.53, 22.82, 22.38 [00:17:31] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.71, 7.22, 7.40 [00:18:49] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.31, 3.57, 3.52 [00:19:27] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.26, 7.91, 7.82 [00:20:09] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:20:27] miraheze/MirahezeMagic - dmehus the build passed. [00:20:42] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23540 bytes in 0.666 second response time [00:20:59] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 3.401 second response time [00:21:07] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 4.905 second response time [00:21:14] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 6.613 second response time [00:21:29] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 26.75, 24.35, 23.09 [00:22:26] miraheze/MirahezeMagic - dmehus the build passed. [00:23:19] [02MirahezeMagic] 07dmehus reviewed pull request 03#311 commit - 13https://git.io/J9ZJ8 [00:23:25] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [00:23:40] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.66, 5.21, 4.45 [00:23:48] PROBLEM - cp21 Stunnel Http for mw8 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:24:07] PROBLEM - mw8 MediaWiki Rendering on mw8 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:24:49] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.51, 3.85, 3.73 [00:25:06] CosmicAlpha, which files do I need to edit, in /miraheze and /overrides? [00:25:24] I'm sooo confused because you've had me go back and forth so many times lol [00:25:40] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 4.76, 5.06, 4.48 [00:25:42] RECOVERY - cp21 Stunnel Http for mw8 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14560 bytes in 0.039 second response time [00:25:56] miraheze/MirahezeMagic - dmehus the build passed. [00:26:02] RECOVERY - mw8 MediaWiki Rendering on mw8 is OK: HTTP OK: HTTP/1.1 200 OK - 23528 bytes in 0.442 second response time [00:26:49] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.86, 4.14, 3.84 [00:27:11] RECOVERY - test3 APT on test3 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [00:27:16] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:27:19] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [00:27:25] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::1b80/cpweb [00:27:26] Can I speak here? [00:27:26] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:27:35] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:27:36] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:27:44] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 5.45, 7.28, 7.98 [00:27:56] dmehus: the files the reviews where for, it will appear on top of the review [00:28:07] YellowFrogger: Yes, what do you need? [00:28:35] I replied to you on phabricator, by the way [00:28:57] CosmicAlpha, ok [00:29:22] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23540 bytes in 8.292 second response time [00:29:41] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 7.622 second response time [00:29:41] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 8.467 second response time [00:30:19] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.013 second response time [00:31:27] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.78, 7.69, 7.59 [00:31:30] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.79, 23.58, 23.58 [00:31:31] RECOVERY - mw101 Puppet on mw101 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [00:31:35] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 3.639 second response time [00:31:44] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.09, 7.51, 7.90 [00:32:23] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/skins/Nimbus-8669188 [+0/-0/±1] 13https://git.io/J9ZJD [00:32:25] [02miraheze/mediawiki] 07dependabot[bot] 0384b79ac - Bump skins/Nimbus from `7b47d15` to `8669188` [00:32:26] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/skins/Nimbus-8669188 - 13https://git.io/vbL5b [00:32:27] RECOVERY - mwtask1 APT on mwtask1 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [00:32:28] [02mediawiki] 07dependabot[bot] opened pull request 03#4865: Bump skins/Nimbus from `7b47d15` to `8669188` - 13https://git.io/J9ZJy [00:32:29] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/Babel-5b01abf [+0/-0/±1] 13https://git.io/J9ZJS [00:32:31] [02miraheze/mediawiki] 07dependabot[bot] 03d8d8d14 - Bump extensions/Babel from `9e55da4` to `5b01abf` [00:32:32] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/Babel-5b01abf - 13https://git.io/vbL5b [00:32:34] [02mediawiki] 07dependabot[bot] opened pull request 03#4866: Bump extensions/Babel from `9e55da4` to `5b01abf` - 13https://git.io/J9ZJ9 [00:32:34] RECOVERY - mw8 APT on mw8 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [00:32:35] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/Popups-669cc18 [+0/-0/±1] 13https://git.io/J9ZJH [00:32:37] [02miraheze/mediawiki] 07dependabot[bot] 0399e9bfe - Bump extensions/Popups from `dc1a397` to `669cc18` [00:32:38] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/Popups-669cc18 - 13https://git.io/vbL5b [00:32:40] [02mediawiki] 07dependabot[bot] opened pull request 03#4867: Bump extensions/Popups from `dc1a397` to `669cc18` - 13https://git.io/J9ZJQ [00:32:41] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/MatomoAnalytics-ba5a4c8 [+0/-0/±1] 13https://git.io/J9ZJ7 [00:32:43] [02miraheze/mediawiki] 07dependabot[bot] 036b68ce7 - Bump extensions/MatomoAnalytics from `0424557` to `ba5a4c8` [00:32:44] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/MatomoAnalytics-ba5a4c8 - 13https://git.io/vbL5b [00:32:46] [02mediawiki] 07dependabot[bot] opened pull request 03#4868: Bump extensions/MatomoAnalytics from `0424557` to `ba5a4c8` - 13https://git.io/J9ZJ5 [00:32:47] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/CreatePageUw-84db84a [+0/-0/±1] 13https://git.io/J9ZJd [00:32:49] [02miraheze/mediawiki] 07dependabot[bot] 038058690 - Bump extensions/CreatePageUw from `074295c` to `84db84a` [00:32:50] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/CreatePageUw-84db84a - 13https://git.io/vbL5b [00:32:52] [02mediawiki] 07dependabot[bot] opened pull request 03#4869: Bump extensions/CreatePageUw from `074295c` to `84db84a` - 13https://git.io/J9ZJF [00:32:54] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/Video-1713c47 [+0/-0/±1] 13https://git.io/J9ZJb [00:32:55] [02miraheze/mediawiki] 07dependabot[bot] 03155f8e5 - Bump extensions/Video from `c5e1d1a` to `1713c47` [00:32:56] [02mediawiki] 07dependabot[bot] opened pull request 03#4870: Bump extensions/Video from `c5e1d1a` to `1713c47` - 13https://git.io/J9ZJN [00:32:58] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/Video-1713c47 - 13https://git.io/vbL5b [00:32:59] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/TimedMediaHandler-16976ec [+0/-0/±1] 13https://git.io/J9ZJA [00:33:01] [02miraheze/mediawiki] 07dependabot[bot] 03f1ae007 - Bump extensions/TimedMediaHandler from `ad5c67f` to `16976ec` [00:33:02] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/TimedMediaHandler-16976ec - 13https://git.io/vbL5b [00:33:04] [02mediawiki] 07dependabot[bot] opened pull request 03#4871: Bump extensions/TimedMediaHandler from `ad5c67f` to `16976ec` - 13https://git.io/J9ZJp [00:33:06] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/skins/Evelution-833020a [+0/-0/±1] 13https://git.io/J9ZJh [00:33:07] [02miraheze/mediawiki] 07dependabot[bot] 038d72094 - Bump skins/Evelution from `3d61d87` to `833020a` [00:33:09] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/skins/Evelution-833020a - 13https://git.io/vbL5b [00:33:10] [02mediawiki] 07dependabot[bot] opened pull request 03#4872: Bump skins/Evelution from `3d61d87` to `833020a` - 13https://git.io/J9ZJj [00:33:11] [02mediawiki] 07github-actions[bot] labeled pull request 03#4866: Bump extensions/Babel from `9e55da4` to `5b01abf` - 13https://git.io/J9ZJ9 [00:33:13] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/StandardDialogs-f601fed [+0/-0/±1] 13https://git.io/J9ZUe [00:33:15] [02miraheze/mediawiki] 07dependabot[bot] 03678c572 - Bump extensions/StandardDialogs from `c116646` to `f601fed` [00:33:16] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/StandardDialogs-f601fed - 13https://git.io/vbL5b [00:33:18] [02mediawiki] 07dependabot[bot] opened pull request 03#4873: Bump extensions/StandardDialogs from `c116646` to `f601fed` - 13https://git.io/J9ZUv [00:33:19] [02mediawiki] 07github-actions[bot] labeled pull request 03#4865: Bump skins/Nimbus from `7b47d15` to `8669188` - 13https://git.io/J9ZJy [00:33:19] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 2001:41d0:801:2000::1b80/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::5ebc/cpweb [00:33:20] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/GlobalUsage-87146f0 [+0/-0/±1] 13https://git.io/J9ZUf [00:33:22] [02miraheze/mediawiki] 07dependabot[bot] 03b3daf98 - Bump extensions/GlobalUsage from `7071682` to `87146f0` [00:33:23] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/GlobalUsage-87146f0 - 13https://git.io/vbL5b [00:33:25] [02mediawiki] 07dependabot[bot] opened pull request 03#4874: Bump extensions/GlobalUsage from `7071682` to `87146f0` - 13https://git.io/J9ZUJ [00:33:26] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/skins/Medik-833353a [+0/-0/±1] 13https://git.io/J9ZUU [00:33:28] [02miraheze/mediawiki] 07dependabot[bot] 03c91d7cb - Bump skins/Medik from `b491fc8` to `833353a` [00:33:29] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/skins/Medik-833353a - 13https://git.io/vbL5b [00:33:31] [02mediawiki] 07dependabot[bot] opened pull request 03#4875: Bump skins/Medik from `b491fc8` to `833353a` - 13https://git.io/J9ZUT [00:33:31] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.20, 7.44, 7.20 [00:33:33] [02mediawiki] 07github-actions[bot] labeled pull request 03#4869: Bump extensions/CreatePageUw from `074295c` to `84db84a` - 13https://git.io/J9ZJF [00:33:34] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/Graph-fc4eb2f [+0/-0/±1] 13https://git.io/J9ZUk [00:33:36] [02miraheze/mediawiki] 07dependabot[bot] 0322515d9 - Bump extensions/Graph from `676ba72` to `fc4eb2f` [00:33:37] [02mediawiki] 07github-actions[bot] labeled pull request 03#4869: Bump extensions/CreatePageUw from `074295c` to `84db84a` - 13https://git.io/J9ZJF [00:33:39] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/Graph-fc4eb2f - 13https://git.io/vbL5b [00:33:40] [02mediawiki] 07dependabot[bot] opened pull request 03#4876: Bump extensions/Graph from `676ba72` to `fc4eb2f` - 13https://git.io/J9ZUI [00:33:42] [02mediawiki] 07github-actions[bot] labeled pull request 03#4867: Bump extensions/Popups from `dc1a397` to `669cc18` - 13https://git.io/J9ZJQ [00:33:43] [02mediawiki] 07github-actions[bot] labeled pull request 03#4868: Bump extensions/MatomoAnalytics from `0424557` to `ba5a4c8` - 13https://git.io/J9ZJ5 [00:33:44] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.16, 7.25, 7.75 [00:33:44] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/UniversalLanguageSelector-d2b84d7 [+0/-0/±1] 13https://git.io/J9ZUL [00:33:46] [02miraheze/mediawiki] 07dependabot[bot] 032732ddb - Bump extensions/UniversalLanguageSelector from `f33d7ce` to `d2b84d7` [00:33:47] [02mediawiki] 07dependabot[bot] opened pull request 03#4877: Bump extensions/UniversalLanguageSelector from `f33d7ce` to `d2b84d7` - 13https://git.io/J9ZUt [00:33:49] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/UniversalLanguageSelector-d2b84d7 - 13https://git.io/vbL5b [00:33:51] [02mediawiki] 07github-actions[bot] labeled pull request 03#4871: Bump extensions/TimedMediaHandler from `ad5c67f` to `16976ec` - 13https://git.io/J9ZJp [00:33:52] [02mediawiki] 07github-actions[bot] labeled pull request 03#4870: Bump extensions/Video from `c5e1d1a` to `1713c47` - 13https://git.io/J9ZJN [00:33:54] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/MobileFrontend-f83e725 [+0/-0/±1] 13https://git.io/J9ZUq [00:33:55] [02miraheze/mediawiki] 07dependabot[bot] 036c1e32c - Bump extensions/MobileFrontend from `8f64d85` to `f83e725` [00:33:57] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/MobileFrontend-f83e725 - 13https://git.io/vbL5b [00:33:58] [02mediawiki] 07dependabot[bot] opened pull request 03#4878: Bump extensions/MobileFrontend from `8f64d85` to `f83e725` - 13https://git.io/J9ZUm [00:33:59] [02mediawiki] 07github-actions[bot] labeled pull request 03#4873: Bump extensions/StandardDialogs from `c116646` to `f601fed` - 13https://git.io/J9ZUv [00:34:01] [02mediawiki] 07github-actions[bot] labeled pull request 03#4872: Bump skins/Evelution from `3d61d87` to `833020a` - 13https://git.io/J9ZJj [00:34:02] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/BetaFeatures-2820d7f [+0/-0/±1] 13https://git.io/J9ZUO [00:34:04] [02miraheze/mediawiki] 07dependabot[bot] 035437e6c - Bump extensions/BetaFeatures from `a8d6bdb` to `2820d7f` [00:34:05] [02mediawiki] 07dependabot[bot] opened pull request 03#4879: Bump extensions/BetaFeatures from `a8d6bdb` to `2820d7f` - 13https://git.io/J9ZU3 [00:34:07] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/BetaFeatures-2820d7f - 13https://git.io/vbL5b [00:34:09] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/skins/Metrolook-eed515c [+0/-0/±1] 13https://git.io/J9ZUs [00:34:10] [02miraheze/mediawiki] 07dependabot[bot] 03b2996c6 - Bump skins/Metrolook from `577b947` to `eed515c` [00:34:12] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/skins/Metrolook-eed515c - 13https://git.io/vbL5b [00:34:13] [02mediawiki] 07dependabot[bot] opened pull request 03#4880: Bump skins/Metrolook from `577b947` to `eed515c` - 13https://git.io/J9ZUG [00:34:15] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/Renameuser-d112548 [+0/-0/±1] 13https://git.io/J9ZUZ [00:34:16] [02miraheze/mediawiki] 07dependabot[bot] 0344f345c - Bump extensions/Renameuser from `cbc7547` to `d112548` [00:34:18] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/Renameuser-d112548 - 13https://git.io/vbL5b [00:34:19] [02mediawiki] 07dependabot[bot] opened pull request 03#4881: Bump extensions/Renameuser from `cbc7547` to `d112548` - 13https://git.io/J9ZUn [00:34:20] [02mediawiki] 07github-actions[bot] labeled pull request 03#4874: Bump extensions/GlobalUsage from `7071682` to `87146f0` - 13https://git.io/J9ZUJ [00:34:22] [02mediawiki] 07github-actions[bot] labeled pull request 03#4875: Bump skins/Medik from `b491fc8` to `833353a` - 13https://git.io/J9ZUT [00:34:24] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/Kartographer-3ba56a9 [+0/-0/±1] 13https://git.io/J9ZUC [00:34:25] [02miraheze/mediawiki] 07dependabot[bot] 039b4733f - Bump extensions/Kartographer from `819f44d` to `3ba56a9` [00:34:27] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/Kartographer-3ba56a9 - 13https://git.io/vbL5b [00:34:28] [02mediawiki] 07dependabot[bot] opened pull request 03#4882: Bump extensions/Kartographer from `819f44d` to `3ba56a9` - 13https://git.io/J9ZUW [00:34:30] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/skins/Vector-e826214 [+0/-0/±1] 13https://git.io/J9ZUl [00:34:31] [02miraheze/mediawiki] 07dependabot[bot] 0390a7ee6 - Bump skins/Vector from `f7e05a9` to `e826214` [00:34:32] [02mediawiki] 07dependabot[bot] opened pull request 03#4883: Bump skins/Vector from `f7e05a9` to `e826214` - 13https://git.io/J9ZU8 [00:34:34] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/skins/Vector-e826214 - 13https://git.io/vbL5b [00:34:35] [02mediawiki] 07github-actions[bot] labeled pull request 03#4876: Bump extensions/Graph from `676ba72` to `fc4eb2f` - 13https://git.io/J9ZUI [00:34:37] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/vendor-56c2e8a [+0/-0/±1] 13https://git.io/J9ZU4 [00:34:39] [02miraheze/mediawiki] 07dependabot[bot] 03a9ae8e7 - Bump vendor from `65b8217` to `56c2e8a` [00:34:40] [02mediawiki] 07github-actions[bot] labeled pull request 03#4877: Bump extensions/UniversalLanguageSelector from `f33d7ce` to `d2b84d7` - 13https://git.io/J9ZUt [00:34:42] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/vendor-56c2e8a - 13https://git.io/vbL5b [00:34:43] [02mediawiki] 07github-actions[bot] labeled pull request 03#4877: Bump extensions/UniversalLanguageSelector from `f33d7ce` to `d2b84d7` - 13https://git.io/J9ZUt [00:34:45] [02mediawiki] 07dependabot[bot] opened pull request 03#4884: Bump vendor from `65b8217` to `56c2e8a` - 13https://git.io/J9ZUB [00:34:46] [02mediawiki] 07github-actions[bot] labeled pull request 03#4877: Bump extensions/UniversalLanguageSelector from `f33d7ce` to `d2b84d7` - 13https://git.io/J9ZUt [00:34:48] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/TwoColConflict-eff532d [+0/-0/±1] 13https://git.io/J9ZUR [00:34:49] [02miraheze/mediawiki] 07dependabot[bot] 0310ccdf7 - Bump extensions/TwoColConflict from `783002e` to `eff532d` [00:34:51] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/TwoColConflict-eff532d - 13https://git.io/vbL5b [00:34:52] [02mediawiki] 07dependabot[bot] opened pull request 03#4885: Bump extensions/TwoColConflict from `783002e` to `eff532d` - 13https://git.io/J9ZU0 [00:34:53] [02mediawiki] 07github-actions[bot] labeled pull request 03#4881: Bump extensions/Renameuser from `cbc7547` to `d112548` - 13https://git.io/J9ZUn [00:34:55] [02mediawiki] 07github-actions[bot] labeled pull request 03#4879: Bump extensions/BetaFeatures from `a8d6bdb` to `2820d7f` - 13https://git.io/J9ZU3 [00:34:57] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/Gadgets-4979fe1 [+0/-0/±1] 13https://git.io/J9ZUu [00:34:58] [02miraheze/mediawiki] 07dependabot[bot] 039237f2c - Bump extensions/Gadgets from `5404436` to `4979fe1` [00:35:00] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/Gadgets-4979fe1 - 13https://git.io/vbL5b [00:35:01] [02mediawiki] 07dependabot[bot] opened pull request 03#4886: Bump extensions/Gadgets from `5404436` to `4979fe1` - 13https://git.io/J9ZUz [00:35:03] [02mediawiki] 07github-actions[bot] labeled pull request 03#4880: Bump skins/Metrolook from `577b947` to `eed515c` - 13https://git.io/J9ZUG [00:35:04] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/skins/Femiwiki-a4f62a4 [+0/-0/±1] 13https://git.io/J9ZUg [00:35:06] [02miraheze/mediawiki] 07dependabot[bot] 030a50dea - Bump skins/Femiwiki from `21a54f2` to `a4f62a4` [00:35:07] [02mediawiki] 07github-actions[bot] labeled pull request 03#4878: Bump extensions/MobileFrontend from `8f64d85` to `f83e725` - 13https://git.io/J9ZUm [00:35:09] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/skins/Femiwiki-a4f62a4 - 13https://git.io/vbL5b [00:35:10] [02mediawiki] 07dependabot[bot] opened pull request 03#4887: Bump skins/Femiwiki from `21a54f2` to `a4f62a4` - 13https://git.io/J9ZU2 [00:35:12] [02mediawiki] 07github-actions[bot] labeled pull request 03#4878: Bump extensions/MobileFrontend from `8f64d85` to `f83e725` - 13https://git.io/J9ZUm [00:35:13] [02mediawiki] 07dependabot[bot] opened pull request 03#4888: Bump extensions/PageTriage from `5b81b50` to `d9440b4` - 13https://git.io/J9ZUV [00:35:15] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/PageTriage-d9440b4 [+0/-0/±1] 13https://git.io/J9ZUw [00:35:16] [02miraheze/mediawiki] 07dependabot[bot] 031635de1 - Bump extensions/PageTriage from `5b81b50` to `d9440b4` [00:35:18] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/PageTriage-d9440b4 - 13https://git.io/vbL5b [00:35:19] [02mediawiki] 07github-actions[bot] labeled pull request 03#4882: Bump extensions/Kartographer from `819f44d` to `3ba56a9` - 13https://git.io/J9ZUW [00:35:19] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [00:35:21] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/Disambiguator-8cb42d8 [+0/-0/±1] 13https://git.io/J9ZUr [00:35:22] [02miraheze/mediawiki] 07dependabot[bot] 03e361ce0 - Bump extensions/Disambiguator from `53e8268` to `8cb42d8` [00:35:24] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/Disambiguator-8cb42d8 - 13https://git.io/vbL5b [00:35:25] [02mediawiki] 07dependabot[bot] opened pull request 03#4889: Bump extensions/Disambiguator from `53e8268` to `8cb42d8` - 13https://git.io/J9ZUo [00:35:27] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/MassMessage-7e532a9 [+0/-0/±1] 13https://git.io/J9ZUK [00:35:28] [02miraheze/mediawiki] 07dependabot[bot] 03987297a - Bump extensions/MassMessage from `bd6de25` to `7e532a9` [00:35:29] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.53, 23.08, 23.28 [00:35:30] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/MassMessage-7e532a9 - 13https://git.io/vbL5b [00:35:31] [02mediawiki] 07dependabot[bot] opened pull request 03#4890: Bump extensions/MassMessage from `bd6de25` to `7e532a9` - 13https://git.io/J9ZU6 [00:35:33] [02mediawiki] 07github-actions[bot] labeled pull request 03#4883: Bump skins/Vector from `f7e05a9` to `e826214` - 13https://git.io/J9ZU8 [00:35:34] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/WikimediaIncubator-33d1ba9 [+0/-0/±1] 13https://git.io/J9ZUi [00:35:36] [02miraheze/mediawiki] 07dependabot[bot] 036357fed - Bump extensions/WikimediaIncubator from `12b9543` to `33d1ba9` [00:35:37] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/WikimediaIncubator-33d1ba9 - 13https://git.io/vbL5b [00:35:39] [02mediawiki] 07dependabot[bot] opened pull request 03#4891: Bump extensions/WikimediaIncubator from `12b9543` to `33d1ba9` - 13https://git.io/J9ZUP [00:35:40] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/MirahezeMagic-00f9ff2 [+0/-0/±1] 13https://git.io/J9ZUX [00:35:42] [02miraheze/mediawiki] 07dependabot[bot] 03521b574 - Bump extensions/MirahezeMagic from `8b32f65` to `00f9ff2` [00:35:43] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/MirahezeMagic-00f9ff2 - 13https://git.io/vbL5b [00:35:45] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.65, 8.02, 7.97 [00:35:45] [02mediawiki] 07dependabot[bot] opened pull request 03#4892: Bump extensions/MirahezeMagic from `8b32f65` to `00f9ff2` - 13https://git.io/J9ZU1 [00:35:46] [02mediawiki] 07github-actions[bot] labeled pull request 03#4886: Bump extensions/Gadgets from `5404436` to `4979fe1` - 13https://git.io/J9ZUz [00:35:48] [02mediawiki] 07github-actions[bot] labeled pull request 03#4885: Bump extensions/TwoColConflict from `783002e` to `eff532d` - 13https://git.io/J9ZU0 [00:35:49] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/Translate-266785e [+0/-0/±1] 13https://git.io/J9ZUM [00:35:51] [02miraheze/mediawiki] 07dependabot[bot] 0342db9fe - Bump extensions/Translate from `941deb5` to `266785e` [00:35:52] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/Translate-266785e - 13https://git.io/vbL5b [00:35:54] [02mediawiki] 07dependabot[bot] opened pull request 03#4893: Bump extensions/Translate from `941deb5` to `266785e` - 13https://git.io/J9ZUD [00:35:55] [02mediawiki] 07github-actions[bot] labeled pull request 03#4887: Bump skins/Femiwiki from `21a54f2` to `a4f62a4` - 13https://git.io/J9ZU2 [00:35:57] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/GlobalWatchlist-d302af9 [+0/-0/±1] 13https://git.io/J9ZUy [00:35:58] [02miraheze/mediawiki] 07dependabot[bot] 03ab0236b - Bump extensions/GlobalWatchlist from `d3cb716` to `d302af9` [00:36:00] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/GlobalWatchlist-d302af9 - 13https://git.io/vbL5b [00:36:01] [02mediawiki] 07dependabot[bot] opened pull request 03#4894: Bump extensions/GlobalWatchlist from `d3cb716` to `d302af9` - 13https://git.io/J9ZUS [00:36:03] [02mediawiki] 07github-actions[bot] labeled pull request 03#4889: Bump extensions/Disambiguator from `53e8268` to `8cb42d8` - 13https://git.io/J9ZUo [00:36:04] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/WikiEditor-f99f482 [+0/-0/±1] 13https://git.io/J9ZUH [00:36:06] [02miraheze/mediawiki] 07dependabot[bot] 03dc16073 - Bump extensions/WikiEditor from `1efc3d8` to `f99f482` [00:36:07] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/WikiEditor-f99f482 - 13https://git.io/vbL5b [00:36:09] [02mediawiki] 07dependabot[bot] opened pull request 03#4895: Bump extensions/WikiEditor from `1efc3d8` to `f99f482` - 13https://git.io/J9ZUQ [00:36:10] [02mediawiki] 07github-actions[bot] labeled pull request 03#4888: Bump extensions/PageTriage from `5b81b50` to `d9440b4` - 13https://git.io/J9ZUV [00:36:12] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/CentralNotice-794de94 [+0/-0/±1] 13https://git.io/J9ZU7 [00:36:13] [02miraheze/mediawiki] 07dependabot[bot] 032430f7a - Bump extensions/CentralNotice from `d3d1795` to `794de94` [00:36:15] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/CentralNotice-794de94 - 13https://git.io/vbL5b [00:36:16] [02mediawiki] 07dependabot[bot] opened pull request 03#4896: Bump extensions/CentralNotice from `d3d1795` to `794de94` - 13https://git.io/J9ZU5 [00:36:18] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/DynamicPageList3-db51e49 [+0/-0/±1] 13https://git.io/J9ZUd [00:36:19] [02miraheze/mediawiki] 07dependabot[bot] 03831280b - Bump extensions/DynamicPageList3 from `b338ff7` to `db51e49` [00:36:21] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/DynamicPageList3-db51e49 - 13https://git.io/vbL5b [00:36:22] [02mediawiki] 07dependabot[bot] opened pull request 03#4897: Bump extensions/DynamicPageList3 from `b338ff7` to `db51e49` - 13https://git.io/J9ZUF [00:36:24] [02mediawiki] 07github-actions[bot] labeled pull request 03#4890: Bump extensions/MassMessage from `bd6de25` to `7e532a9` - 13https://git.io/J9ZU6 [00:36:25] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/ProofreadPage-a4aedd2 [+0/-0/±1] 13https://git.io/J9ZUb [00:36:27] [02miraheze/mediawiki] 07dependabot[bot] 0359d5993 - Bump extensions/ProofreadPage from `6878509` to `a4aedd2` [00:36:28] [02mediawiki] 07dependabot[bot] opened pull request 03#4898: Bump extensions/ProofreadPage from `6878509` to `a4aedd2` - 13https://git.io/J9ZUN [00:36:30] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/ProofreadPage-a4aedd2 - 13https://git.io/vbL5b [00:36:31] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/FlaggedRevs-f3a8ffc [+0/-0/±1] 13https://git.io/J9ZUA [00:36:33] [02miraheze/mediawiki] 07dependabot[bot] 03038eeb5 - Bump extensions/FlaggedRevs from `796798a` to `f3a8ffc` [00:36:34] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/FlaggedRevs-f3a8ffc - 13https://git.io/vbL5b [00:36:36] [02mediawiki] 07dependabot[bot] opened pull request 03#4899: Bump extensions/FlaggedRevs from `796798a` to `f3a8ffc` - 13https://git.io/J9ZUx [00:36:37] [02mediawiki] 07github-actions[bot] labeled pull request 03#4891: Bump extensions/WikimediaIncubator from `12b9543` to `33d1ba9` - 13https://git.io/J9ZUP [00:36:39] [02mediawiki] 07github-actions[bot] labeled pull request 03#4892: Bump extensions/MirahezeMagic from `8b32f65` to `00f9ff2` - 13https://git.io/J9ZU1 [00:36:40] [02mediawiki] 07github-actions[bot] labeled pull request 03#4892: Bump extensions/MirahezeMagic from `8b32f65` to `00f9ff2` - 13https://git.io/J9ZU1 [00:36:42] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/PortableInfobox-c1fd031 [+0/-0/±1] 13https://git.io/J9ZUp [00:36:43] [02miraheze/mediawiki] 07dependabot[bot] 03b1bf9ec - Bump extensions/PortableInfobox from `04cfa29` to `c1fd031` [00:36:45] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/PortableInfobox-c1fd031 - 13https://git.io/vbL5b [00:36:46] [02mediawiki] 07dependabot[bot] opened pull request 03#4900: Bump extensions/PortableInfobox from `04cfa29` to `c1fd031` - 13https://git.io/J9ZUh [00:36:48] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/CheckUser-72187a6 [+0/-0/±1] 13https://git.io/J9ZUj [00:36:49] [02miraheze/mediawiki] 07dependabot[bot] 0317cd56d - Bump extensions/CheckUser from `b2255f3` to `72187a6` [00:36:51] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/CheckUser-72187a6 - 13https://git.io/vbL5b [00:36:52] [02mediawiki] 07dependabot[bot] opened pull request 03#4901: Bump extensions/CheckUser from `b2255f3` to `72187a6` - 13https://git.io/J9ZTe [00:36:54] [02mediawiki] 07github-actions[bot] labeled pull request 03#4894: Bump extensions/GlobalWatchlist from `d3cb716` to `d302af9` - 13https://git.io/J9ZUS [00:36:55] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/AdminLinks-3daf69d [+0/-0/±1] 13https://git.io/J9ZTv [00:36:57] [02miraheze/mediawiki] 07dependabot[bot] 03ca58ddc - Bump extensions/AdminLinks from `f385200` to `3daf69d` [00:36:58] [02mediawiki] 07dependabot[bot] opened pull request 03#4902: Bump extensions/AdminLinks from `f385200` to `3daf69d` - 13https://git.io/J9ZTf [00:37:00] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/AdminLinks-3daf69d - 13https://git.io/vbL5b [00:37:01] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/CommonsMetadata-cc10042 [+0/-0/±1] 13https://git.io/J9ZTJ [00:37:03] [02miraheze/mediawiki] 07dependabot[bot] 036dae833 - Bump extensions/CommonsMetadata from `f06ab37` to `cc10042` [00:37:04] [02mediawiki] 07dependabot[bot] opened pull request 03#4903: Bump extensions/CommonsMetadata from `f06ab37` to `cc10042` - 13https://git.io/J9ZTU [00:37:06] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/CommonsMetadata-cc10042 - 13https://git.io/vbL5b [00:37:07] [02mediawiki] 07github-actions[bot] labeled pull request 03#4895: Bump extensions/WikiEditor from `1efc3d8` to `f99f482` - 13https://git.io/J9ZUQ [00:37:09] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/DismissableSiteNotice-f88ae6c [+0/-0/±1] 13https://git.io/J9ZTT [00:37:10] [02miraheze/mediawiki] 07dependabot[bot] 03786542b - Bump extensions/DismissableSiteNotice from `29bb34c` to `f88ae6c` [00:37:12] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/DismissableSiteNotice-f88ae6c - 13https://git.io/vbL5b [00:37:13] [02mediawiki] 07dependabot[bot] opened pull request 03#4904: Bump extensions/DismissableSiteNotice from `29bb34c` to `f88ae6c` - 13https://git.io/J9ZTk [00:37:15] [02mediawiki] 07github-actions[bot] labeled pull request 03#4893: Bump extensions/Translate from `941deb5` to `266785e` - 13https://git.io/J9ZUD [00:37:16] [02mediawiki] 07github-actions[bot] labeled pull request 03#4893: Bump extensions/Translate from `941deb5` to `266785e` - 13https://git.io/J9ZUD [00:37:18] [02mediawiki] 07github-actions[bot] labeled pull request 03#4893: Bump extensions/Translate from `941deb5` to `266785e` - 13https://git.io/J9ZUD [00:37:19] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/DiscussionTools-2fcc0eb [+0/-0/±1] 13https://git.io/J9ZTI [00:37:21] [02miraheze/mediawiki] 07dependabot[bot] 03a436b51 - Bump extensions/DiscussionTools from `7209e5f` to `2fcc0eb` [00:37:22] [02mediawiki] 07dependabot[bot] opened pull request 03#4905: Bump extensions/DiscussionTools from `7209e5f` to `2fcc0eb` - 13https://git.io/J9ZTL [00:37:24] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/DiscussionTools-2fcc0eb - 13https://git.io/vbL5b [00:37:25] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/Flow-8643327 [+0/-0/±1] 13https://git.io/J9ZTt [00:37:27] [02miraheze/mediawiki] 07dependabot[bot] 03c748e14 - Bump extensions/Flow from `fd89ee0` to `8643327` [00:37:28] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.00, 22.39, 22.99 [00:37:28] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/Flow-8643327 - 13https://git.io/vbL5b [00:37:30] [02mediawiki] 07dependabot[bot] opened pull request 03#4906: Bump extensions/Flow from `fd89ee0` to `8643327` - 13https://git.io/J9ZTq [00:37:31] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/Cite-797d179 [+0/-0/±1] 13https://git.io/J9ZTm [00:37:33] [02miraheze/mediawiki] 07dependabot[bot] 03158adca - Bump extensions/Cite from `a78f570` to `797d179` [00:37:34] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/Cite-797d179 - 13https://git.io/vbL5b [00:37:36] [02mediawiki] 07dependabot[bot] opened pull request 03#4907: Bump extensions/Cite from `a78f570` to `797d179` - 13https://git.io/J9ZTY [00:37:37] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/ContactPage-fbd203c [+0/-0/±1] 13https://git.io/J9ZTO [00:37:39] [02miraheze/mediawiki] 07dependabot[bot] 03c26b5e9 - Bump extensions/ContactPage from `d5a4827` to `fbd203c` [00:37:40] [02mediawiki] 07github-actions[bot] labeled pull request 03#4897: Bump extensions/DynamicPageList3 from `b338ff7` to `db51e49` - 13https://git.io/J9ZUF [00:37:42] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/ContactPage-fbd203c - 13https://git.io/vbL5b [00:37:43] [02mediawiki] 07dependabot[bot] opened pull request 03#4908: Bump extensions/ContactPage from `d5a4827` to `fbd203c` - 13https://git.io/J9ZT3 [00:37:45] [02mediawiki] 07github-actions[bot] labeled pull request 03#4902: Bump extensions/AdminLinks from `f385200` to `3daf69d` - 13https://git.io/J9ZTf [00:37:46] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/RatePage-5c1e4b0 [+0/-0/±1] 13https://git.io/J9ZTs [00:37:48] [02miraheze/mediawiki] 07dependabot[bot] 0371caf0b - Bump extensions/RatePage from `a9d6b32` to `5c1e4b0` [00:37:49] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/RatePage-5c1e4b0 - 13https://git.io/vbL5b [00:37:51] [02mediawiki] 07dependabot[bot] opened pull request 03#4909: Bump extensions/RatePage from `a9d6b32` to `5c1e4b0` - 13https://git.io/J9ZTG [00:37:52] [02mediawiki] 07github-actions[bot] labeled pull request 03#4898: Bump extensions/ProofreadPage from `6878509` to `a4aedd2` - 13https://git.io/J9ZUN [00:37:54] [02mediawiki] 07github-actions[bot] labeled pull request 03#4900: Bump extensions/PortableInfobox from `04cfa29` to `c1fd031` - 13https://git.io/J9ZUh [00:37:55] [02mediawiki] 07github-actions[bot] labeled pull request 03#4903: Bump extensions/CommonsMetadata from `f06ab37` to `cc10042` - 13https://git.io/J9ZTU [00:37:57] [02mediawiki] 07github-actions[bot] labeled pull request 03#4900: Bump extensions/PortableInfobox from `04cfa29` to `c1fd031` - 13https://git.io/J9ZUh [00:37:58] [02mediawiki] 07github-actions[bot] labeled pull request 03#4896: Bump extensions/CentralNotice from `d3d1795` to `794de94` - 13https://git.io/J9ZU5 [00:38:00] [02mediawiki] 07github-actions[bot] labeled pull request 03#4904: Bump extensions/DismissableSiteNotice from `29bb34c` to `f88ae6c` - 13https://git.io/J9ZTk [00:38:01] [02mediawiki] 07github-actions[bot] labeled pull request 03#4901: Bump extensions/CheckUser from `b2255f3` to `72187a6` - 13https://git.io/J9ZTe [00:38:03] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/CiteThisPage-67239de [+0/-0/±1] 13https://git.io/J9ZTZ [00:38:04] [02miraheze/mediawiki] 07dependabot[bot] 039d5751e - Bump extensions/CiteThisPage from `a6721da` to `67239de` [00:38:06] [02mediawiki] 07dependabot[bot] opened pull request 03#4910: Bump extensions/CiteThisPage from `a6721da` to `67239de` - 13https://git.io/J9ZTn [00:38:07] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/CiteThisPage-67239de - 13https://git.io/vbL5b [00:38:09] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/ApprovedRevs-083cc00 [+0/-0/±1] 13https://git.io/J9ZTC [00:38:10] [02miraheze/mediawiki] 07dependabot[bot] 032c79ace - Bump extensions/ApprovedRevs from `609df70` to `083cc00` [00:38:12] [02mediawiki] 07dependabot[bot] opened pull request 03#4911: Bump extensions/ApprovedRevs from `609df70` to `083cc00` - 13https://git.io/J9ZTW [00:38:13] [02mediawiki] 07github-actions[bot] labeled pull request 03#4899: Bump extensions/FlaggedRevs from `796798a` to `f3a8ffc` - 13https://git.io/J9ZUx [00:38:15] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/ApprovedRevs-083cc00 - 13https://git.io/vbL5b [00:38:15] PROBLEM - cp21 Stunnel Http for mw11 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:38:16] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/WikibaseQualityConstraints-7b9ee7f [+0/-0/±1] 13https://git.io/J9ZTl [00:38:18] [02miraheze/mediawiki] 07dependabot[bot] 032ceba3d - Bump extensions/WikibaseQualityConstraints from `9752e3d` to `7b9ee7f` [00:38:19] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/WikibaseQualityConstraints-7b9ee7f - 13https://git.io/vbL5b [00:38:21] [02mediawiki] 07dependabot[bot] opened pull request 03#4912: Bump extensions/WikibaseQualityConstraints from `9752e3d` to `7b9ee7f` - 13https://git.io/J9ZT8 [00:38:22] [02mediawiki] 07github-actions[bot] labeled pull request 03#4905: Bump extensions/DiscussionTools from `7209e5f` to `2fcc0eb` - 13https://git.io/J9ZTL [00:38:23] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.35, 6.90, 7.95 [00:38:24] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/Thanks-9421bd5 [+0/-0/±1] 13https://git.io/J9ZT4 [00:38:25] [02miraheze/mediawiki] 07dependabot[bot] 03515f51a - Bump extensions/Thanks from `06f0c44` to `9421bd5` [00:38:27] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/Thanks-9421bd5 - 13https://git.io/vbL5b [00:38:27] PROBLEM - cp30 Stunnel Http for mw11 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:38:28] [02mediawiki] 07dependabot[bot] opened pull request 03#4913: Bump extensions/Thanks from `06f0c44` to `9421bd5` - 13https://git.io/J9ZTB [00:38:30] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/Interwiki-a525e3e [+0/-0/±1] 13https://git.io/J9ZTR [00:38:31] [02miraheze/mediawiki] 07dependabot[bot] 03f7315a9 - Bump extensions/Interwiki from `25ba830` to `a525e3e` [00:38:33] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/Interwiki-a525e3e - 13https://git.io/vbL5b [00:38:34] [02mediawiki] 07dependabot[bot] opened pull request 03#4914: Bump extensions/Interwiki from `25ba830` to `a525e3e` - 13https://git.io/J9ZT0 [00:38:36] [02mediawiki] 07github-actions[bot] labeled pull request 03#4908: Bump extensions/ContactPage from `d5a4827` to `fbd203c` - 13https://git.io/J9ZT3 [00:38:37] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/Echo-34677a0 [+0/-0/±1] 13https://git.io/J9ZTE [00:38:39] [02miraheze/mediawiki] 07dependabot[bot] 03d49ad50 - Bump extensions/Echo from `e2d6620` to `34677a0` [00:38:40] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/Echo-34677a0 - 13https://git.io/vbL5b [00:38:42] [02mediawiki] 07dependabot[bot] opened pull request 03#4915: Bump extensions/Echo from `e2d6620` to `34677a0` - 13https://git.io/J9ZTu [00:38:43] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/PageForms-cc6d16f [+0/-0/±1] 13https://git.io/J9ZTz [00:38:45] [02miraheze/mediawiki] 07dependabot[bot] 033d3e65c - Bump extensions/PageForms from `7c19d92` to `cc6d16f` [00:38:46] [02mediawiki] 07github-actions[bot] labeled pull request 03#4907: Bump extensions/Cite from `a78f570` to `797d179` - 13https://git.io/J9ZTY [00:38:48] [02mediawiki] 07dependabot[bot] opened pull request 03#4916: Bump extensions/PageForms from `7c19d92` to `cc6d16f` - 13https://git.io/J9ZTg [00:38:49] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.53, 3.74, 3.88 [00:38:49] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/PageForms-cc6d16f - 13https://git.io/vbL5b [00:38:51] [02mediawiki] 07github-actions[bot] labeled pull request 03#4906: Bump extensions/Flow from `fd89ee0` to `8643327` - 13https://git.io/J9ZTq [00:38:52] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/Moderation-041c357 [+0/-0/±1] 13https://git.io/J9ZT2 [00:38:54] [02miraheze/mediawiki] 07dependabot[bot] 0398266a0 - Bump extensions/Moderation from `92d530f` to `041c357` [00:38:55] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/Moderation-041c357 - 13https://git.io/vbL5b [00:38:57] [02mediawiki] 07github-actions[bot] labeled pull request 03#4909: Bump extensions/RatePage from `a9d6b32` to `5c1e4b0` - 13https://git.io/J9ZTG [00:38:58] [02mediawiki] 07github-actions[bot] labeled pull request 03#4911: Bump extensions/ApprovedRevs from `609df70` to `083cc00` - 13https://git.io/J9ZTW [00:39:00] [02mediawiki] 07github-actions[bot] labeled pull request 03#4910: Bump extensions/CiteThisPage from `a6721da` to `67239de` - 13https://git.io/J9ZTn [00:39:01] [02mediawiki] 07github-actions[bot] labeled pull request 03#4911: Bump extensions/ApprovedRevs from `609df70` to `083cc00` - 13https://git.io/J9ZTW [00:39:03] [02mediawiki] 07dependabot[bot] opened pull request 03#4917: Bump extensions/Moderation from `92d530f` to `041c357` - 13https://git.io/J9ZTa [00:39:04] [02mediawiki] 07github-actions[bot] labeled pull request 03#4911: Bump extensions/ApprovedRevs from `609df70` to `083cc00` - 13https://git.io/J9ZTW [00:39:06] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/CreateWiki-5698d35 [+0/-0/±1] 13https://git.io/J9ZTV [00:39:07] [02miraheze/mediawiki] 07dependabot[bot] 03e479a3f - Bump extensions/CreateWiki from `e1ee9ab` to `5698d35` [00:39:09] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/CreateWiki-5698d35 - 13https://git.io/vbL5b [00:39:10] [02mediawiki] 07dependabot[bot] opened pull request 03#4918: Bump extensions/CreateWiki from `e1ee9ab` to `5698d35` - 13https://git.io/J9ZTw [00:39:12] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/Wikibase-2261946 [+0/-0/±1] 13https://git.io/J9ZTr [00:39:13] PROBLEM - cp31 Stunnel Http for mw11 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:39:13] [02miraheze/mediawiki] 07dependabot[bot] 03826a402 - Bump extensions/Wikibase from `da93956` to `2261946` [00:39:15] [02mediawiki] 07dependabot[bot] opened pull request 03#4919: Bump extensions/Wikibase from `da93956` to `2261946` - 13https://git.io/J9ZTo [00:39:16] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/Wikibase-2261946 - 13https://git.io/vbL5b [00:39:18] [02mediawiki] 07github-actions[bot] labeled pull request 03#4912: Bump extensions/WikibaseQualityConstraints from `9752e3d` to `7b9ee7f` - 13https://git.io/J9ZT8 [00:39:19] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/MultimediaViewer-6b87177 [+0/-0/±1] 13https://git.io/J9ZTK [00:39:21] [02miraheze/mediawiki] 07dependabot[bot] 0384a1140 - Bump extensions/MultimediaViewer from `412fcea` to `6b87177` [00:39:22] [02mediawiki] 07dependabot[bot] opened pull request 03#4920: Bump extensions/MultimediaViewer from `412fcea` to `6b87177` - 13https://git.io/J9ZT6 [00:39:24] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/MultimediaViewer-6b87177 - 13https://git.io/vbL5b [00:39:25] [02mediawiki] 07github-actions[bot] labeled pull request 03#4913: Bump extensions/Thanks from `06f0c44` to `9421bd5` - 13https://git.io/J9ZTB [00:39:27] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/AntiSpoof-ed0a896 [+0/-0/±1] 13https://git.io/J9ZTi [00:39:28] [02miraheze/mediawiki] 07dependabot[bot] 033a0bb53 - Bump extensions/AntiSpoof from `56b4b17` to `ed0a896` [00:39:30] [02mediawiki] 07github-actions[bot] labeled pull request 03#4914: Bump extensions/Interwiki from `25ba830` to `a525e3e` - 13https://git.io/J9ZT0 [00:39:31] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/AntiSpoof-ed0a896 - 13https://git.io/vbL5b [00:39:33] [02mediawiki] 07dependabot[bot] opened pull request 03#4921: Bump extensions/AntiSpoof from `56b4b17` to `ed0a896` - 13https://git.io/J9ZTP [00:39:34] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/Cargo-b1d6c6e [+0/-0/±1] 13https://git.io/J9ZTX [00:39:36] [02miraheze/mediawiki] 07dependabot[bot] 03980e3f5 - Bump extensions/Cargo from `07f465d` to `b1d6c6e` [00:39:37] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/Cargo-b1d6c6e - 13https://git.io/vbL5b [00:39:39] [02mediawiki] 07dependabot[bot] opened pull request 03#4922: Bump extensions/Cargo from `07f465d` to `b1d6c6e` - 13https://git.io/J9ZT1 [00:39:40] [02mediawiki] 07github-actions[bot] labeled pull request 03#4915: Bump extensions/Echo from `e2d6620` to `34677a0` - 13https://git.io/J9ZTu [00:39:42] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/CategoryTree-64c6802 [+0/-0/±1] 13https://git.io/J9ZTM [00:39:43] [02miraheze/mediawiki] 07dependabot[bot] 03eff5527 - Bump extensions/CategoryTree from `21c4cb0` to `64c6802` [00:39:45] [02mediawiki] 07dependabot[bot] opened pull request 03#4923: Bump extensions/CategoryTree from `21c4cb0` to `64c6802` - 13https://git.io/J9ZTD [00:39:46] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/CategoryTree-64c6802 - 13https://git.io/vbL5b [00:39:48] [02mediawiki] 07github-actions[bot] labeled pull request 03#4916: Bump extensions/PageForms from `7c19d92` to `cc6d16f` - 13https://git.io/J9ZTg [00:39:49] [02mediawiki] 07github-actions[bot] labeled pull request 03#4916: Bump extensions/PageForms from `7c19d92` to `cc6d16f` - 13https://git.io/J9ZTg [00:39:50] PROBLEM - cp20 Stunnel Http for mw11 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:39:51] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/MediaWikiChat-768cb67 [+0/-0/±1] 13https://git.io/J9ZTy [00:39:52] [02miraheze/mediawiki] 07dependabot[bot] 03fefc9f4 - Bump extensions/MediaWikiChat from `cda9558` to `768cb67` [00:39:54] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/MediaWikiChat-768cb67 - 13https://git.io/vbL5b [00:39:55] [02mediawiki] 07github-actions[bot] labeled pull request 03#4916: Bump extensions/PageForms from `7c19d92` to `cc6d16f` - 13https://git.io/J9ZTg [00:39:57] [02mediawiki] 07dependabot[bot] opened pull request 03#4924: Bump extensions/MediaWikiChat from `cda9558` to `768cb67` - 13https://git.io/J9ZTS [00:39:58] [02mediawiki] 07github-actions[bot] labeled pull request 03#4917: Bump extensions/Moderation from `92d530f` to `041c357` - 13https://git.io/J9ZTa [00:40:00] [02mediawiki] 07github-actions[bot] labeled pull request 03#4917: Bump extensions/Moderation from `92d530f` to `041c357` - 13https://git.io/J9ZTa [00:40:01] [02mediawiki] 07github-actions[bot] labeled pull request 03#4917: Bump extensions/Moderation from `92d530f` to `041c357` - 13https://git.io/J9ZTa [00:40:03] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/CommentStreams-702ce1c [+0/-0/±1] 13https://git.io/J9ZT9 [00:40:04] [02miraheze/mediawiki] 07dependabot[bot] 03abc1aa7 - Bump extensions/CommentStreams from `9e2d8f1` to `702ce1c` [00:40:06] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/CommentStreams-702ce1c - 13https://git.io/vbL5b [00:40:07] [02mediawiki] 07dependabot[bot] opened pull request 03#4925: Bump extensions/CommentStreams from `9e2d8f1` to `702ce1c` - 13https://git.io/J9ZTH [00:40:09] [02mediawiki] 07github-actions[bot] labeled pull request 03#4918: Bump extensions/CreateWiki from `e1ee9ab` to `5698d35` - 13https://git.io/J9ZTw [00:40:10] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/ConfirmEdit-a591b21 [+0/-0/±1] 13https://git.io/J9ZTQ [00:40:12] [02miraheze/mediawiki] 07dependabot[bot] 03148b177 - Bump extensions/ConfirmEdit from `2916872` to `a591b21` [00:40:12] PROBLEM - mw11 MediaWiki Rendering on mw11 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:40:13] [02mediawiki] 07dependabot[bot] opened pull request 03#4926: Bump extensions/ConfirmEdit from `2916872` to `a591b21` - 13https://git.io/J9ZT7 [00:40:15] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/ConfirmEdit-a591b21 - 13https://git.io/vbL5b [00:40:16] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/GlobalUserPage-6819809 - 13https://git.io/vbL5b [00:40:18] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/GlobalUserPage-6819809 [+0/-0/±1] 13https://git.io/J9ZTd [00:40:19] [02miraheze/mediawiki] 07dependabot[bot] 035c3f7e2 - Bump extensions/GlobalUserPage from `dfb9f81` to `6819809` [00:40:21] [02mediawiki] 07dependabot[bot] opened pull request 03#4927: Bump extensions/GlobalUserPage from `dfb9f81` to `6819809` - 13https://git.io/J9ZTF [00:40:22] [02mediawiki] 07github-actions[bot] labeled pull request 03#4920: Bump extensions/MultimediaViewer from `412fcea` to `6b87177` - 13https://git.io/J9ZT6 [00:40:24] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/AbuseFilter-601ffab [+0/-0/±1] 13https://git.io/J9ZTb [00:40:25] [02miraheze/mediawiki] 07dependabot[bot] 0347b8eba - Bump extensions/AbuseFilter from `4bf6a6a` to `601ffab` [00:40:27] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/AbuseFilter-601ffab - 13https://git.io/vbL5b [00:40:28] [02mediawiki] 07dependabot[bot] opened pull request 03#4928: Bump extensions/AbuseFilter from `4bf6a6a` to `601ffab` - 13https://git.io/J9ZTN [00:40:30] [02mediawiki] 07github-actions[bot] labeled pull request 03#4921: Bump extensions/AntiSpoof from `56b4b17` to `ed0a896` - 13https://git.io/J9ZTP [00:40:31] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/CentralAuth-80029dd [+0/-0/±1] 13https://git.io/J9ZTA [00:40:33] [02miraheze/mediawiki] 07dependabot[bot] 03bd9ed48 - Bump extensions/CentralAuth from `d59e785` to `80029dd` [00:40:34] [02mediawiki] 07dependabot[bot] opened pull request 03#4929: Bump extensions/CentralAuth from `d59e785` to `80029dd` - 13https://git.io/J9ZTx [00:40:36] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/CentralAuth-80029dd - 13https://git.io/vbL5b [00:40:37] [02mediawiki] 07github-actions[bot] labeled pull request 03#4919: Bump extensions/Wikibase from `da93956` to `2261946` - 13https://git.io/J9ZTo [00:40:39] [02mediawiki] 07github-actions[bot] labeled pull request 03#4923: Bump extensions/CategoryTree from `21c4cb0` to `64c6802` - 13https://git.io/J9ZTD [00:40:40] [02mediawiki] 07github-actions[bot] labeled pull request 03#4922: Bump extensions/Cargo from `07f465d` to `b1d6c6e` - 13https://git.io/J9ZT1 [00:40:42] [02mediawiki] 07github-actions[bot] labeled pull request 03#4922: Bump extensions/Cargo from `07f465d` to `b1d6c6e` - 13https://git.io/J9ZT1 [00:40:43] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/PageSchemas-7795965 [+0/-0/±1] 13https://git.io/J9ZTp [00:40:45] [02miraheze/mediawiki] 07dependabot[bot] 0348c639c - Bump extensions/PageSchemas from `7645ba0` to `7795965` [00:40:46] [02mediawiki] 07dependabot[bot] opened pull request 03#4930: Bump extensions/PageSchemas from `7645ba0` to `7795965` - 13https://git.io/J9ZTh [00:40:48] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/PageSchemas-7795965 - 13https://git.io/vbL5b [00:40:49] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.07, 4.12, 4.01 [00:40:49] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/VisualEditor-3aeea63 [+0/-0/±1] 13https://git.io/J9ZTj [00:40:51] [02miraheze/mediawiki] 07dependabot[bot] 03adb9d53 - Bump extensions/VisualEditor from `d12a0aa` to `3aeea63` [00:40:52] [02mediawiki] 07dependabot[bot] opened pull request 03#4931: Bump extensions/VisualEditor from `d12a0aa` to `3aeea63` - 13https://git.io/J9Zke [00:40:54] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/VisualEditor-3aeea63 - 13https://git.io/vbL5b [00:40:55] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/ArticlePlaceholder-23d9495 [+0/-0/±1] 13https://git.io/J9Zkv [00:40:57] [02miraheze/mediawiki] 07dependabot[bot] 03faf174d - Bump extensions/ArticlePlaceholder from `d226109` to `23d9495` [00:40:58] [02mediawiki] 07dependabot[bot] opened pull request 03#4932: Bump extensions/ArticlePlaceholder from `d226109` to `23d9495` - 13https://git.io/J9Zkf [00:41:00] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/ArticlePlaceholder-23d9495 - 13https://git.io/vbL5b [00:41:01] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/skins/MinervaNeue-15047cd [+0/-0/±1] 13https://git.io/J9ZkJ [00:41:03] [02miraheze/mediawiki] 07dependabot[bot] 036604250 - Bump skins/MinervaNeue from `7beea49` to `15047cd` [00:41:04] [02mediawiki] 07dependabot[bot] opened pull request 03#4933: Bump skins/MinervaNeue from `7beea49` to `15047cd` - 13https://git.io/J9ZkU [00:41:06] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/skins/MinervaNeue-15047cd - 13https://git.io/vbL5b [00:41:07] [02mediawiki] 07github-actions[bot] labeled pull request 03#4925: Bump extensions/CommentStreams from `9e2d8f1` to `702ce1c` - 13https://git.io/J9ZTH [00:41:09] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/skins/Cosmos-c0cbc1a [+0/-0/±1] 13https://git.io/J9ZkT [00:41:10] [02miraheze/mediawiki] 07dependabot[bot] 03cee80b1 - Bump skins/Cosmos from `95efb34` to `c0cbc1a` [00:41:12] [02mediawiki] 07dependabot[bot] opened pull request 03#4934: Bump skins/Cosmos from `95efb34` to `c0cbc1a` - 13https://git.io/J9Zkk [00:41:13] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/skins/Cosmos-c0cbc1a - 13https://git.io/vbL5b [00:41:15] [02mediawiki] 07github-actions[bot] labeled pull request 03#4924: Bump extensions/MediaWikiChat from `cda9558` to `768cb67` - 13https://git.io/J9ZTS [00:41:16] [02mediawiki] 07github-actions[bot] labeled pull request 03#4926: Bump extensions/ConfirmEdit from `2916872` to `a591b21` - 13https://git.io/J9ZT7 [00:41:18] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/TemplateData-2896aeb [+0/-0/±1] 13https://git.io/J9ZkI [00:41:19] [02miraheze/mediawiki] 07dependabot[bot] 0352a0d30 - Bump extensions/TemplateData from `998163a` to `2896aeb` [00:41:21] [02mediawiki] 07dependabot[bot] opened pull request 03#4935: Bump extensions/TemplateData from `998163a` to `2896aeb` - 13https://git.io/J9ZkL [00:41:22] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/TemplateData-2896aeb - 13https://git.io/vbL5b [00:41:24] [02mediawiki] 07github-actions[bot] labeled pull request 03#4927: Bump extensions/GlobalUserPage from `dfb9f81` to `6819809` - 13https://git.io/J9ZTF [00:41:25] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/BlogPage-c884723 [+0/-0/±1] 13https://git.io/J9Zkm [00:41:27] [02miraheze/mediawiki] 07dependabot[bot] 035989103 - Bump extensions/BlogPage from `c18c1c3` to `c884723` [00:41:28] [02mediawiki] 07dependabot[bot] opened pull request 03#4936: Bump extensions/BlogPage from `c18c1c3` to `c884723` - 13https://git.io/J9ZkY [00:41:30] [02mediawiki] 07github-actions[bot] labeled pull request 03#4930: Bump extensions/PageSchemas from `7645ba0` to `7795965` - 13https://git.io/J9ZTh [00:41:31] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/BlogPage-c884723 - 13https://git.io/vbL5b [00:41:33] [02mediawiki] 07github-actions[bot] labeled pull request 03#4930: Bump extensions/PageSchemas from `7645ba0` to `7795965` - 13https://git.io/J9ZTh [00:41:34] [02mediawiki] 07github-actions[bot] labeled pull request 03#4932: Bump extensions/ArticlePlaceholder from `d226109` to `23d9495` - 13https://git.io/J9Zkf [00:41:36] [02mediawiki] 07github-actions[bot] labeled pull request 03#4931: Bump extensions/VisualEditor from `d12a0aa` to `3aeea63` - 13https://git.io/J9Zke [00:41:37] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/Math-4cf0a3a [+0/-0/±1] 13https://git.io/J9Zk3 [00:41:39] [02miraheze/mediawiki] 07dependabot[bot] 03f4ab4a4 - Bump extensions/Math from `9ec9134` to `4cf0a3a` [00:41:40] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/Math-4cf0a3a - 13https://git.io/vbL5b [00:41:42] [02mediawiki] 07dependabot[bot] opened pull request 03#4937: Bump extensions/Math from `9ec9134` to `4cf0a3a` - 13https://git.io/J9Zks [00:41:43] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/WikiForum-b1e5d4e [+0/-0/±1] 13https://git.io/J9ZkG [00:41:45] [02miraheze/mediawiki] 07dependabot[bot] 03a99762d - Bump extensions/WikiForum from `d8b6e08` to `b1e5d4e` [00:41:46] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/WikiForum-b1e5d4e - 13https://git.io/vbL5b [00:41:48] [02mediawiki] 07dependabot[bot] opened pull request 03#4938: Bump extensions/WikiForum from `d8b6e08` to `b1e5d4e` - 13https://git.io/J9ZkZ [00:41:49] [02mediawiki] 07github-actions[bot] labeled pull request 03#4928: Bump extensions/AbuseFilter from `4bf6a6a` to `601ffab` - 13https://git.io/J9ZTN [00:41:51] [02mediawiki] 07github-actions[bot] labeled pull request 03#4929: Bump extensions/CentralAuth from `d59e785` to `80029dd` - 13https://git.io/J9ZTx [00:41:52] [02mediawiki] 07github-actions[bot] labeled pull request 03#4934: Bump skins/Cosmos from `95efb34` to `c0cbc1a` - 13https://git.io/J9Zkk [00:41:54] [02mediawiki] 07github-actions[bot] labeled pull request 03#4934: Bump skins/Cosmos from `95efb34` to `c0cbc1a` - 13https://git.io/J9Zkk [00:41:55] [02mediawiki] 07github-actions[bot] labeled pull request 03#4933: Bump skins/MinervaNeue from `7beea49` to `15047cd` - 13https://git.io/J9ZkU [00:41:57] [02mediawiki] 07github-actions[bot] labeled pull request 03#4935: Bump extensions/TemplateData from `998163a` to `2896aeb` - 13https://git.io/J9ZkL [00:41:58] [02mediawiki] 07github-actions[bot] labeled pull request 03#4938: Bump extensions/WikiForum from `d8b6e08` to `b1e5d4e` - 13https://git.io/J9ZkZ [00:42:00] [02mediawiki] 07github-actions[bot] labeled pull request 03#4936: Bump extensions/BlogPage from `c18c1c3` to `c884723` - 13https://git.io/J9ZkY [00:42:01] [02mediawiki] 07github-actions[bot] labeled pull request 03#4937: Bump extensions/Math from `9ec9134` to `4cf0a3a` - 13https://git.io/J9Zks [00:42:24] [02mediawiki] 07Universal-Omega closed pull request 03#4897: Bump extensions/DynamicPageList3 from `b338ff7` to `db51e49` - 13https://git.io/J9ZUF [00:42:26] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/DynamicPageList3-db51e49 [00:42:27] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/DynamicPageList3-db51e49 - 13https://git.io/vbL5b [00:42:28] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/J9Zkg [00:42:30] [02miraheze/mediawiki] 07dependabot[bot] 03c89ab51 - Bump extensions/DynamicPageList3 from `b338ff7` to `db51e49` (#4897) [00:42:38] [02mediawiki] 07Universal-Omega closed pull request 03#4900: Bump extensions/PortableInfobox from `04cfa29` to `c1fd031` - 13https://git.io/J9ZUh [00:42:39] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/PortableInfobox-c1fd031 [00:42:41] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/J9Zka [00:42:42] [02miraheze/mediawiki] 07dependabot[bot] 03053c89f - Bump extensions/PortableInfobox from `04cfa29` to `c1fd031` (#4900) [00:42:44] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/PortableInfobox-c1fd031 - 13https://git.io/vbL5b [00:42:49] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.77, 3.97, 3.97 [00:43:08] !log apt-get upgrade on mw8, mwtask1, and test3 [00:43:19] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 149.56.140.43/cpweb, 2607:5300:201:3100::929a/cpweb [00:44:01] RECOVERY - cp20 Stunnel Http for mw11 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 4.496 second response time [00:44:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:44:14] RECOVERY - mw11 MediaWiki Rendering on mw11 is OK: HTTP OK: HTTP/1.1 200 OK - 23540 bytes in 0.484 second response time [00:44:26] RECOVERY - cp21 Stunnel Http for mw11 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.028 second response time [00:44:47] RECOVERY - cp30 Stunnel Http for mw11 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.319 second response time [00:45:24] RECOVERY - cp31 Stunnel Http for mw11 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.392 second response time [00:45:28] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.21, 21.65, 22.06 [00:46:49] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.02, 3.93, 3.96 [00:46:51] RECOVERY - mw101 APT on mw101 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [00:47:09] RECOVERY - mw102 APT on mw102 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [00:47:27] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.50, 21.87, 22.09 [00:47:49] RECOVERY - mw111 APT on mw111 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [00:48:00] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:48:03] RECOVERY - mw112 APT on mw112 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [00:48:07] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:48:13] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:48:13] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:48:23] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.59, 7.42, 7.62 [00:48:49] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.55, 3.72, 3.88 [00:49:09] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:49:15] RECOVERY - test101 APT on test101 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [00:49:18] !log apt-get upgrade on mw* (SCSVG) and test101 [00:49:27] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.15, 22.94, 22.47 [00:49:30] RECOVERY - mw122 APT on mw122 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [00:49:40] RECOVERY - mw121 APT on mw121 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [00:50:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:51:28] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.37, 23.12, 22.60 [00:53:27] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 27.28, 23.76, 22.85 [00:56:26] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.85, 5.16, 4.38 [00:56:48] !log [@mw11] finished deploy of {'l10nupdate': True} to ovlon - SUCCESS in 7006s [00:56:49] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.00, 3.67, 3.65 [00:57:26] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [00:57:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:58:01] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 21.82, 21.15, 19.66 [00:58:27] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.61, 5.29, 4.53 [00:58:29] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.024 second response time [00:58:30] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.321 second response time [00:58:41] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.314 second response time [00:58:49] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.91, 3.65, 3.64 [00:58:49] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.007 second response time [00:59:20] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [00:59:27] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.10, 23.61, 23.36 [00:59:44] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23540 bytes in 1.738 second response time [01:00:23] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.79, 7.41, 7.76 [01:00:26] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.80, 5.72, 4.77 [01:01:28] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 30.47, 26.98, 24.65 [01:01:48] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 16.58, 19.49, 19.37 [01:02:49] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.95, 3.94, 3.73 [01:04:24] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 2.60, 5.41, 4.99 [01:05:28] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 14.63, 21.29, 22.94 [01:06:23] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 2.79, 4.59, 4.74 [01:07:52] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.45, 6.42, 7.74 [01:09:27] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.23, 20.98, 22.26 [01:09:52] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.21, 6.98, 7.77 [01:10:23] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.38, 7.03, 7.19 [01:11:28] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 19.55, 20.39, 21.89 [01:11:52] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.95, 7.40, 7.83 [01:12:49] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.12, 3.96, 3.96 [01:13:53] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 10.09, 8.19, 8.06 [01:14:49] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.26, 4.20, 4.05 [01:15:52] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.90, 7.38, 7.76 [01:16:23] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.18, 7.40, 7.42 [01:17:41] !log apt-get upgrade on mwtask111 [01:17:47] RECOVERY - mwtask111 APT on mwtask111 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [01:17:52] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.52, 7.88, 7.88 [01:17:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:18:23] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.05, 8.23, 7.73 [01:18:59] !log reboot mwtask111 [01:20:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:21:19] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 51.195.220.68/cpweb, 2001:41d0:801:2000::4c25/cpweb, 149.56.140.43/cpweb [01:21:32] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:22:02] PROBLEM - mwtask111 Check Gluster Clients on mwtask111 is CRITICAL: PROCS CRITICAL: 0 processes with args '/usr/sbin/glusterfs' [01:22:10] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:22:46] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb [01:23:30] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 0.328 second response time [01:24:05] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.942 second response time [01:25:28] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.09, 21.90, 21.42 [01:26:36] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [01:27:19] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [01:27:27] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.98, 22.56, 21.72 [01:34:23] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.65, 7.59, 7.92 [01:35:26] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 198.244.148.90/cpweb, 149.56.141.75/cpweb [01:37:25] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [01:37:27] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.48, 7.39, 7.89 [01:42:23] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.30, 7.92, 7.77 [01:45:25] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2607:5300:201:3100::5ebc/cpweb [01:46:43] PROBLEM - mw13 MediaWiki Rendering on mw13 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:46:44] PROBLEM - cp31 Stunnel Http for mw13 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:47:27] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.81, 7.82, 7.76 [01:47:28] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 18.18, 19.67, 20.29 [01:48:00] PROBLEM - cp20 Stunnel Http for mw13 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:48:15] PROBLEM - cp21 Stunnel Http for mw13 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:48:21] PROBLEM - cp30 Stunnel Http for mw13 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:50:03] RECOVERY - cp20 Stunnel Http for mw13 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 8.415 second response time [01:50:15] RECOVERY - cp21 Stunnel Http for mw13 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 4.228 second response time [01:50:23] RECOVERY - cp30 Stunnel Http for mw13 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 3.646 second response time [01:51:28] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.01, 21.02, 20.70 [01:51:35] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:51:50] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 51.195.220.68/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [01:52:06] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:52:11] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:52:13] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:52:21] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [01:52:50] RECOVERY - cp31 Stunnel Http for mw13 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 1.340 second response time [01:52:54] RECOVERY - mw13 MediaWiki Rendering on mw13 is OK: HTTP OK: HTTP/1.1 200 OK - 23540 bytes in 1.404 second response time [01:53:36] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 3.606 second response time [01:54:02] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 2.713 second response time [01:54:12] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 2.399 second response time [01:54:14] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23540 bytes in 2.788 second response time [01:54:20] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 1.390 second response time [01:55:28] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 18.53, 19.83, 20.28 [01:57:25] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [01:59:31] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [02:00:14] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:00:21] dmehus: still 3 reviews left on PR, then I can probably merge [02:00:37] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:00:39] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:00:45] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:02:09] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 1.399 second response time [02:02:36] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.012 second response time [02:02:40] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23540 bytes in 2.109 second response time [02:02:43] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.307 second response time [02:11:20] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 51.195.220.68/cpweb, 2607:5300:201:3100::929a/cpweb [02:11:52] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.77, 7.53, 7.97 [02:11:59] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 7 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [02:13:52] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.40, 8.01, 8.10 [02:16:57] !log [universalomega@mwtask111] starting deploy of {'world': True} to scsvg [02:17:01] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:17:05] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:17:17] !log [universalomega@test101] starting deploy of {'world': True} to skip [02:17:59] !log [universalomega@test3] starting deploy of {'world': True} to skip [02:18:07] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:18:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:18:26] !log [universalomega@mwtask111] starting deploy of {'world': True} to scsvg [02:18:37] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:18:49] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.26, 3.61, 3.95 [02:18:52] !log [universalomega@test101] starting deploy of {'world': True} to skip [02:19:06] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:19:22] !log [universalomega@mw11] starting deploy of {'world': True} to all [02:20:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:20:47] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:20:49] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.14, 4.08, 4.10 [02:21:01] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:21:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:22:11] !log [universalomega@test101] finished deploy of {'world': True} to skip - SUCCESS in 198s [02:22:50] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 4.110 second response time [02:22:59] PROBLEM - cloud5 Current Load on cloud5 is CRITICAL: CRITICAL - load average: 27.55, 21.92, 19.98 [02:23:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:23:02] !log [universalomega@mwtask111] finished deploy of {'world': True} to scsvg - SUCCESS in 276s [02:23:04] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23538 bytes in 7.110 second response time [02:23:08] PROBLEM - mw11 MediaWiki Rendering on mw11 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:24:19] !log [universalomega@test3] finished deploy of {'world': True} to skip - SUCCESS in 379s [02:24:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:24:46] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 2.846 second response time [02:24:49] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.55, 3.74, 3.97 [02:24:53] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 22.44, 21.78, 20.16 [02:24:54] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 19.58, 20.99, 20.40 [02:25:07] PROBLEM - cp31 Stunnel Http for mw13 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:25:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:25:21] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.320 second response time [02:25:21] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.030 second response time [02:25:31] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23540 bytes in 0.194 second response time [02:25:44] PROBLEM - cp30 Stunnel Http for mw13 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:26:13] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.031 second response time [02:26:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:26:41] PROBLEM - cp20 Stunnel Http for mw10 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:26:47] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:26:51] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 18.17, 19.35, 19.84 [02:26:52] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:26:55] PROBLEM - cp30 Stunnel Http for mw10 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:26:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:27:04] PROBLEM - cp31 Stunnel Http for mw11 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:27:07] PROBLEM - cp20 Stunnel Http for mw11 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:27:11] RECOVERY - cp31 Stunnel Http for mw13 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 8.129 second response time [02:27:16] RECOVERY - mw111 Check Gluster Clients on mw111 is OK: PROCS OK: 1 process with args '/usr/sbin/glusterfs' [02:27:18] RECOVERY - mw102 Check Gluster Clients on mw102 is OK: PROCS OK: 1 process with args '/usr/sbin/glusterfs' [02:27:19] RECOVERY - mw11 MediaWiki Rendering on mw11 is OK: HTTP OK: HTTP/1.1 200 OK - 23540 bytes in 8.500 second response time [02:27:38] RECOVERY - mw121 Check Gluster Clients on mw121 is OK: PROCS OK: 1 process with args '/usr/sbin/glusterfs' [02:27:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:28:02] RECOVERY - mwtask111 Check Gluster Clients on mwtask111 is OK: PROCS OK: 1 process with args '/usr/sbin/glusterfs' [02:28:09] RECOVERY - mw112 Check Gluster Clients on mw112 is OK: PROCS OK: 1 process with args '/usr/sbin/glusterfs' [02:28:38] RECOVERY - mw122 Check Gluster Clients on mw122 is OK: PROCS OK: 1 process with args '/usr/sbin/glusterfs' [02:28:42] !log [universalomega@mw11] finished deploy of {'world': True} to all - SUCCESS in 560s [02:28:44] RECOVERY - cp20 Stunnel Http for mw10 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 8.308 second response time [02:28:49] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.94, 4.05, 4.03 [02:28:59] RECOVERY - cp31 Stunnel Http for mw10 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 9.405 second response time [02:29:00] RECOVERY - cp30 Stunnel Http for mw10 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 9.249 second response time [02:29:02] RECOVERY - cp31 Stunnel Http for mw11 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.470 second response time [02:29:05] RECOVERY - cp20 Stunnel Http for mw11 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.016 second response time [02:29:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:29:56] RECOVERY - cp30 Stunnel Http for mw13 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 3.983 second response time [02:30:23] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.62, 7.55, 8.00 [02:30:49] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.361 second response time [02:31:25] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [02:32:23] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 7.95, 7.72, 8.01 [02:33:00] RECOVERY - mon111 APT on mon111 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [02:33:03] RECOVERY - jobchron121 APT on jobchron121 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [02:33:19] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [02:34:23] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 4.99, 6.57, 7.54 [02:34:36] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 16.88, 19.23, 19.84 [02:40:23] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.27, 7.10, 7.43 [02:44:11] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 4 datacenters are down: 198.244.148.90/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 2607:5300:201:3100::5ebc/cpweb [02:45:45] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:45:50] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:46:01] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:46:07] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:47:49] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 5.901 second response time [02:47:56] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 7.753 second response time [02:48:09] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23540 bytes in 9.632 second response time [02:48:10] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 9.639 second response time [02:49:10] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 4 datacenters are down: 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb [02:50:58] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:51:10] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:51:14] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:51:24] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:51:25] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:51:27] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:51:39] [02miraheze/puppet] 07paladox pushed 031 commit to 03elasticsearch [+15/-20/±140] 13https://git.io/J9ZW8 [02:51:40] [02miraheze/puppet] 07paladox 031d54286 - Update elasticsearch module [02:51:42] [02puppet] 07paladox created branch 03elasticsearch - 13https://git.io/vbiAS [02:51:48] [02puppet] 07paladox opened pull request 03#2289: Update elasticsearch module - 13https://git.io/J9ZWR [02:52:31] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:52:40] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:53:00] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 6.986 second response time [02:53:10] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 5.672 second response time [02:53:16] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 3.283 second response time [02:53:23] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 2.455 second response time [02:53:23] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23538 bytes in 2.892 second response time [02:53:31] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 5.911 second response time [02:53:31] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.77, 7.15, 7.98 [02:54:33] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 7.215 second response time [02:54:46] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23540 bytes in 7.821 second response time [02:55:31] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 7.95, 7.64, 8.07 [02:56:50] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [02:57:38] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [03:02:23] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.68, 7.25, 7.77 [03:02:53] RECOVERY - es121 Puppet on es121 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [03:03:23] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://git.io/J9ZWH [03:03:25] [02miraheze/puppet] 07paladox 03b497374 - Fix puppet runs on es* [03:03:26] [02puppet] 07paladox created branch 03paladox-patch-5 - 13https://git.io/vbiAS [03:03:28] [02puppet] 07paladox opened pull request 03#2290: Fix puppet runs on es* - 13https://git.io/J9ZWQ [03:03:41] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://git.io/J9ZW7 [03:03:43] [02miraheze/puppet] 07paladox 03ef4a2f3 - Update es111.yaml [03:03:44] [02puppet] 07paladox synchronize pull request 03#2290: Fix puppet runs on es* - 13https://git.io/J9ZWQ [03:03:50] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://git.io/J9ZWF [03:03:52] [02miraheze/puppet] 07paladox 03300b1fd - Update es121.yaml [03:03:52] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.11, 7.48, 8.00 [03:03:53] [02puppet] 07paladox synchronize pull request 03#2290: Fix puppet runs on es* - 13https://git.io/J9ZWQ [03:04:03] [02puppet] 07paladox closed pull request 03#2290: Fix puppet runs on es* - 13https://git.io/J9ZWQ [03:04:05] [02puppet] 07paladox deleted branch 03paladox-patch-5 - 13https://git.io/vbiAS [03:04:06] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±3] 13https://git.io/J9ZWb [03:04:08] [02miraheze/puppet] 07paladox 033e43779 - Fix puppet runs on es* (#2290) [03:04:09] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-5 [03:04:23] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.26, 7.32, 7.71 [03:05:52] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.43, 7.81, 8.05 [03:06:23] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.94, 7.45, 7.73 [03:06:36] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 19.88, 20.42, 19.87 [03:06:53] PROBLEM - es121 Puppet on es121 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Elasticsearch_template[graylog-internal] [03:08:23] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.02, 7.84, 7.85 [03:08:57] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/J9ZlI [03:08:58] [02miraheze/puppet] 07paladox 03ac9a251 - Fix elasticsearch firewall [03:09:06] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±3] 13https://git.io/J9ZlL [03:09:08] [02miraheze/puppet] 07paladox 03cc91816 - Revert "Fix puppet runs on es* (#2290)" [03:10:11] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 4 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.141.75/cpweb [03:10:36] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 20.06, 20.24, 19.94 [03:10:39] RECOVERY - es111 Puppet on es111 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [03:10:53] RECOVERY - es121 Puppet on es121 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [03:11:25] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 7 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb [03:11:29] PROBLEM - cp30 Stunnel Http for mw11 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:11:50] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.85, 20.36, 19.58 [03:13:10] [02miraheze/puppet] 07paladox pushed 031 commit to 03elasticsearch [+0/-0/±1] 13https://git.io/J9Zln [03:13:11] [02miraheze/puppet] 07paladox 038c4eade - Update common.yaml [03:13:13] [02puppet] 07paladox synchronize pull request 03#2289: Update elasticsearch module - 13https://git.io/J9ZWR [03:13:30] RECOVERY - cp30 Stunnel Http for mw11 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 1.235 second response time [03:13:37] paladox: how come there is ns2 alerts on icinga-new? Just wondering. [03:13:47] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 17.27, 19.30, 19.29 [03:13:53] someone must have started ns2 vm [03:14:03] (i'm guessing john) [03:14:10] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:14:23] Ah. Alright. Thanks! [03:14:52] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:14:58] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:14:59] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:15:23] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:16:07] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 3.957 second response time [03:16:53] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 2.932 second response time [03:16:57] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 1.465 second response time [03:16:58] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 1.048 second response time [03:17:22] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23540 bytes in 0.240 second response time [03:17:52] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [03:19:33] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.92, 7.39, 7.94 [03:20:06] [02puppet] 07paladox closed pull request 03#2289: Update elasticsearch module - 13https://git.io/J9ZWR [03:20:08] [02miraheze/puppet] 07paladox deleted branch 03elasticsearch [03:20:09] [02puppet] 07paladox deleted branch 03elasticsearch - 13https://git.io/vbiAS [03:20:15] [02puppet] 07paladox closed pull request 03#2288: matomo/icinga: adjust php config - 13https://git.io/J9GAh [03:20:22] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-4 [03:20:24] [02puppet] 07paladox deleted branch 03paladox-patch-4 - 13https://git.io/vbiAS [03:21:31] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.39, 8.21, 8.18 [03:21:44] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 7 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::5ebc/cpweb [03:22:23] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:22:29] RECOVERY - mail121 APT on mail121 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [03:22:32] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.63, 20.84, 19.93 [03:22:39] RECOVERY - phab121 APT on phab121 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [03:23:39] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [03:24:17] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.041 second response time [03:24:30] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 19.55, 20.10, 19.77 [03:25:24] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 21.55, 20.63, 19.83 [03:27:18] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 19.35, 20.37, 19.85 [03:27:25] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [03:27:56] [02mw-config] 07lens0021 opened pull request 03#4370: Adds VE Temporary flags as experimental - 13https://git.io/J9Z8e [03:29:05] miraheze/mw-config - lens0021 the build passed. [03:31:06] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 20.74, 20.74, 20.14 [03:31:59] [02mw-config] 07Universal-Omega commented on pull request 03#4370: Adds VE Temporary flags as experimental - 13https://git.io/J9Z8G [03:32:04] [02mw-config] 07Universal-Omega closed pull request 03#4370: Adds VE Temporary flags as experimental - 13https://git.io/J9Z8e [03:32:06] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/J9Z8n [03:32:07] [02miraheze/mw-config] 07lens0021 03415ead4 - Adds VE Temporary flags as experimental (#4370) [03:33:00] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 18.54, 20.32, 20.07 [03:33:27] miraheze/mw-config - Universal-Omega the build passed. [03:33:31] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.62, 7.63, 7.97 [03:34:15] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [03:34:24] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.12, 7.05, 7.94 [03:34:46] PROBLEM - cp30 Stunnel Http for mw10 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:34:54] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 21.26, 9.90, 5.37 [03:34:54] PROBLEM - cp31 Stunnel Http for mw13 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:35:16] PROBLEM - mw13 MediaWiki Rendering on mw13 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:35:17] PROBLEM - cp21 Stunnel Http for mw13 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:35:18] PROBLEM - cp30 Stunnel Http for mw13 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:35:25] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 8 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [03:35:34] PROBLEM - cp20 Stunnel Http for mw13 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:35:38] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:35:40] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:36:12] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 48.89, 39.60, 27.92 [03:36:44] RECOVERY - cp30 Stunnel Http for mw10 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.318 second response time [03:36:59] RECOVERY - cp31 Stunnel Http for mw13 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 9.328 second response time [03:37:01] [02mw-config] 07Universal-Omega closed pull request 03#4367: Add HasSomeColours settings to ManageWikiSettings and add narrow and wide options for wgMedikContentWidth - 13https://git.io/J9Gqo [03:37:02] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/J9Z8y [03:37:04] [02miraheze/mw-config] 07Kelvs599 034dd3125 - Add HasSomeColours settings to ManageWikiSettings and add narrow and wide options for wgMedikContentWidth (#4367) [03:37:36] RECOVERY - cp31 Stunnel Http for mw10 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 1.308 second response time [03:37:38] RECOVERY - cp20 Stunnel Http for mw13 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 8.413 second response time [03:37:38] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 23540 bytes in 2.852 second response time [03:38:10] miraheze/mw-config - Universal-Omega the build passed. [03:38:23] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.01, 6.85, 7.59 [03:39:21] RECOVERY - cp21 Stunnel Http for mw13 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 3.308 second response time [03:39:23] RECOVERY - mw13 MediaWiki Rendering on mw13 is OK: HTTP OK: HTTP/1.1 200 OK - 23540 bytes in 3.565 second response time [03:39:25] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [03:39:29] RECOVERY - cp30 Stunnel Http for mw13 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 2.988 second response time [03:39:31] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.26, 7.40, 7.66 [03:40:01] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [03:40:23] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.60, 7.24, 7.65 [03:41:17] !log [@mw11] starting deploy of {'config': True} to ovlon [03:41:56] !log [@mw11] finished deploy of {'config': True} to ovlon - SUCCESS in 39s [03:42:04] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/J9Z4G [03:42:05] [02miraheze/mw-config] 07Universal-Omega 03a05293a - Remove PageViewInfo from $wmgMonologChannels [03:42:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:42:23] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.58, 7.87, 7.84 [03:43:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:43:12] miraheze/mw-config - Universal-Omega the build passed. [03:43:15] PROBLEM - cp31 Stunnel Http for mw13 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:43:25] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 5 datacenters are down: 51.195.220.68/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 2607:5300:201:3100::929a/cpweb [03:43:44] PROBLEM - cp21 Stunnel Http for mw13 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:43:56] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 4 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 149.56.140.43/cpweb, 2607:5300:201:3100::929a/cpweb [03:44:48] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:44:53] PROBLEM - cp20 Stunnel Http for mw10 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:45:04] PROBLEM - cp30 Stunnel Http for mw10 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:45:13] RECOVERY - cp31 Stunnel Http for mw13 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 1.951 second response time [03:45:43] RECOVERY - cp21 Stunnel Http for mw13 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 3.772 second response time [03:46:01] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:46:04] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:46:22] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.00, 7.27, 7.65 [03:47:31] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.60, 7.58, 7.83 [03:47:45] PROBLEM - cp20 Stunnel Http for mw11 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:47:51] !log [@test101] starting deploy of {'config': True} to skip [03:47:52] !log [@test101] finished deploy of {'config': True} to skip - SUCCESS in 0s [03:48:23] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.92, 7.80, 7.79 [03:48:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:48:35] PROBLEM - mw11 MediaWiki Rendering on mw11 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:48:43] PROBLEM - cp30 Stunnel Http for mw11 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:48:50] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 2.215 second response time [03:48:55] RECOVERY - cp20 Stunnel Http for mw10 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 2.389 second response time [03:49:10] RECOVERY - cp30 Stunnel Http for mw10 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 4.461 second response time [03:49:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:49:51] RECOVERY - cp20 Stunnel Http for mw11 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 7.547 second response time [03:50:06] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 23540 bytes in 4.142 second response time [03:50:12] RECOVERY - cp31 Stunnel Http for mw10 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 3.406 second response time [03:50:33] RECOVERY - mw11 MediaWiki Rendering on mw11 is OK: HTTP OK: HTTP/1.1 200 OK - 23540 bytes in 2.422 second response time [03:50:44] RECOVERY - cp30 Stunnel Http for mw11 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 1.307 second response time [03:51:25] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:51:29] !log [@mwtask111] starting deploy of {'config': True} to scsvg [03:51:30] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [03:51:31] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.20, 7.93, 7.92 [03:51:35] !log [@mwtask111] finished deploy of {'config': True} to scsvg - SUCCESS in 6s [03:52:23] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.19, 7.99, 7.92 [03:52:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:52:53] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.23, 5.50, 5.94 [03:53:29] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 9.588 second response time [03:53:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:54:23] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.15, 8.37, 8.06 [03:54:43] !log [@test3] starting deploy of {'config': True} to skip [03:54:44] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 1s [03:55:36] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 4.971 second response time [03:55:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:56:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:56:36] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 21.05, 20.03, 19.36 [03:56:54] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 7.28, 6.23, 6.11 [03:58:36] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 16.90, 18.78, 18.98 [03:59:25] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [04:00:23] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.80, 7.97, 7.97 [04:01:20] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [04:04:53] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.31, 5.09, 5.77 [04:06:23] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.32, 7.52, 7.70 [04:07:33] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.43, 21.32, 24.00 [04:08:23] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.25, 7.31, 7.60 [04:10:23] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.63, 7.84, 7.73 [04:11:55] !log [@mw11] starting deploy of {'config': True} to ovlon [04:12:17] !log [@mw11] finished deploy of {'config': True} to ovlon - SUCCESS in 22s [04:12:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:12:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:12:53] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.92, 4.11, 4.98 [04:14:23] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.68, 7.56, 7.67 [04:15:40] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:18:26] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.27, 7.67, 7.67 [04:22:23] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.19, 7.64, 7.69 [04:22:59] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 21.68, 20.71, 19.76 [04:28:41] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 19.89, 20.38, 19.96 [04:32:36] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 20.51, 20.74, 20.23 [04:35:31] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.14, 6.91, 7.99 [04:37:28] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 18.01, 18.20, 20.14 [04:38:09] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 5.89, 6.77, 7.82 [04:38:36] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 19.37, 20.24, 20.18 [04:39:31] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.02, 7.50, 7.99 [04:42:09] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.75, 7.48, 7.79 [04:42:23] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.87, 7.24, 7.19 [04:42:36] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 19.05, 20.44, 20.32 [04:44:23] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.99, 7.65, 7.37 [04:45:11] [02MirahezeMagic] 07dmehus synchronize pull request 03#311: Re-add global interface messages for default user groups - 13https://git.io/J9GqS [04:45:39] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 51.195.220.68/cpweb [04:45:51] CosmicAlpha: https://github.com/miraheze/MirahezeMagic/pull/311 Can you see if I got them all and then hopefully merge? [04:45:52] [url] Re-add global interface messages for default user groups by dmehus · Pull Request #311 · miraheze/MirahezeMagic · GitHub | github.com [04:47:46] [02MirahezeMagic] 07Universal-Omega reviewed pull request 03#311 commit - 13https://git.io/J9Z6P [04:47:58] dmehus: ^ [04:48:25] [02MirahezeMagic] 07dmehus reviewed pull request 03#311 commit - 13https://git.io/J9Z6X [04:49:08] [02MirahezeMagic] 07dmehus synchronize pull request 03#311: Re-add global interface messages for default user groups - 13https://git.io/J9GqS [04:49:22] CosmicAlpha, thanks! done :) [04:49:27] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 5 datacenters are down: 51.195.220.68/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::5ebc/cpweb [04:49:34] !log [@test3] finished deploy of {'l10nupdate': True} to skip - SUCCESS in 20970s [04:50:01] dmehus: no problem! I find I can finally merge! [04:50:07] *I think [04:50:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:50:18] What.... 20970s! [04:50:23] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.80, 7.91, 7.47 [04:50:24] That is insane!!! [04:50:39] I thought that finished hours ago.... [04:50:48] It almost took 5 hours to finish... [04:51:03] CosmicAlpha, lol yeah [04:51:09] I mean almost 6 hours... [04:51:16] How did that happen. [04:51:19] what is 20970s? [04:51:28] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:51:29] Almost 6 hours... [04:51:32] oh wow [04:51:33] dmehus: ^ [04:51:47] yeah, I had 45 commits on that PR alone [04:51:50] well 39 actually [04:51:56] since 6 were Agent's [04:52:04] How did that happen?? Usually its less then 45 minutes. [04:52:09] so my total commits will go up significantly in that repo [04:52:11] And yeah lol I'll merge now. [04:52:19] not sure [04:52:23] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.56, 7.84, 7.50 [04:52:33] If I can find the PR again lol... [04:52:53] https://github.com/miraheze/MirahezeMagic/pull/311#discussion_r780890815 [04:52:54] [url] Re-add global interface messages for default user groups by dmehus · Pull Request #311 · miraheze/MirahezeMagic · GitHub | github.com [04:53:02] miraheze/MirahezeMagic - dmehus the build passed. [04:53:11] [02MirahezeMagic] 07Universal-Omega closed pull request 03#311: Re-add global interface messages for default user groups - 13https://git.io/J9GqS [04:53:12] [02miraheze/MirahezeMagic] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±5] 13https://git.io/J9ZMQ [04:53:14] [02miraheze/MirahezeMagic] 07dmehus 03d400802 - Re-add global interface messages for default user groups (#311) [04:53:17] thanks :) [04:53:19] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [04:53:25] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [04:53:29] No problem! [04:53:29] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23540 bytes in 3.021 second response time [04:53:38] can you possibly do the Git part tonight too? [04:53:50] Sure [04:53:53] One second. [04:53:53] forget what that is called [04:53:55] thanks :) [04:53:56] bp [04:54:00] It's git [04:54:00] s/bp/np [04:54:00] dmehus meant to say: np [04:54:03] oh [04:54:11] well the step I mean [04:54:19] [[Tech:Adding a new extension]] [04:54:19] https://meta.miraheze.org/wiki/Tech:Adding_a_new_extension [04:54:23] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.06, 7.78, 7.51 [04:54:28] submodule [04:54:59] yeah [04:55:02] that part [04:55:21] that's the part of adding a new extension I don't know how to do :P [04:55:31] seems like the trickiest part tbh [04:55:45] My internet is being super slow... [04:56:23] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.23, 7.58, 7.47 [04:56:36] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 16.41, 19.08, 20.09 [04:57:01] miraheze/MirahezeMagic - dmehus the build passed. [04:58:23] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.54, 7.85, 7.57 [04:59:31] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 4.39, 6.77, 7.81 [05:00:23] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.92, 7.47, 7.47 [05:00:35] miraheze/MirahezeMagic - Universal-Omega the build passed. [05:01:13] well my internet is slow as hell... it is being 30x slower then normal. [05:01:27] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.31, 7.02, 7.92 [05:01:31] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.03, 7.31, 7.88 [05:02:49] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.48, 3.27, 3.80 [05:03:30] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/J9nz8 [05:03:31] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.00, 7.19, 7.77 [05:03:32] [02miraheze/mediawiki] 07Universal-Omega 033f66634 - Update MirahezeMagic [05:03:44] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 5.74, 6.76, 7.76 [05:03:49] !log [universalomega@test3] starting deploy of {'world': True, 'l10n': True} to skip [05:03:52] !log [universalomega@test101] starting deploy of {'world': True, 'l10n': True} to skip [05:03:56] !log [universalomega@mwtask111] starting deploy of {'world': True, 'l10n': True} to scsvg [05:04:00] !log [universalomega@mw11] starting deploy of {'world': True, 'l10n': True} to all [05:04:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:04:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:04:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:04:20] dmehus: ^ deploying [05:04:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:04:50] CosmicAlpha, cool :) [05:05:00] oh, is that what those deploy messages are sometimes? [05:05:27] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.05, 7.27, 7.82 [05:05:45] dmehus: yes. you have to manually run deploy-mediawiki on the canary servers to deploy miraheze/mediawiki changes. [05:07:31] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.40, 7.47, 7.71 [05:07:55] [02mediawiki] 07Universal-Omega closed pull request 03#4892: Bump extensions/MirahezeMagic from `8b32f65` to `00f9ff2` - 13https://git.io/J9ZU1 [05:07:57] [02mediawiki] 07dependabot[bot] commented on pull request 03#4892: Bump extensions/MirahezeMagic from `8b32f65` to `00f9ff2` - 13https://git.io/J9nHO [05:07:59] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/MirahezeMagic-00f9ff2 - 13https://git.io/vbL5b [05:08:00] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/MirahezeMagic-00f9ff2 [05:08:23] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 3.84, 5.65, 6.71 [05:08:36] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 22.88, 19.89, 19.40 [05:08:49] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.06, 3.46, 3.68 [05:10:48] CosmicAlpha, cool. Thanks :) [05:10:49] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.87, 3.53, 3.68 [05:10:55] I just realized something [05:10:58] `"grouppage-oversight": "m:OSpecial:MyLanguage/versight",` [05:11:12] Can you push a commit to correct that and redeploy? [05:11:44] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.98, 7.86, 7.76 [05:11:49] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 6 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb [05:11:54] it was in `en.json` [05:12:23] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.33, 6.54, 6.81 [05:12:26] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [05:12:28] PROBLEM - cp30 Stunnel Http for mw10 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [05:12:29] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:12:43] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [05:12:49] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.13, 3.85, 3.78 [05:13:11] !log [universalomega@test3] finished deploy of {'world': True, 'l10n': True} to skip - SUCCESS in 562s [05:13:15] Oh crap. I just shut my laptop down once it finished deploying code changes to test3 (hooks) to verify that there was no issues.... the deploys are now just running in screen. Do you think it can wait for a few hours dmehus, for Reception123, or is it need to be immediately? [05:13:39] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 2001:41d0:801:2000::4c25/cpweb, 149.56.140.43/cpweb, 2607:5300:201:3100::5ebc/cpweb [05:14:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:14:08] [02MirahezeMagic] 07dmehus opened pull request 03#312: Fix missed typo - 13https://git.io/J9nHW [05:14:11] PROBLEM - cp20 Stunnel Http for mw10 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [05:14:23] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.09, 7.25, 7.03 [05:14:30] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 9.460 second response time [05:14:34] RECOVERY - cp30 Stunnel Http for mw10 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 8.989 second response time [05:14:34] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 23540 bytes in 9.137 second response time [05:14:38] !log [universalomega@test101] finished deploy of {'world': True, 'l10n': True} to skip - SUCCESS in 645s [05:14:46] RECOVERY - cp31 Stunnel Http for mw10 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 6.297 second response time [05:14:49] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.12, 3.60, 3.70 [05:14:49] [02miraheze/MirahezeMagic] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/J9nH8 [05:14:51] [02miraheze/MirahezeMagic] 07Universal-Omega 03fb3b068 - Fix [05:14:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:15:03] No problem [05:15:11] Reception123 or paladox could deploy maybe [05:15:21] paladox is usually pretty good if you ask him nicely [05:15:33] dmehus: oh, I didn't see your PR, I just pushed a commit to fix it myself. [05:15:34] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [05:15:39] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [05:15:45] yeah it can wait [05:15:47] oh sorry [05:15:49] no problem [05:15:52] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.76, 7.33, 7.94 [05:16:00] And yeah they are both great! But they are both off right now likely. [05:16:07] RECOVERY - cp20 Stunnel Http for mw10 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 1.103 second response time [05:16:09] [02MirahezeMagic] 07dmehus commented on pull request 03#312: Fix missed typo - 13https://git.io/J9nH4 [05:16:11] [02MirahezeMagic] 07dmehus closed pull request 03#312: Fix missed typo - 13https://git.io/J9nHW [05:16:23] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.11, 7.11, 7.00 [05:16:50] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.05, 3.92, 3.79 [05:16:58] but yeah it can wait a few hours not a big deal [05:17:05] Thanks! [05:17:15] there's also a script you have to run after deploying, right? [05:17:24] is it initSiteStat.php or something? [05:17:57] dmehus: deploy-mediawiki does it all. And it's rebuildLocalisationCache, initSiteStats just rebuilds stats after an import or something. [05:18:07] CosmicAlpha, oh wow [05:18:23] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.69, 7.47, 7.13 [05:18:25] you guys don't have to run rebuildLocalisationCache anymore or initSiteStats? [05:18:25] dmehus: but when I specify --l10n in deploy-mediawiki it runs that. [05:18:27] that's cool [05:18:34] ah, true [05:18:42] you just specify that script, and it handles it [05:18:49] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.30, 3.39, 3.62 [05:18:58] Yep. So it should be currently rebuilding it right now. [05:19:20] ah that is a plus then [05:19:28] I just wish the SAL logs were more specific [05:19:29] I just left the deploy running in a screen session before I shut down my PC. So it should all be done soon. [05:19:37] instead of just `deploy` [05:19:52] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 10.37, 8.32, 8.14 [05:20:18] there is no 'deploy' the SAL log entries are very specific.... what do you mean more specific? [05:20:23] !log [universalomega@mwtask111] finished deploy of {'world': True, 'l10n': True} to scsvg - SUCCESS in 986s [05:20:31] ^ seems specific to me? [05:20:36] dmehus: ^ [05:20:39] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 18.36, 19.88, 20.03 [05:20:47] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 27.61, 22.67, 20.42 [05:21:26] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 198.244.148.90/cpweb, 2607:5300:201:3100::929a/cpweb [05:21:31] miraheze/MirahezeMagic - dmehus the build passed. [05:21:42] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [05:21:47] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:21:47] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [05:21:50] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:21:58] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [05:22:01] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [05:22:06] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [05:22:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:22:16] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [05:22:20] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 149.56.140.43/cpweb [05:22:40] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [05:23:25] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [05:23:40] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 0.311 second response time [05:23:43] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 0.395 second response time [05:23:43] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 0.008 second response time [05:23:56] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.417 second response time [05:23:57] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 8.638 second response time [05:24:01] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 5.374 second response time [05:24:05] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 5.204 second response time [05:24:13] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 1.293 second response time [05:24:39] miraheze/MirahezeMagic - Universal-Omega the build passed. [05:24:43] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 4.691 second response time [05:26:10] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [05:26:36] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 22.75, 21.11, 20.39 [05:26:38] !log [universalomega@mw11] finished deploy of {'world': True, 'l10n': True} to all - SUCCESS in 1357s [05:26:39] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 18.76, 22.73, 21.48 [05:26:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:26:49] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.45, 3.67, 3.65 [05:28:49] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.93, 3.45, 3.58 [05:29:02] CosmicAlpha, I mean, it'd be nice if it logged what it did, like "finished deploy of {'world': True, 'l10n': True} to scsvv, which included rebuildLocalisationCache and initSiteStats" [05:29:33] [02puppet] 07Universal-Omega opened pull request 03#2291: Overwrite server name with ovlon if all before running - 13https://git.io/J9nHr [05:30:13] dmehus: it doesn't include initSiteStats.... [05:30:18] That is hardly ever ran. [05:30:23] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 4.58, 6.97, 7.52 [05:30:31] Its not part of deploy-mediawiki [05:30:36] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 18.04, 19.88, 20.08 [05:30:40] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:30:51] The logging of the l10n implies rebuildLocalisationCache. [05:31:13] As l10n = localisation [05:31:52] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.40, 7.02, 7.90 [05:31:57] Well Internationalisation and localisation [05:32:09] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 5.31, 7.07, 7.89 [05:32:31] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 14.81, 18.02, 19.84 [05:32:49] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.92, 3.95, 3.69 [05:32:59] I18n= Internationalisation and l10n=localisation to be precise [05:33:37] I think [05:37:31] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.94, 6.50, 7.72 [05:37:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:38:09] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.18, 7.32, 7.68 [05:38:23] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.57, 5.66, 6.63 [05:38:24] CosmicAlpha, oh [05:38:43] Would localisation be a better word than l10n then maybe? [05:39:11] l10n is a known acronym for localization [05:39:17] ^ [05:39:31] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.05, 7.15, 7.81 [05:39:44] yeah, I guess, but to the layperson reading SAL logs... [05:39:48] Localisation is to long really and l10n is much easier for the command. [05:39:52] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.15, 7.07, 7.42 [05:39:59] CosmicAlpha: s/to/too [05:39:59] dmehus thinks CosmicAlpha meant to say: Localisation is too long really and l10n is much easier for the command. [05:40:04] Since the log =the commands used [05:40:49] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.33, 3.95, 3.82 [05:41:42] CosmicAlpha, oh you have to ender =l10n in the deploy command? [05:41:49] s/ender/end [05:41:49] dmehus meant to say: CosmicAlpha, oh you have to end =l10n in the deploy command? [05:41:53] err [05:42:04] s/ender/enter [05:42:04] dmehus meant to say: CosmicAlpha, oh you have to enter =l10n in the deploy command? [05:42:23] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.45, 7.42, 7.09 [05:42:49] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.49, 4.19, 3.93 [05:42:51] dmehus: yes to deploy with localisation rebuild and extension update or similar it is `deploy-mediawiki --world --l10n --servers=all` [05:43:34] CosmicAlpha, ah that makes sense [05:43:41] that's a good use case for the acronym then [05:43:46] localisation would be too long [05:43:59] seems to be deployed on Meta Wiki [05:44:04] Yep [05:44:09] so I will move the interface messages for the default groups [05:44:17] Ok [05:44:23] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.41, 7.03, 6.99 [05:44:58] Only if they are customised though is that necessary. [05:45:03] dmehus [05:45:31] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.35, 7.81, 7.94 [05:46:49] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.20, 3.92, 3.91 [05:47:31] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.42, 8.08, 7.99 [05:47:44] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.54, 7.15, 7.91 [05:48:23] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.76, 7.84, 7.31 [05:48:35] CosmicAlpha, it will be on Meta for Meta:Bureaucrats and related default groups [05:48:45] but yeah just the grouppage ones [05:49:05] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.58, 20.55, 19.73 [05:49:22] Agent: should we do a central notice informing users how to update their locally defined group page locations? [05:49:44] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 10.62, 8.33, 8.24 [05:50:02] dmehus: sure though I think we should link to a page with instructions rather than putting a lengthy explanation [05:50:12] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 5 datacenters are down: 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::5ebc/cpweb [05:50:21] with CentralNotice enabled, some wikis will now have 3 sitenotices! [05:51:45] Agent, yeah, which section should link to on User groups? [05:52:00] I'll write the guide into the page [05:52:04] and also do we need Meta:Users on Meta anymore? [05:52:07] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [05:52:14] I'm currently in the process of adding some stuff to the page [05:52:17] or can we just use User groups#{{int:group-user}} [05:52:17] https://meta.miraheze.org/wiki/int:Template:group-user [05:52:34] Agent, okay, I think we can wait on the CN then till tomorrow maybe [05:52:53] if you want, you can draft a CN and I can deploy later [05:53:44] I mean, technically CNs are primarily a Steward responsibility, but this is an edge case where either could do it [05:54:23] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.36, 7.98, 7.61 [05:54:49] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.20, 3.93, 3.86 [05:55:55] dmehus: I don't think a CentralNotice is a good idea with 3 site notices already on some wikis, fourth or even third wouldn't be great.... [05:56:05] I mean some wikis might have 4 already... [05:56:23] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.61, 8.25, 7.76 [05:56:59] If they are closed + a custom wiki one + the 2 in Sitenotice.php. [05:57:38] We have to many already... 1 is enough I think..... but some wikis already have 2 just from Sitenotice.php. [05:58:23] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.51, 7.71, 7.63 [05:59:31] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.05, 7.46, 7.84 [06:00:49] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.33, 3.89, 3.92 [06:00:51] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 19.04, 20.40, 20.35 [06:01:19] Agent: ^ [06:02:07] CosmicAlpha: agreed, perhaps let's wait out until after the migration/RfF closes [06:02:25] Yeah, that's what I'm thinking. [06:05:52] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.88, 7.04, 7.90 [06:06:24] Well I'm gonna head off for the night now. (Earlier then I have in months) [06:07:31] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.71, 7.22, 7.40 [06:08:36] PROBLEM - cloud5 Current Load on cloud5 is CRITICAL: CRITICAL - load average: 24.21, 20.47, 19.17 [06:10:23] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.21, 7.36, 7.39 [06:10:36] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 23.46, 21.24, 19.60 [06:12:23] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.55, 6.77, 7.18 [06:12:36] PROBLEM - cloud5 Current Load on cloud5 is CRITICAL: CRITICAL - load average: 24.77, 22.18, 20.13 [06:12:49] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.31, 3.64, 3.64 [06:12:57] PROBLEM - cp30 Stunnel Http for mw11 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:13:20] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 4 datacenters are down: 51.195.220.68/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [06:13:25] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 5 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 149.56.140.43/cpweb, 2607:5300:201:3100::5ebc/cpweb [06:13:40] PROBLEM - cp20 Stunnel Http for mw11 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:14:27] PROBLEM - cp31 Stunnel Http for mw11 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:14:36] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 21.89, 22.00, 20.32 [06:14:49] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.74, 3.39, 3.56 [06:15:52] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.01, 6.64, 7.17 [06:16:17] CosmicAlpha, okay fair enough [06:16:18] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:16:22] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:16:26] PROBLEM - cp30 Stunnel Http for mw13 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:16:45] PROBLEM - cp31 Stunnel Http for mw13 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:16:53] Agent: I'll send you my draft wording and then I might delegate you to set it up as a translateable CN [06:17:03] sounds good [06:17:12] but we can hold off enabling until we lose a CN/sitenotice [06:17:14] PROBLEM - mw13 MediaWiki Rendering on mw13 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:17:14] or two [06:17:20] alright [06:17:22] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:17:25] PROBLEM - cp21 Stunnel Http for mw11 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:17:52] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.87, 6.68, 7.12 [06:17:57] in the meantime, if anyone asks, I guess we can refer them to [[User groups#your section]] [06:17:57] https://meta.miraheze.org/wiki/User_groups#your_section [06:18:05] yeah [06:18:23] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.05, 6.12, 6.77 [06:18:23] RECOVERY - cp31 Stunnel Http for mw10 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 8.357 second response time [06:18:27] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 9.376 second response time [06:18:28] PROBLEM - mw11 MediaWiki Rendering on mw11 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:18:35] RECOVERY - cp30 Stunnel Http for mw13 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 9.538 second response time [06:18:49] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.60, 3.06, 3.39 [06:18:49] RECOVERY - cp31 Stunnel Http for mw13 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 9.231 second response time [06:18:52] Can you ask Reception123 to deploy the latest MirahezeMagic update when he's around? [06:19:03] so that the Oversight typo gets fixed lo [06:19:07] lol* [06:19:21] dmehus: see the WMF Phab task lol [06:19:22] RECOVERY - cp30 Stunnel Http for mw11 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 4.698 second response time [06:19:26] RECOVERY - cp21 Stunnel Http for mw11 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 4.052 second response time [06:19:41] Agent, okay [06:19:42] I guess that's why Orain had a seperate local importer group [06:19:49] looking [06:19:52] with only `importupload` in it [06:19:52] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.40, 7.29, 7.29 [06:20:03] RECOVERY - cp20 Stunnel Http for mw11 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 8.604 second response time [06:20:17] yeah [06:20:31] RECOVERY - mw11 MediaWiki Rendering on mw11 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 7.691 second response time [06:20:42] RECOVERY - cp31 Stunnel Http for mw11 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 4.215 second response time [06:21:01] I mean, Legoktm was quite dismissive [06:21:08] typical of Legoktm if you ask me [06:21:11] yeah... [06:21:27] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 3.997 second response time [06:21:43] PROBLEM - cp21 Stunnel Http for mw13 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:21:52] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.35, 6.73, 7.09 [06:21:57] I would prefer to have a fix that could allow Stewards to enable that variable for trusted wikis [06:22:02] PROBLEM - cp20 Stunnel Http for mw13 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:22:09] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.29, 6.86, 7.83 [06:22:24] well, most imports fail either way if they're over 20kbs [06:22:45] perhaps we can delegate Stewards and Global Sysops the task of processing imports for now [06:22:48] but if we have to, I guess we add a local `import-uploader` group that's assignable by stewards [06:22:50] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.04, 3.92, 3.66 [06:22:56] Agent, yeah [06:23:02] primarily I refer users to Phabricator [06:23:16] but for small imports, I'd suggest Stewards and Global Sysops can perform that [06:23:25] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [06:23:29] if any is around then that'd be good for now [06:23:30] but if it's going to be a long term thing, we probably need to have an RfC [06:23:34] RECOVERY - mw13 MediaWiki Rendering on mw13 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 7.736 second response time [06:23:46] RECOVERY - cp21 Stunnel Http for mw13 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 8.186 second response time [06:24:00] dmehus: time for a Global Interface Import Interwiki Administrator? :P [06:24:01] RECOVERY - cp20 Stunnel Http for mw13 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 4.167 second response time [06:24:09] Agent: lol no [06:24:12] aww [06:24:16] better to have a niche group :P [06:24:20] `global-importer` [06:24:36] `global-import-sysop` :P [06:24:57] I can already imagine it, a small Global Importer group with 10 users, all of them inactive :P [06:25:01] akin to the Meta rollbackers [06:25:05] lol true [06:25:10] and half of the interwiki administrators [06:25:16] but if we make it grantable by stewards [06:25:37] make it easily grantable and revocable [06:25:44] no activity in 30-90 days, it's removed [06:25:52] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.34, 6.14, 6.77 [06:25:53] as long as there's some sort of housekeeping, I'd be ok with it [06:26:13] yeah, and global groups will have temporary expiry periods in MW 1.38 [06:26:15] so there's that [06:26:18] ooo [06:26:22] interesting [06:26:25] yeah [06:26:50] {{Done}} for three months. Please do use the tool, and use it responsibly. --Dmehus [06:26:50] https://meta.miraheze.org/wiki/Template:Done [06:27:25] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 198.244.148.90/cpweb, 149.56.140.43/cpweb [06:28:09] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.95, 7.21, 7.59 [06:30:09] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 4.64, 6.29, 7.21 [06:31:19] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [06:31:25] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [06:34:36] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 15.25, 19.01, 20.34 [06:39:27] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 4.86, 6.52, 7.99 [06:40:40] dmehus: I already sent Reception123 a DM to do so. [06:40:54] Agent: I replied to the WMF phab task also. [06:41:27] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.30, 7.25, 8.07 [06:42:23] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.81, 7.03, 6.32 [06:42:32] But we shouldn't be discussing a private task publicly I don't think. [06:43:22] Also, dmehus: ^ [06:43:27] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.74, 7.20, 7.97 [06:43:52] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.13, 7.03, 6.81 [06:44:23] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.92, 6.87, 6.35 [06:45:32] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.74, 6.98, 7.92 [06:47:27] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.74, 7.60, 7.93 [06:48:23] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.35, 6.71, 6.42 [06:48:49] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.49, 3.60, 3.95 [06:49:27] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.48, 7.40, 7.82 [06:49:31] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.21, 7.41, 7.85 [06:49:53] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.62, 7.76, 7.21 [06:50:09] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.42, 7.96, 7.44 [06:51:27] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.29, 7.71, 7.88 [06:51:31] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.61, 6.98, 7.64 [06:51:53] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.19, 6.96, 6.99 [06:52:49] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.69, 3.87, 3.96 [06:53:01] CosmicAlpha: ack, yeah, fair enough [06:53:31] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.10, 7.93, 7.92 [06:55:27] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.65, 7.80, 7.93 [06:55:31] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.75, 7.58, 7.79 [06:56:49] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.14, 3.77, 3.93 [06:57:44] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 4.36, 6.80, 7.90 [06:59:50] CosmicAlpha: ack too [07:00:09] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.43, 7.75, 7.81 [07:01:31] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.74, 7.58, 7.66 [07:01:52] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 6.24, 6.21, 6.67 [07:03:31] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.91, 7.11, 7.46 [07:06:45] [02miraheze/mediawiki] 07Reception123 pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/J9nxU [07:06:47] [02miraheze/mediawiki] 07Reception123 03b986137 - Update MirahezeMagic [07:07:44] !log [reception@mw11] starting deploy of {'world': True, 'l10n': True} to all [07:07:47] ^ dmehus [07:07:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:11:31] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 10.00, 7.82, 7.46 [07:11:44] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.02, 7.44, 7.51 [07:12:49] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.36, 3.96, 3.76 [07:13:52] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.95, 6.27, 6.24 [07:14:49] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.81, 3.87, 3.75 [07:15:27] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.00, 7.77, 7.37 [07:16:09] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.12, 7.22, 7.29 [07:17:27] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.82, 7.97, 7.51 [07:19:52] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.73, 7.82, 6.84 [07:20:49] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.51, 3.99, 3.81 [07:21:27] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.12, 7.47, 7.36 [07:21:28] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.28, 20.21, 18.02 [07:21:56] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.02, 7.49, 6.56 [07:22:49] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.30, 3.71, 3.73 [07:23:27] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.61, 7.46, 7.37 [07:23:55] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.33, 6.97, 6.48 [07:25:27] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.94, 8.02, 7.59 [07:25:27] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 19.15, 20.38, 18.66 [07:25:52] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.52, 7.63, 7.20 [07:25:53] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.61, 6.70, 6.43 [07:26:36] PROBLEM - cloud5 Current Load on cloud5 is CRITICAL: CRITICAL - load average: 26.93, 22.45, 20.17 [07:26:43] !log [reception@mw11] finished deploy of {'world': True, 'l10n': True} to all - SUCCESS in 1139s [07:26:49] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.97, 4.36, 3.97 [07:26:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:27:44] Reception123: also SCSVG please [07:28:36] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 22.00, 21.65, 20.13 [07:29:23] RhinosF1: oh right, which server is the deploy server there? [07:29:27] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.46, 7.60, 7.55 [07:29:31] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.64, 7.73, 7.98 [07:29:39] Reception123: task [07:30:09] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.51, 7.69, 7.87 [07:30:17] thanks [07:30:36] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 16.77, 19.87, 19.67 [07:30:47] !log [reception@mwtask111] starting deploy of {'world': True, 'l10n': True} to all [07:30:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:31:55] Reception123: no not --servers=all, --servers=scsvg for that. [07:32:47] ah [07:33:01] !log [reception@mwtask111] starting deploy of {'world': True, 'l10n': True} to scsvg [07:33:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:33:44] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.05, 7.62, 7.94 [07:35:44] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 7.86, 7.92, 8.02 [07:35:53] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.08, 5.94, 6.65 [07:36:49] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.51, 3.77, 3.84 [07:37:22] [02puppet] 07Reception123 closed pull request 03#2291: Overwrite server name with ovlon if all before running - 13https://git.io/J9nHr [07:37:24] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/J9nhY [07:37:25] [02miraheze/puppet] 07Universal-Omega 032b721ed - Overwrite server name with ovlon if all before running (#2291) [07:38:09] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.11, 7.48, 7.62 [07:39:44] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.32, 7.16, 7.72 [07:42:47] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.40, 7.89, 7.25 [07:43:36] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.76, 7.60, 6.91 [07:44:45] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.03, 6.86, 6.95 [07:45:27] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.63, 7.39, 7.31 [07:45:35] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.30, 7.53, 6.94 [07:48:10] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.38, 7.84, 8.00 [07:49:27] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.88, 7.52, 7.42 [07:49:27] !log [reception@mwtask111] finished deploy of {'world': True, 'l10n': True} to scsvg - SUCCESS in 986s [07:49:32] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.54, 7.54, 7.09 [07:49:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:50:09] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 10.30, 8.60, 8.24 [07:51:27] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.12, 7.69, 7.49 [07:52:38] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.14, 7.85, 7.39 [07:52:49] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.48, 3.59, 3.58 [07:53:27] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.35, 7.60, 7.48 [07:53:44] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.27, 8.00, 7.82 [07:54:36] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.47, 6.92, 7.10 [07:55:27] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 10.88, 8.87, 7.96 [07:55:31] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.21, 8.06, 7.47 [07:56:49] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.93, 3.40, 3.51 [07:57:27] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.54, 7.95, 7.74 [07:57:31] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.60, 7.68, 7.40 [07:57:44] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.28, 7.52, 7.70 [07:59:24] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.01, 6.54, 6.80 [08:00:49] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.36, 3.05, 3.36 [08:01:54] PROBLEM - wiki.konjitownmc.cf - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.konjitownmc.cf could not be found [08:02:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:03:27] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.33, 7.32, 7.44 [08:03:31] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.27, 6.85, 6.99 [08:04:49] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.30, 3.98, 3.65 [08:05:27] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.79, 6.67, 7.18 [08:05:31] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 6.10, 6.37, 6.79 [08:05:44] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 6.22, 5.93, 6.77 [08:06:09] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 4.20, 6.60, 7.79 [08:06:26] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.68, 5.90, 6.58 [08:08:49] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 1.98, 3.33, 3.49 [08:09:31] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.18, 6.86, 6.95 [08:11:27] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 3.91, 5.52, 6.59 [08:11:31] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.91, 6.24, 6.72 [08:12:09] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 10.11, 7.75, 7.81 [08:12:49] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.37, 2.91, 3.29 [08:14:09] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.19, 7.44, 7.69 [08:15:27] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.70, 6.86, 6.91 [08:15:31] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.00, 6.83, 6.86 [08:15:44] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.21, 6.73, 6.81 [08:16:10] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.73, 7.97, 7.84 [08:16:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:17:44] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 6.32, 6.54, 6.73 [08:18:14] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.73, 7.44, 6.82 [08:19:27] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.96, 6.40, 6.76 [08:21:31] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 6.04, 6.58, 6.78 [08:21:44] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.14, 6.86, 6.85 [08:23:27] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.99, 6.99, 6.94 [08:24:08] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.77, 7.93, 7.30 [08:25:27] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 10.72, 8.29, 7.42 [08:25:44] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.44, 7.58, 7.12 [08:27:27] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.23, 7.64, 7.28 [08:27:44] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.33, 7.32, 7.07 [08:28:26] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.42, 3.42, 3.38 [08:29:44] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.61, 8.15, 7.41 [08:30:52] Agent, ah, good point [08:31:24] I completely forgot what I said but thank you [08:31:27] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.34, 7.85, 7.43 [08:32:15] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.01, 3.39, 3.39 [08:33:18] Agent, you said if you can't see ChanServ, you're on the split of side the network, or a very good chance of it [08:33:27] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.71, 7.14, 7.23 [08:33:44] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 5.81, 7.38, 7.32 [08:35:52] dmehus: what could point? [08:36:01] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.71, 5.89, 6.62 [08:36:09] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 5.06, 7.07, 7.82 [08:36:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:37:24] CosmicAlpha, point Agent made about if we can't see ChanServ, we're likely on the split side of the network in a netsplit [08:38:10] dmehus: oh you must've been talking there then because I don't see anything about that here. [08:38:27] CosmicAlpha, oh [08:38:36] CosmicAlpha: yeah, we were talking on the split side of the network [08:39:44] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.86, 7.06, 7.05 [08:39:55] Lol everytime I was on the split end I was all alone with no one to talk to so I just talked to myself.... wandering where everyone went... [08:40:15] That's what happened last time anyways. [08:41:52] Just reconnect and rejoin the non-split side :P [08:42:09] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.76, 7.08, 7.45 [08:42:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:43:44] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.85, 7.59, 7.31 [08:44:09] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.63, 6.98, 7.38 [08:44:22] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.53, 7.01, 6.72 [08:45:44] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.75, 8.24, 7.57 [08:49:44] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.37, 7.81, 7.57 [08:50:08] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.46, 7.71, 7.09 [08:50:09] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.06, 7.48, 7.43 [08:51:27] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.82, 7.51, 7.13 [08:51:52] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.64, 7.78, 6.98 [08:52:03] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.70, 7.69, 7.15 [08:52:09] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.31, 7.15, 7.32 [08:53:27] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.44, 7.16, 7.05 [08:53:52] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.79, 7.86, 7.12 [08:53:58] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.21, 7.79, 7.25 [08:55:27] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.18, 7.75, 7.26 [08:55:44] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.13, 7.66, 7.54 [08:57:08] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.63, 3.44, 3.28 [08:57:27] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.42, 7.10, 7.08 [08:57:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:57:44] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.36, 7.16, 7.35 [08:57:48] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.34, 7.28, 7.20 [08:58:09] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.44, 7.48, 7.35 [08:59:03] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 1.55, 2.88, 3.10 [08:59:21] CosmicAlpha, lol [08:59:27] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.67, 6.27, 6.78 [08:59:53] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.15, 6.29, 6.69 [09:00:09] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 5.83, 6.84, 7.13 [09:01:38] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.07, 6.02, 6.71 [09:03:44] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.19, 5.69, 6.64 [09:04:09] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.56, 5.95, 6.69 [09:14:09] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.02, 6.68, 6.50 [09:15:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:16:09] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 6.35, 6.56, 6.49 [09:20:09] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 12.00, 8.74, 7.32 [09:20:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:21:44] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.97, 8.30, 6.90 [09:23:42] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.55, 7.59, 6.79 [09:24:07] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.49, 6.87, 6.45 [09:25:12] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.17, 3.47, 3.08 [09:25:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:26:03] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.89, 7.33, 6.68 [09:26:09] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.57, 7.61, 7.31 [09:26:24] [02puppet] 07RhinosF1 commented on pull request 03#2252: mwupgradetool: switch to SCSVG servers - 13https://git.io/J9cTO [09:27:07] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.89, 3.72, 3.23 [09:27:38] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.51, 7.59, 6.94 [09:28:00] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.21, 6.33, 6.41 [09:29:36] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.02, 6.81, 6.75 [09:31:35] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.41, 6.09, 6.50 [09:34:09] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.62, 5.67, 6.55 [09:34:49] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.13, 3.13, 3.14 [09:35:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:49:44] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.95, 6.80, 6.23 [09:49:51] PROBLEM - tep.wiki - reverse DNS on sslhost is WARNING: /usr/lib/nagios/plugins/check_reverse_dns.py:101: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead resolved_ip_addr = str(dns_resolver.query(hostname, 'AAAA')[0])Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 148, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 116, in main rdns_hostname = [09:49:51] verse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 101, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.query(hostname, 'AAAA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1089, in query return self.resolve(qname, rdtype, rdclass, tcp, source, File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) [09:49:51] "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.407044172286987 seconds [09:51:13] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.11, 7.55, 6.63 [09:51:47] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.72, 7.68, 6.94 [09:51:48] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.23, 6.53, 5.86 [09:53:12] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.84, 7.36, 6.67 [09:53:44] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 4.57, 6.00, 6.06 [09:55:45] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.42, 6.87, 6.21 [09:57:43] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.68, 6.59, 6.17 [09:59:06] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 6.02, 6.68, 6.63 [10:01:33] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.59, 6.32, 6.75 [10:20:44] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.10, 6.90, 6.27 [10:20:55] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.25, 6.78, 6.47 [10:22:43] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.77, 6.30, 6.15 [10:22:52] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 6.27, 6.58, 6.43 [10:23:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:26:23] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.14, 6.24, 5.56 [10:26:33] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.95, 6.73, 5.73 [10:26:37] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.13, 7.78, 6.77 [10:26:43] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.55, 8.16, 7.09 [10:28:23] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.20, 5.96, 5.55 [10:28:29] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.74, 6.33, 5.70 [10:28:35] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.09, 6.99, 6.60 [10:28:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:28:40] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 5.70, 7.25, 6.88 [10:30:34] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.37, 6.09, 6.31 [10:32:33] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.02, 6.19, 6.55 [10:43:19] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.35, 6.65, 6.29 [10:44:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:45:17] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 6.37, 6.63, 6.33 [10:45:27] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.13, 6.36, 5.57 [10:49:28] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.32, 6.21, 5.73 [10:49:57] PROBLEM - tep.wiki - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 148, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 129, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 79, in check_records cname = str(dns_resolver.query(hostname, 'CNAME')[0]) File "/usr/lib/python3/di [10:49:57] es/dns/resolver.py", line 898, in query raise NoNameservers(request=request, errors=errors)dns.resolver.NoNameservers: All nameservers failed to answer the query tep.wiki. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700: [10:49:57] port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [10:50:57] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.07, 7.10, 6.78 [10:51:10] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 10.54, 8.93, 7.36 [10:52:09] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.29, 7.50, 6.61 [10:52:54] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.18, 7.78, 7.06 [10:54:08] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.08, 7.26, 6.62 [10:54:50] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.30, 7.88, 7.20 [10:55:08] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.59, 7.91, 7.34 [10:56:06] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.00, 6.80, 6.53 [10:59:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:03:01] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 6.14, 6.14, 6.67 [11:04:35] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 6.03, 6.21, 6.69 [11:04:41] RECOVERY - tep.wiki - reverse DNS on sslhost is OK: SSL OK - tep.wiki reverse DNS resolves to cp20.miraheze.org - CNAME OK [11:08:26] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.41, 6.77, 6.80 [11:10:22] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 6.00, 6.42, 6.67 [11:13:52] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.12, 7.48, 6.70 [11:14:13] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.96, 7.55, 7.09 [11:14:48] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.84, 6.92, 6.64 [11:15:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:15:50] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.38, 6.80, 6.55 [11:16:46] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.32, 6.08, 6.37 [11:19:46] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.84, 7.51, 6.91 [11:20:09] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.75, 7.72, 7.18 [11:22:40] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.98, 7.57, 6.94 [11:23:43] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.67, 8.35, 7.36 [11:24:17] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.76, 6.81, 6.04 [11:24:19] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.75, 7.07, 6.38 [11:24:38] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.38, 7.88, 7.13 [11:26:13] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.08, 6.09, 5.87 [11:26:36] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.88, 7.17, 6.96 [11:27:40] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.38, 7.26, 7.16 [11:28:05] [02puppet] 07Reception123 closed pull request 03#2252: mwupgradetool: switch to SCSVG servers - 13https://git.io/JSDmp [11:28:06] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/J9c3l [11:28:08] [02miraheze/puppet] 07Universal-Omega 030ab3c50 - mwupgradetool: switch to SCSVG servers (#2252) [11:28:09] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.25, 7.86, 7.69 [11:28:12] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.76, 6.26, 6.24 [11:28:35] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.68, 6.25, 6.65 [11:29:39] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 3.60, 5.83, 6.64 [11:30:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:36:02] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.94, 6.68, 6.32 [11:36:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:37:45] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.94, 6.68, 6.10 [11:37:58] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.58, 6.32, 6.24 [11:38:11] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.42, 6.98, 7.10 [11:40:22] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.01, 6.31, 6.33 [11:41:35] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 6.59, 6.69, 6.23 [11:44:19] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.10, 7.27, 6.71 [11:44:25] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.00, 7.66, 6.97 [11:45:49] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.03, 6.81, 6.51 [11:46:03] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.59, 7.07, 6.39 [11:46:17] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.91, 7.38, 6.84 [11:46:24] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.19, 7.61, 7.05 [11:47:46] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.51, 6.52, 6.46 [11:47:59] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 4.77, 6.40, 6.25 [11:50:13] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.65, 7.85, 7.08 [11:50:23] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.41, 7.85, 7.22 [11:52:23] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.99, 7.25, 7.06 [11:53:11] [02mediawiki] 07RhinosF1 labeled pull request 03#4934: Bump skins/Cosmos from `95efb34` to `c0cbc1a` - 13https://git.io/J9Zkk [11:53:38] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.29, 6.36, 6.29 [11:54:10] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.77, 7.35, 7.05 [11:54:39] [02mediawiki] 07RhinosF1 labeled pull request 03#4925: Bump extensions/CommentStreams from `9e2d8f1` to `702ce1c` - 13https://git.io/J9ZTH [11:54:50] [02mediawiki] 07RhinosF1 labeled pull request 03#4922: Bump extensions/Cargo from `07f465d` to `b1d6c6e` - 13https://git.io/J9ZT1 [11:55:31] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.84, 6.39, 6.14 [11:55:35] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.38, 7.05, 6.55 [11:55:47] [02mediawiki] 07Reception123 closed pull request 03#4917: Bump extensions/Moderation from `92d530f` to `041c357` - 13https://git.io/J9ZTa [11:55:49] [02mediawiki] 07Reception123 deleted branch 03dependabot/submodules/REL1_37/extensions/Moderation-041c357 - 13https://git.io/vbL5b [11:55:50] [02miraheze/mediawiki] 07Reception123 deleted branch 03dependabot/submodules/REL1_37/extensions/Moderation-041c357 [11:55:52] [02miraheze/mediawiki] 07Reception123 pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/J9cGc [11:55:53] [02miraheze/mediawiki] 07dependabot[bot] 03d8dad8c - Bump extensions/Moderation from `92d530f` to `041c357` (#4917) [11:56:08] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.68, 7.89, 7.29 [11:56:12] [02mediawiki] 07RhinosF1 labeled pull request 03#4911: Bump extensions/ApprovedRevs from `609df70` to `083cc00` - 13https://git.io/J9ZTW [11:56:28] [02mediawiki] 07RhinosF1 unlabeled pull request 03#4911: Bump extensions/ApprovedRevs from `609df70` to `083cc00` - 13https://git.io/J9ZTW [11:57:04] [02mediawiki] 07RhinosF1 labeled pull request 03#4893: Bump extensions/Translate from `941deb5` to `266785e` - 13https://git.io/J9ZUD [11:57:26] [02mediawiki] 07RhinosF1 labeled pull request 03#4887: Bump skins/Femiwiki from `21a54f2` to `a4f62a4` - 13https://git.io/J9ZU2 [11:57:28] [02miraheze/mediawiki] 07Reception123 pushed 032 commits to 03REL1_37 [+0/-0/±1] 13https://git.io/J9cGa [11:57:28] !log [reception@mw11] starting deploy of {'world': True, 'l10n': True} to ovlon [11:57:29] [02miraheze/mediawiki] 07Reception123 032750cd1 - Update Moderation [11:57:31] [02miraheze/mediawiki] 07Reception123 034f0e908 - Merge branch 'REL1_37' of https://github.com/miraheze/mediawiki into REL1_37 [11:57:31] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.41, 6.54, 6.43 [11:57:31] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 6.72, 6.52, 6.22 [11:57:31] [url] GitHub - miraheze/mediawiki: The collaborative editing software that runs Wikipedia. | github.com [11:57:32] !log [reception@mwtask111] starting deploy of {'world': True, 'l10n': True} to scsvg [11:57:40] <020AAMC42> Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:57:41] [url] Tech:Server admin log - Miraheze Meta | meta.miraheze.org [11:57:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:57:49] <020AAMC42> Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:57:50] [url] Tech:Server admin log - Miraheze Meta | meta.miraheze.org [11:57:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:57:54] [02mediawiki] 07RhinosF1 labeled pull request 03#4880: Bump skins/Metrolook from `577b947` to `eed515c` - 13https://git.io/J9ZUG [11:58:06] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.73, 7.14, 7.09 [11:58:33] [02mediawiki] 07RhinosF1 labeled pull request 03#4878: Bump extensions/MobileFrontend from `8f64d85` to `f83e725` - 13https://git.io/J9ZUm [11:58:35] [02mediawiki] 07RhinosF1 labeled pull request 03#4878: Bump extensions/MobileFrontend from `8f64d85` to `f83e725` - 13https://git.io/J9ZUm [11:59:22] [02mediawiki] 07RhinosF1 labeled pull request 03#4875: Bump skins/Medik from `b491fc8` to `833353a` - 13https://git.io/J9ZUT [12:00:22] [02mediawiki] 07RhinosF1 labeled pull request 03#4869: Bump extensions/CreatePageUw from `074295c` to `84db84a` - 13https://git.io/J9ZJF [12:00:23] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 3.53, 5.72, 6.53 [12:01:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:03:31] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.74, 7.26, 6.60 [12:04:01] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.24, 6.06, 6.65 [12:04:09] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.00, 7.07, 7.79 [12:04:33] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.41, 6.92, 6.63 [12:05:32] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.22, 7.23, 6.64 [12:07:03] [02miraheze/GlobalNewFiles] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/J9cZI [12:07:04] [02miraheze/GlobalNewFiles] 07translatewiki 03f97664c - Localisation updates from https://translatewiki.net. [12:07:05] [url] Main page - translatewiki.net | translatewiki.net [12:07:06] [02miraheze/landing] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/J9cZL [12:07:07] [02miraheze/landing] 07translatewiki 03c8ab791 - Localisation updates from https://translatewiki.net. [12:07:08] [url] Main page - translatewiki.net | translatewiki.net [12:07:09] [02miraheze/CreateWiki] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/J9cZt [12:07:10] [02miraheze/CreateWiki] 07translatewiki 03654a711 - Localisation updates from https://translatewiki.net. [12:07:11] [url] Main page - translatewiki.net | translatewiki.net [12:07:12] [02miraheze/ManageWiki] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/J9cZq [12:07:13] [02miraheze/ManageWiki] 07translatewiki 03a5a55c2 - Localisation updates from https://translatewiki.net. [12:07:14] [url] Main page - translatewiki.net | translatewiki.net [12:07:15] [02miraheze/MirahezeMagic] 07translatewiki pushed 031 commit to 03master [+0/-0/±24] 13https://git.io/J9cZm [12:07:16] [02miraheze/MirahezeMagic] 07translatewiki 03324fad0 - Localisation updates from https://translatewiki.net. [12:07:17] [url] Main page - translatewiki.net | translatewiki.net [12:07:18] [02miraheze/MatomoAnalytics] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/J9cZY [12:07:19] [02miraheze/MatomoAnalytics] 07translatewiki 03a79a960 - Localisation updates from https://translatewiki.net. [12:07:20] ... [12:07:27] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.31, 7.26, 6.70 [12:07:31] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.83, 7.57, 6.85 [12:09:31] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.06, 7.91, 7.05 [12:09:49] miraheze/landing - translatewiki the build passed. [12:10:20] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 11.43, 8.46, 7.29 [12:11:27] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.52, 8.32, 7.29 [12:11:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:12:09] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.62, 7.90, 7.72 [12:13:19] PROBLEM - cloud5 Current Load on cloud5 is CRITICAL: CRITICAL - load average: 26.23, 22.04, 19.05 [12:13:19] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 149.56.140.43/cpweb, 2607:5300:201:3100::929a/cpweb [12:13:44] !log [reception@mw11] finished deploy of {'world': True, 'l10n': True} to ovlon - SUCCESS in 976s [12:13:59] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.55, 6.96, 6.80 [12:14:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:14:14] <020AAMC42> Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:14:16] [url] Tech:Server admin log - Miraheze Meta | meta.miraheze.org [12:14:23] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.29, 6.99, 6.51 [12:14:32] !log [reception@mwtask111] finished deploy of {'world': True, 'l10n': True} to scsvg - SUCCESS in 1019s [12:14:44] <020AAMC42> Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:14:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:14:50] [url] Tech:Server admin log - Miraheze Meta | meta.miraheze.org [12:15:13] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 20.02, 21.03, 19.03 [12:15:19] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [12:15:58] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 6.19, 6.69, 6.72 [12:16:09] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 5.91, 7.33, 7.58 [12:16:23] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.09, 6.15, 6.25 [12:19:00] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 18.70, 20.14, 19.12 [12:19:49] miraheze/GlobalNewFiles - translatewiki the build passed. [12:19:53] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 11.26, 8.74, 7.52 [12:20:11] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.15, 8.33, 7.91 [12:20:48] miraheze/ManageWiki - translatewiki the build passed. [12:21:34] miraheze/CreateWiki - translatewiki the build passed. [12:22:23] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.71, 7.30, 6.73 [12:22:53] miraheze/MatomoAnalytics - translatewiki the build passed. [12:25:22] miraheze/MirahezeMagic - translatewiki the build passed. [12:25:52] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 4.97, 7.18, 7.34 [12:26:09] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.10, 7.92, 7.92 [12:26:23] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.22, 6.07, 6.38 [12:26:36] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 20.28, 20.89, 19.91 [12:27:52] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.31, 7.31, 7.34 [12:28:09] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.58, 8.40, 8.08 [12:29:52] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.13, 6.53, 7.06 [12:32:09] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.17, 7.85, 7.94 [12:32:36] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 17.87, 19.61, 19.74 [12:33:31] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.62, 7.77, 7.97 [12:34:09] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.50, 7.98, 7.94 [12:35:01] PROBLEM - tep.wiki - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 148, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 129, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 79, in check_records cname = str(dns_resolver.query(hostname, 'CNAME')[0]) File "/usr/lib/python3/di [12:35:01] es/dns/resolver.py", line 992, in query timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 799, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 30.003291368484497 seconds [12:35:52] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.16, 7.19, 7.10 [12:36:15] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.36, 7.28, 6.83 [12:37:31] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.58, 7.55, 7.78 [12:37:52] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.72, 6.97, 7.03 [12:38:14] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.07, 7.11, 6.82 [12:39:31] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.96, 7.36, 7.70 [12:40:09] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.50, 7.93, 7.98 [12:40:12] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.63, 6.38, 6.58 [12:43:31] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.22, 7.83, 7.80 [12:43:44] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.18, 7.47, 7.96 [12:44:08] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 10.03, 8.03, 7.18 [12:44:09] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.08, 8.04, 8.01 [12:45:31] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.30, 7.74, 7.79 [12:45:44] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 7.84, 7.81, 8.03 [12:45:52] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.30, 8.03, 7.41 [12:46:06] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.81, 7.74, 7.17 [12:47:44] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 5.55, 7.10, 7.76 [12:49:27] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.82, 7.40, 7.94 [12:49:57] RECOVERY - tep.wiki - reverse DNS on sslhost is OK: SSL OK - tep.wiki reverse DNS resolves to cp21.miraheze.org - CNAME OK [12:50:04] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 10.61, 8.81, 7.71 [12:51:27] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.87, 8.20, 8.16 [12:51:44] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.58, 7.88, 7.95 [12:51:45] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.71, 20.67, 19.45 [12:52:08] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 149.56.140.43/cpweb [12:53:43] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 17.04, 20.12, 19.45 [12:53:44] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.89, 7.43, 7.77 [12:54:01] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.08, 7.85, 7.61 [12:54:04] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [12:56:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:57:58] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 7.96, 8.13, 7.78 [12:59:56] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.76, 7.57, 7.61 [13:00:57] PROBLEM - tep.wiki - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 148, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 129, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 79, in check_records cname = str(dns_resolver.query(hostname, 'CNAME')[0]) File "/usr/lib/python3/di [13:00:57] es/dns/resolver.py", line 992, in query timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 799, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 30.001383066177368 seconds [13:01:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:05:31] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.33, 6.20, 6.78 [13:05:52] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.38, 7.43, 7.95 [13:07:27] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.22, 7.33, 7.92 [13:07:44] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 4.09, 5.73, 6.71 [13:09:49] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.27, 5.94, 6.77 [13:11:52] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.64, 7.65, 7.79 [13:12:01] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.53, 3.30, 3.08 [13:13:46] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.43, 6.84, 6.96 [13:13:55] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.05, 3.24, 3.09 [13:15:27] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.70, 7.65, 7.62 [13:15:44] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.74, 7.11, 6.85 [13:16:22] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.70, 7.40, 6.88 [13:17:27] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.75, 7.72, 7.66 [13:17:43] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.69, 8.37, 7.55 [13:17:44] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.26, 7.26, 6.95 [13:18:18] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.36, 7.92, 7.12 [13:19:27] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.72, 8.11, 7.81 [13:19:40] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.34, 3.52, 3.28 [13:19:44] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.35, 7.27, 6.96 [13:21:34] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.08, 3.71, 3.37 [13:21:58] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::1b80/cpweb [13:22:17] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.64, 20.89, 19.90 [13:22:30] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:22:55] [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+1/-0/±0] 13https://git.io/J9c0s [13:22:56] [02miraheze/dns] 07Reception123 039ae1ac3 - add wildbowiki.net zone [13:23:53] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [13:24:29] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.008 second response time [13:25:24] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.82, 3.38, 3.32 [13:26:12] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 18.25, 19.87, 19.73 [13:27:27] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.59, 7.93, 7.95 [13:27:44] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 5.81, 7.37, 7.28 [13:27:53] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.78, 7.73, 7.55 [13:29:14] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.03, 3.44, 3.37 [13:29:38] [02miraheze/ssl] 07Reception123 pushed 033 commits to 03master [+3/-0/±3] 13https://git.io/J9c0P [13:29:39] [02miraheze/ssl] 07Reception123 03fb2b98a - add wiki.pixlies.net cert [13:29:41] [02miraheze/ssl] 07Reception123 03f9ac68c - add wildbowiki.net cert [13:29:42] [02miraheze/ssl] 07Reception123 038f27741 - add wiki.lu9.io cert [13:29:44] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 9.05, 7.80, 7.42 [13:29:48] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.70, 7.85, 7.59 [13:31:09] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.79, 3.74, 3.48 [13:31:43] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.49, 7.32, 7.43 [13:33:03] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.70, 3.44, 3.41 [13:36:53] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.26, 3.84, 3.58 [13:36:55] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.09, 20.61, 20.11 [13:37:54] RECOVERY - tep.wiki - reverse DNS on sslhost is OK: SSL OK - tep.wiki reverse DNS resolves to cp20.miraheze.org - CNAME OK [13:39:44] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 5.20, 7.49, 7.67 [13:40:49] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 19.35, 20.06, 20.01 [13:41:27] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.92, 7.78, 7.66 [13:41:31] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 11.98, 9.00, 7.97 [13:41:44] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.34, 7.94, 7.81 [13:42:51] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.92, 3.99, 3.75 [13:43:27] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.55, 7.63, 7.62 [13:43:31] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.45, 7.75, 7.63 [13:44:09] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 149.56.140.43/cpweb, 2607:5300:201:3100::929a/cpweb [13:44:15] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 198.244.148.90/cpweb, 2001:41d0:801:2000::1b80/cpweb [13:44:50] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:44:55] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:44:59] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:45:05] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:45:24] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:45:44] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.74, 7.82, 7.81 [13:46:10] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [13:46:45] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 0.019 second response time [13:46:51] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 0.321 second response time [13:46:55] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 0.421 second response time [13:47:00] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 0.015 second response time [13:47:22] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.319 second response time [13:47:31] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.48, 8.07, 7.77 [13:47:44] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.45, 7.90, 7.83 [13:47:59] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [13:48:31] PROBLEM - tep.wiki - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 148, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 129, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 66, in check_records nameserversans = dns_resolver.query(root_domain, 'NS') File "/usr/lib/python3/d [13:48:31] ges/dns/resolver.py", line 992, in query timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 799, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 30.004544258117676 seconds [13:48:39] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.62, 20.66, 20.28 [13:49:31] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.48, 7.73, 7.68 [13:49:40] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:49:48] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:50:37] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.78, 22.42, 20.98 [13:50:49] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.00, 3.74, 3.63 [13:50:53] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:51:17] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 9.212 second response time [13:51:19] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:51:24] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:51:31] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.54, 7.61, 7.51 [13:51:31] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 7.763 second response time [13:51:39] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 5 datacenters are down: 51.195.220.68/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [13:51:45] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 6 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [13:51:52] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:52:12] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.37, 4.91, 4.18 [13:52:22] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:52:34] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.45, 22.23, 21.10 [13:52:49] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.96, 3.67, 3.61 [13:53:05] RECOVERY - tep.wiki - reverse DNS on sslhost is OK: /usr/lib/nagios/plugins/check_reverse_dns.py:101: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead resolved_ip_addr = str(dns_resolver.query(hostname, 'AAAA')[0])/usr/lib/nagios/plugins/check_reverse_dns.py:103: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead rev_host = str(dns_resolver.query(ptr_record, "PTR")[0]).rstrip('.')/usr/lib/nagios/plu [13:53:05] heck_reverse_dns.py:66: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead nameserversans = dns_resolver.query(root_domain, 'NS')/usr/lib/nagios/plugins/check_reverse_dns.py:79: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead cname = str(dns_resolver.query(hostname, 'CNAME')[0])SSL OK - tep.wiki reverse DNS resolves to cp20.miraheze.org - CNAME OK [13:53:43] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:53:45] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.20, 8.38, 7.92 [13:54:09] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.66, 5.03, 4.30 [13:54:55] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 1.527 second response time [13:55:14] RECOVERY - tep.wiki - reverse DNS on sslhost is OK: SSL OK - tep.wiki reverse DNS resolves to cp20.miraheze.org - CNAME OK [13:55:35] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [13:55:35] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 8.872 second response time [13:55:36] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 9.383 second response time [13:55:49] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 8.144 second response time [13:56:06] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 5.02, 4.85, 4.31 [13:56:08] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 9.014 second response time [13:56:30] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 6.619 second response time [13:57:19] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.26, 7.46, 7.56 [13:57:39] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 5.52, 7.55, 7.92 [13:58:33] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.10, 3.03, 3.37 [13:59:41] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.74, 7.94, 7.91 [14:01:27] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [14:01:34] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.67, 8.10, 8.03 [14:01:40] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.72, 8.14, 7.98 [14:03:32] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.79, 7.69, 7.87 [14:03:39] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.63, 7.18, 7.65 [14:05:02] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.08, 7.43, 7.43 [14:05:30] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.34, 7.78, 7.88 [14:05:37] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.10, 7.82, 7.84 [14:06:26] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 18.67, 19.43, 20.20 [14:07:02] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.16, 7.61, 7.52 [14:09:02] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.74, 8.13, 7.73 [14:13:10] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb [14:13:49] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 5 datacenters are down: 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [14:15:23] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.93, 20.16, 20.06 [14:15:44] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [14:15:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.78, 3.58, 3.41 [14:16:39] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 21.44, 20.36, 19.33 [14:17:05] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [14:17:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.25, 3.78, 3.50 [14:18:34] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 18.66, 19.76, 19.24 [14:18:41] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:18:43] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:18:51] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:19:16] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:19:29] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:19:34] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 3 datacenters are down: 2001:41d0:801:2000::4c25/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb [14:19:55] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.58, 3.40, 3.40 [14:20:59] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 2001:41d0:801:2000::4c25/cpweb, 149.56.140.43/cpweb [14:21:54] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:22:17] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:22:31] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:22:42] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.013 second response time [14:22:45] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.474 second response time [14:22:47] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:22:59] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 1.032 second response time [14:23:00] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:23:08] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 18.52, 20.25, 20.30 [14:23:23] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 0.416 second response time [14:23:26] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.57, 7.20, 7.82 [14:23:35] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.054 second response time [14:25:24] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.34, 7.83, 7.99 [14:25:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.96, 3.58, 3.45 [14:26:52] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [14:27:03] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.04, 21.62, 20.83 [14:27:23] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.20, 7.71, 7.94 [14:28:04] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 1.505 second response time [14:28:33] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 3.980 second response time [14:28:49] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 0.328 second response time [14:28:54] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 0.014 second response time [14:29:19] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 2.397 second response time [14:29:22] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.08, 7.92, 7.99 [14:31:20] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.60, 6.86, 7.58 [14:31:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.27, 4.17, 3.74 [14:32:57] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [14:33:19] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.17, 7.27, 7.63 [14:35:17] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.86, 7.42, 7.64 [14:35:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.19, 3.86, 3.73 [14:37:16] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.16, 7.55, 7.66 [14:37:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.50, 4.01, 3.79 [14:39:15] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.61, 7.68, 7.70 [14:39:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.01, 3.77, 3.74 [14:41:13] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.48, 8.24, 7.89 [14:43:22] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:43:34] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 51.195.220.68/cpweb [14:43:39] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2607:5300:201:3100::929a/cpweb [14:43:56] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:43:56] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:43:59] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:45:11] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.52, 7.73, 7.75 [14:45:36] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:45:55] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:46:45] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:46:51] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.76, 5.32, 4.54 [14:47:09] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.85, 8.02, 7.85 [14:47:21] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:47:26] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:47:33] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.336 second response time [14:47:50] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.008 second response time [14:48:03] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 1.048 second response time [14:48:03] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 0.917 second response time [14:48:03] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 1.703 second response time [14:48:16] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:48:26] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.02, 22.39, 21.56 [14:48:51] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 8.41, 6.36, 5.01 [14:49:08] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.49, 7.83, 7.80 [14:49:51] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 20.59, 20.42, 19.77 [14:50:16] PROBLEM - cp30 Stunnel Http for mw13 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:51:07] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.62, 8.12, 7.91 [14:51:09] PROBLEM - cp30 Stunnel Http for mw11 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:51:26] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.326 second response time [14:51:27] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 0.007 second response time [14:51:49] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 0.481 second response time [14:52:13] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:52:16] PROBLEM - cp20 Stunnel Http for mw11 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:52:25] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:52:26] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:52:28] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.17, 22.80, 21.94 [14:52:29] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:52:29] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 2.576 second response time [14:52:47] PROBLEM - mw13 MediaWiki Rendering on mw13 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:52:49] PROBLEM - cp20 Stunnel Http for mw13 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:52:54] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.45, 5.80, 5.13 [14:52:56] PROBLEM - cp21 Stunnel Http for mw13 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:53:01] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 5.814 second response time [14:53:05] PROBLEM - cp31 Stunnel Http for mw13 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:53:33] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 1 backends are down. mw12 [14:53:55] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.17, 3.05, 3.34 [14:54:00] PROBLEM - cp31 Stunnel Http for mw11 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:54:02] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.72, 1.77, 1.33 [14:54:10] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:54:12] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:54:12] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 3.471 second response time [14:54:21] RECOVERY - cp30 Stunnel Http for mw13 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 4.700 second response time [14:54:22] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.024 second response time [14:54:23] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.330 second response time [14:54:27] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 0.913 second response time [14:54:29] PROBLEM - cp20 Stunnel Http for mw10 on cp20 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.023 second response time [14:54:36] PROBLEM - mw11 MediaWiki Rendering on mw11 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:54:39] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.253 second response time [14:54:49] RECOVERY - mw13 MediaWiki Rendering on mw13 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 5.699 second response time [14:54:50] RECOVERY - cp20 Stunnel Http for mw13 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 6.122 second response time [14:54:53] PROBLEM - cp30 Stunnel Http for mw10 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [14:55:02] RECOVERY - cp21 Stunnel Http for mw13 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 6.924 second response time [14:55:10] RECOVERY - cp31 Stunnel Http for mw13 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 8.797 second response time [14:55:30] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 18 backends are healthy [14:56:00] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.73, 1.43, 1.26 [14:56:02] RECOVERY - cp31 Stunnel Http for mw11 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 6.893 second response time [14:56:22] RECOVERY - cp20 Stunnel Http for mw11 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 1.478 second response time [14:56:35] RECOVERY - mw11 MediaWiki Rendering on mw11 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 0.899 second response time [14:56:51] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.51, 4.68, 4.83 [14:57:17] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [14:57:29] RECOVERY - cp30 Stunnel Http for mw11 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 3.068 second response time [14:59:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.82, 3.72, 3.55 [15:00:51] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.41, 5.67, 5.18 [15:01:10] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb [15:01:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.13, 3.74, 3.57 [15:02:51] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.91, 5.27, 5.10 [15:03:08] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [15:03:14] RECOVERY - cp31 Stunnel Http for mw10 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 5.038 second response time [15:03:19] RECOVERY - cp30 Stunnel Http for mw10 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 4.239 second response time [15:03:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.66, 3.71, 3.58 [15:04:34] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 1.193 second response time [15:04:48] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 0.405 second response time [15:04:49] RECOVERY - cp20 Stunnel Http for mw10 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.008 second response time [15:05:14] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 18.60, 20.02, 20.25 [15:06:49] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [15:06:51] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.39, 4.67, 4.94 [15:09:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.02, 3.89, 3.69 [15:11:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.71, 3.79, 3.68 [15:12:33] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:12:43] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:12:48] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:12:49] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 6 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb [15:12:51] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 198.244.148.90/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.141.75/cpweb [15:13:01] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:13:09] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:13:13] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 20.56, 19.89, 19.97 [15:13:20] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:13:38] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:13:53] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:13:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.14, 3.94, 3.74 [15:14:18] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:14:27] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.01, 21.18, 20.95 [15:14:40] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:15:14] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 19.46, 19.64, 19.86 [15:15:43] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 9.437 second response time [15:15:57] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 7.065 second response time [15:16:24] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 6.157 second response time [15:16:26] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.34, 20.51, 20.72 [15:17:26] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 9.984 second response time [15:17:30] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 7.266 second response time [15:18:49] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [15:19:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.77, 3.60, 3.70 [15:20:21] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:20:50] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:21:52] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:21:54] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:22:09] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:22:49] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 6 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 2607:5300:201:3100::929a/cpweb [15:22:52] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.188 second response time [15:22:58] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.961 second response time [15:23:06] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 1.537 second response time [15:23:16] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 1.363 second response time [15:23:35] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 4.084 second response time [15:24:26] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.28, 21.95, 21.22 [15:25:40] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.04, 1.96, 1.32 [15:26:51] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 7.77, 5.75, 4.93 [15:27:13] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 22.79, 20.37, 19.76 [15:27:41] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.98, 1.57, 1.25 [15:28:48] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 7.533 second response time [15:29:13] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 19.54, 20.21, 19.79 [15:29:25] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 7.868 second response time [15:29:35] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:29:42] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:30:06] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:30:22] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:30:26] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 9.016 second response time [15:30:26] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.91, 23.74, 22.35 [15:30:34] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 9.424 second response time [15:30:35] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:32:15] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 0.011 second response time [15:32:49] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [15:32:51] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 3.87, 5.44, 5.14 [15:33:55] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.75, 3.07, 3.36 [15:34:51] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.65, 4.87, 4.97 [15:35:19] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.35, 7.36, 7.86 [15:35:47] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 5.125 second response time [15:36:00] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 4.348 second response time [15:36:27] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 7.262 second response time [15:36:34] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 5.707 second response time [15:36:37] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.09, 6.96, 7.84 [15:36:51] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 5.830 second response time [15:38:34] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [15:39:19] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.91, 8.20, 8.10 [15:40:34] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.03, 7.40, 7.82 [15:42:27] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 25.59, 23.28, 22.35 [15:42:34] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2607:5300:201:3100::5ebc/cpweb [15:43:20] PROBLEM - mw11 MediaWiki Rendering on mw11 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:43:34] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 3 datacenters are down: 149.56.140.43/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [15:44:26] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.39, 22.12, 22.04 [15:45:18] RECOVERY - mw11 MediaWiki Rendering on mw11 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 0.372 second response time [15:46:01] [02miraheze/MirahezeMagic] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-1 [+0/-0/±1] 13https://git.io/J9cP5 [15:46:03] [02miraheze/MirahezeMagic] 07RhinosF1 038343742 - Recaptcha: use enterprise version [15:46:04] [02MirahezeMagic] 07RhinosF1 created branch 03RhinosF1-patch-1 - 13https://git.io/fQRGX [15:46:05] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:46:06] [02MirahezeMagic] 07RhinosF1 opened pull request 03#313: Recaptcha: use enterprise version - 13https://git.io/J9cPd [15:46:13] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:46:26] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.38, 22.89, 22.32 [15:46:41] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.67, 3.90, 3.55 [15:47:23] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:47:24] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [15:47:40] [02miraheze/MirahezeMagic] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-1 [+0/-0/±1] 13https://git.io/J9cPN [15:47:41] [02miraheze/MirahezeMagic] 07RhinosF1 03f269ec1 - Update ReCaptchaHtml.php [15:47:43] [02MirahezeMagic] 07RhinosF1 synchronize pull request 03#313: Recaptcha: use enterprise version - 13https://git.io/J9cPd [15:47:49] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:47:55] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:48:26] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.55, 22.31, 22.17 [15:48:36] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.79, 3.39, 3.41 [15:49:30] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 7.614 second response time [15:49:31] [02miraheze/MirahezeMagic] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-1 [+0/-0/±1] 13https://git.io/J9cXe [15:49:33] [02miraheze/MirahezeMagic] 07RhinosF1 03792ce4a - Update ReCaptchaNoCaptcha.php [15:49:34] [02MirahezeMagic] 07RhinosF1 synchronize pull request 03#313: Recaptcha: use enterprise version - 13https://git.io/J9cPd [15:49:48] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 2.387 second response time [15:49:51] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.345 second response time [15:50:16] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 4.003 second response time [15:50:21] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 2.595 second response time [15:51:03] PROBLEM - cp30 Stunnel Http for mw8 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:51:11] PROBLEM - cp20 Stunnel Http for mw8 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:51:14] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 6 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 149.56.140.43/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [15:52:08] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:52:25] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.39, 3.33, 3.38 [15:52:26] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.75, 7.56, 7.97 [15:52:30] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:52:55] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:53:00] RECOVERY - cp30 Stunnel Http for mw8 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14554 bytes in 1.347 second response time [15:53:11] RECOVERY - cp20 Stunnel Http for mw8 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 1.452 second response time [15:53:14] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:53:27] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [15:54:25] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.31, 8.21, 8.15 [15:54:26] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.15, 22.41, 22.22 [15:56:18] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.52, 5.36, 5.12 [15:56:21] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 7.175 second response time [15:56:27] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.69, 22.53, 22.30 [15:56:41] miraheze/MirahezeMagic - RhinosF1 the build passed. [15:56:43] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 7.970 second response time [15:58:23] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.75, 7.25, 7.82 [15:58:46] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:59:02] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 4.68, 7.09, 7.78 [15:59:22] [02mediawiki] 07RhinosF1 labeled pull request 03#4930: Bump extensions/PageSchemas from `7645ba0` to `7795965` - 13https://git.io/J9ZTh [16:00:10] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:00:12] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 4.31, 5.00, 5.05 [16:00:16] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:00:17] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:00:44] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:01:02] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.51, 7.69, 7.92 [16:01:13] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 6.217 second response time [16:01:25] [02mediawiki] 07RhinosF1 labeled pull request 03#4909: Bump extensions/RatePage from `a9d6b32` to `5c1e4b0` - 13https://git.io/J9ZTG [16:01:40] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23579 bytes in 3.729 second response time [16:01:49] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 3.956 second response time [16:02:07] [02mediawiki] 07RhinosF1 labeled pull request 03#4877: Bump extensions/UniversalLanguageSelector from `f33d7ce` to `d2b84d7` - 13https://git.io/J9ZUt [16:02:20] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 9.607 second response time [16:02:21] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 7.851 second response time [16:02:23] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 8.110 second response time [16:02:23] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.47, 7.74, 7.83 [16:02:44] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 4.089 second response time [16:03:00] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 6.436 second response time [16:03:02] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.83, 7.80, 7.94 [16:04:49] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [16:04:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.60, 3.46, 3.38 [16:05:02] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.81, 8.24, 8.08 [16:05:03] !log deploying security fixes [16:05:26] [02mediawiki] 07RhinosF1 commented on pull request 03#4878: Bump extensions/MobileFrontend from `8f64d85` to `f83e725` - 13https://git.io/J9cXD [16:05:31] [02mediawiki] 07RhinosF1 closed pull request 03#4878: Bump extensions/MobileFrontend from `8f64d85` to `f83e725` - 13https://git.io/J9ZUm [16:05:32] [02mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/extensions/MobileFrontend-f83e725 - 13https://git.io/vbL5b [16:05:34] [02miraheze/mediawiki] 07RhinosF1 deleted branch 03dependabot/submodules/REL1_37/extensions/MobileFrontend-f83e725 [16:05:35] [02miraheze/mediawiki] 07RhinosF1 pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/J9cXy [16:05:37] [02miraheze/mediawiki] 07dependabot[bot] 031e4393d - Bump extensions/MobileFrontend from `8f64d85` to `f83e725` (#4878) [16:05:37] <020AAMC42> Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:05:51] [url] Tech:Server admin log - Miraheze Meta | meta.miraheze.org [16:05:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:06:34] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [16:08:49] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::1b80/cpweb [16:10:34] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb [16:11:40] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 22.77, 21.25, 19.99 [16:11:46] PROBLEM - cp30 Stunnel Http for mw13 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:11:59] PROBLEM - cp21 Stunnel Http for mw13 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:12:07] PROBLEM - cp31 Stunnel Http for mw13 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:13:05] PROBLEM - cp20 Stunnel Http for mw10 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:13:33] !log cd extensions/MobileFrontend && sudo -u www-data git reset origin --hard && cd ../.. on all servers to clear previous local hack and deploy 1e4393d [16:13:47] PROBLEM - mw11 MediaWiki Rendering on mw11 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:13:47] PROBLEM - mw13 MediaWiki Rendering on mw13 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:13:53] PROBLEM - cp20 Stunnel Http for mw13 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:14:29] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.31, 3.89, 3.62 [16:14:59] PROBLEM - cp30 Stunnel Http for mw11 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:15:01] RECOVERY - cp20 Stunnel Http for mw10 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.012 second response time [16:15:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:15:18] <020AAMC42> Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:15:22] [url] Tech:Server admin log - Miraheze Meta | meta.miraheze.org [16:15:31] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 19.08, 20.07, 19.80 [16:15:55] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.18, 5.21, 5.02 [16:16:24] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.70, 3.36, 3.45 [16:16:40] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:16:42] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:16:42] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:17:05] RECOVERY - cp30 Stunnel Http for mw11 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 6.194 second response time [16:17:22] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:17:24] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:17:52] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.70, 4.61, 4.81 [16:17:53] !log [rhinos@test3] starting deploy of {'world': True, 'l10n': True} to skip [16:17:58] !log [rhinos@test101] starting deploy of {'world': True, 'l10n': True} to skip [16:17:59] RECOVERY - mw11 MediaWiki Rendering on mw11 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 4.384 second response time [16:18:04] !log [rhinos@mw11] starting deploy of {'world': True, 'l10n': True} to ovlon [16:18:12] !log [rhinos@mwtask111] starting deploy of {'world': True, 'l10n': True} to scsvg [16:18:19] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 1.95, 2.90, 3.27 [16:19:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:19:59] <020AAMC42> Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:20:33] [url] Tech:Server admin log - Miraheze Meta | meta.miraheze.org [16:20:34] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:20:42] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:20:58] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:22:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:22:33] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.376 second response time [16:22:38] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 0.024 second response time [16:22:50] <020AAMC42> Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:22:51] [url] Tech:Server admin log - Miraheze Meta | meta.miraheze.org [16:23:00] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 9.420 second response time [16:23:01] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 1.010 second response time [16:23:09] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 8.351 second response time [16:23:14] PROBLEM - cloud5 Current Load on cloud5 is CRITICAL: CRITICAL - load average: 25.57, 21.86, 20.44 [16:23:19] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.99, 7.23, 7.90 [16:23:37] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 8.213 second response time [16:23:42] PROBLEM - cp31 Stunnel Http for mw11 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:23:45] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 7.023 second response time [16:23:57] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:24:07] PROBLEM - cp30 Stunnel Http for mw10 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:24:09] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:24:25] PROBLEM - mw11 MediaWiki Rendering on mw11 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:24:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:24:44] RECOVERY - cp31 Stunnel Http for mw13 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 4.387 second response time [16:24:47] RECOVERY - cp21 Stunnel Http for mw13 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 3.467 second response time [16:24:53] <020AAMC42> Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:24:56] PROBLEM - cp20 Stunnel Http for mw11 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:25:02] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.44, 3.62, 3.44 [16:25:19] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.89, 7.90, 8.04 [16:25:21] [url] Tech:Server admin log - Miraheze Meta | meta.miraheze.org [16:25:23] PROBLEM - cp20 Stunnel Http for mw10 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:25:35] PROBLEM - cp21 Stunnel Http for mw11 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:25:40] PROBLEM - cp30 Stunnel Http for mw11 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:26:05] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 1 backends are down. mw10 [16:26:15] RECOVERY - cp30 Stunnel Http for mw10 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 7.594 second response time [16:26:17] RECOVERY - cp31 Stunnel Http for mw10 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 7.656 second response time [16:26:21] RECOVERY - cp20 Stunnel Http for mw13 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 6.987 second response time [16:26:21] RECOVERY - mw13 MediaWiki Rendering on mw13 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 8.381 second response time [16:26:26] <020AAMC42> Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:26:27] RECOVERY - cp30 Stunnel Http for mw13 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 7.252 second response time [16:26:35] !log [rhinos@test3] finished deploy of {'world': True, 'l10n': True} to skip - SUCCESS in 523s [16:26:42] ... [16:27:00] RECOVERY - cp20 Stunnel Http for mw11 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 6.960 second response time [16:27:11] <020AAMC42> Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:27:13] ... [16:27:13] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 21.37, 22.45, 21.07 [16:27:21] RECOVERY - cp20 Stunnel Http for mw10 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 3.008 second response time [16:27:26] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:27:32] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:27:34] RECOVERY - cp21 Stunnel Http for mw11 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.062 second response time [16:27:40] RECOVERY - cp30 Stunnel Http for mw11 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.311 second response time [16:27:42] RECOVERY - cp31 Stunnel Http for mw11 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.309 second response time [16:27:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:27:59] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.009 second response time [16:28:00] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:28:05] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 18 backends are healthy [16:28:09] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:28:32] !log [rhinos@test101] finished deploy of {'world': True, 'l10n': True} to skip - SUCCESS in 634s [16:28:34] RECOVERY - mw11 MediaWiki Rendering on mw11 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 2.235 second response time [16:28:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:29:10] <020AAMC42> Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:29:12] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 7.353 second response time [16:29:15] [url] Tech:Server admin log - Miraheze Meta | meta.miraheze.org [16:29:26] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 2.844 second response time [16:29:31] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 1.965 second response time [16:29:56] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.331 second response time [16:30:08] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 1.127 second response time [16:30:26] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:30:27] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 25.14, 22.31, 21.51 [16:30:47] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.38, 3.76, 3.59 [16:31:36] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:32:06] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:32:44] PROBLEM - cp30 Stunnel Http for mw13 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:33:08] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:33:10] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:33:19] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.57, 7.69, 7.97 [16:33:26] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.74, 5.47, 4.91 [16:34:26] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.90, 24.00, 22.52 [16:34:36] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 5.668 second response time [16:34:46] RECOVERY - cp30 Stunnel Http for mw13 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 6.377 second response time [16:35:06] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 1.577 second response time [16:35:09] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 2.212 second response time [16:35:29] !log [rhinos@mwtask111] finished deploy of {'world': True, 'l10n': True} to scsvg - SUCCESS in 1036s [16:35:49] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 4.503 second response time [16:35:53] <020AAMC42> Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:36:03] [url] Tech:Server admin log - Miraheze Meta | meta.miraheze.org [16:36:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:36:31] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.89, 3.91, 3.66 [16:37:14] PROBLEM - cloud5 Current Load on cloud5 is CRITICAL: CRITICAL - load average: 28.06, 23.89, 22.00 [16:37:19] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.19, 7.94, 8.00 [16:37:20] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.89, 4.96, 4.84 [16:37:31] !log [rhinos@mw11] finished deploy of {'world': True, 'l10n': True} to ovlon - SUCCESS in 1166s [16:37:40] !log security deploys done [16:38:15] <020AAMC42> Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:38:17] [url] Tech:Server admin log - Miraheze Meta | meta.miraheze.org [16:38:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:38:24] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 0.462 second response time [16:38:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:39:03] <020AAMC42> Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:39:12] ... [16:39:14] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 22.57, 23.60, 22.15 [16:39:19] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.12, 7.32, 7.77 [16:40:20] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.57, 3.87, 3.71 [16:40:56] .kick 020AAMC42 dupe [16:40:56] Please wait... [16:40:59] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:40:59] PROBLEM - cp30 Stunnel Http for mw10 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:41:07] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:41:19] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.43, 8.11, 8.00 [16:41:28] hi Voidwalker [16:41:32] PROBLEM - cp20 Stunnel Http for mw10 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:41:36] hi [16:42:19] Voidwalker: netsplit created 2 logbots [16:42:20] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:42:25] i just kicked one [16:42:32] ooh, fun [16:43:07] RECOVERY - cp31 Stunnel Http for mw10 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 7.969 second response time [16:43:07] RECOVERY - cp30 Stunnel Http for mw10 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 7.894 second response time [16:43:12] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:43:14] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 7.871 second response time [16:43:31] RECOVERY - cp20 Stunnel Http for mw10 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 3.449 second response time [16:44:09] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.59, 3.01, 3.38 [16:44:16] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.009 second response time [16:44:30] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:45:19] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 9.038 second response time [16:45:22] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:49:05] RhinosF1: no it didn't, one is from new infra one is from current infra. There has been 2 here for a few days I think. [16:49:09] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:49:25] Ah [16:49:40] that makes a bit more sense :) [16:49:45] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:51:05] Although paladox somehow stopped it from actually logging to SAL before I think. [16:51:12] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 6.973 second response time [16:51:15] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 16.90, 18.35, 20.30 [16:51:36] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 2.068 second response time [16:51:43] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 1.431 second response time [16:52:41] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:52:46] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 3.586 second response time [16:52:53] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:53:33] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:54:42] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.70, 3.19, 3.21 [16:54:45] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 8.742 second response time [16:54:57] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 7.846 second response time [16:55:38] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 7.910 second response time [16:56:36] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.09, 3.58, 3.36 [16:58:31] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.53, 3.50, 3.35 [16:58:34] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [17:00:26] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.66, 3.90, 3.51 [17:01:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [17:02:21] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.71, 3.87, 3.55 [17:04:15] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.24, 4.10, 3.68 [17:06:10] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.99, 3.98, 3.68 [17:06:29] PROBLEM - cp20 Stunnel Http for mw13 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:06:34] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2607:5300:201:3100::5ebc/cpweb [17:06:39] PROBLEM - cp30 Stunnel Http for mw13 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:06:54] PROBLEM - mw13 MediaWiki Rendering on mw13 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:07:14] PROBLEM - cp21 Stunnel Http for mw13 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:07:42] PROBLEM - cp31 Stunnel Http for mw13 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:08:05] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.12, 4.01, 3.72 [17:08:26] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 17.54, 18.98, 20.29 [17:09:19] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.13, 6.99, 7.98 [17:09:47] RECOVERY - cp31 Stunnel Http for mw13 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 8.301 second response time [17:10:34] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [17:10:36] RECOVERY - cp20 Stunnel Http for mw13 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 6.316 second response time [17:10:45] RECOVERY - cp30 Stunnel Http for mw13 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 6.031 second response time [17:11:04] RECOVERY - mw13 MediaWiki Rendering on mw13 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 8.373 second response time [17:11:19] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.04, 7.47, 8.04 [17:11:28] RECOVERY - cp21 Stunnel Http for mw13 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 6.227 second response time [17:11:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.40, 3.85, 3.73 [17:14:34] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::5ebc/cpweb [17:14:47] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:15:20] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.60, 7.38, 7.97 [17:15:28] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:16:29] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:18:56] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:19:17] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.26, 21.09, 20.61 [17:19:53] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 20.87, 20.63, 20.07 [17:19:54] PROBLEM - cp20 Stunnel Http for mw10 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:19:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.25, 3.26, 3.43 [17:19:58] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:20:01] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:20:01] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:20:08] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:20:08] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:20:26] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:20:35] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:20:38] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:20:45] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:21:01] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 9.097 second response time [17:21:55] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.11, 3.10, 3.35 [17:22:25] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 3.528 second response time [17:22:37] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 4.210 second response time [17:22:43] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 4.368 second response time [17:22:48] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 5.098 second response time [17:23:14] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 1.696 second response time [17:23:19] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.02, 7.79, 7.88 [17:23:43] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 19.03, 20.17, 20.02 [17:23:55] RECOVERY - cp20 Stunnel Http for mw10 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.057 second response time [17:23:57] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 0.928 second response time [17:24:07] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 7.826 second response time [17:24:12] RECOVERY - cp31 Stunnel Http for mw10 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.308 second response time [17:26:21] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 8.060 second response time [17:26:27] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 7.302 second response time [17:26:30] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 7.417 second response time [17:26:49] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [17:26:50] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 6.665 second response time [17:28:59] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 17.75, 19.90, 20.39 [17:30:39] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:30:43] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:30:53] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:30:54] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:31:12] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:32:34] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [17:32:36] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.631 second response time [17:32:42] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 1.197 second response time [17:32:51] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 1.232 second response time [17:32:53] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 1.004 second response time [17:32:53] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 19.50, 20.42, 20.54 [17:33:11] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 3.155 second response time [17:35:40] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2607:5300:201:3100::5ebc/cpweb [17:36:34] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 51.195.220.68/cpweb, 2607:5300:201:3100::5ebc/cpweb [17:38:34] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [17:42:16] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:42:34] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 4 datacenters are down: 2001:41d0:801:2000::1b80/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [17:42:35] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 19.46, 20.14, 20.38 [17:44:21] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:46:27] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 9.772 second response time [17:46:33] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 9.208 second response time [17:48:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.63, 3.56, 3.35 [17:49:26] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.06, 20.89, 20.65 [17:51:41] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:52:00] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:52:15] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:52:19] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:52:24] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:52:34] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:52:45] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.52, 3.23, 3.27 [17:52:56] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:53:00] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [17:56:09] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 3.449 second response time [17:56:22] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 1.716 second response time [17:56:34] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [17:56:41] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 1.749 second response time [17:56:49] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [17:58:28] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 3.011 second response time [17:58:42] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 1.568 second response time [17:59:07] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 2.121 second response time [17:59:15] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 2.458 second response time [18:00:17] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:00:24] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:00:34] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:00:45] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:01:03] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:01:24] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.67, 3.48, 3.37 [18:02:23] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.47, 7.08, 7.92 [18:02:49] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 149.56.140.43/cpweb [18:02:51] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:03:18] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.01, 3.34, 3.34 [18:03:26] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 5 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb [18:03:32] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:03:37] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:04:24] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 5.022 second response time [18:04:34] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 4.681 second response time [18:04:45] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 5.139 second response time [18:04:52] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 6.105 second response time [18:05:15] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 6.976 second response time [18:05:28] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 0.011 second response time [18:05:35] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 0.598 second response time [18:06:23] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 10.67, 8.17, 8.10 [18:06:34] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 0.323 second response time [18:06:53] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 0.009 second response time [18:08:49] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 16.73, 19.12, 20.14 [18:08:54] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:09:07] PROBLEM - cp30 Stunnel Http for mw13 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:09:11] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:09:13] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 20.83, 20.24, 19.55 [18:09:30] PROBLEM - mw13 MediaWiki Rendering on mw13 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:10:58] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.13, 4.03, 3.56 [18:12:49] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.075 second response time [18:12:53] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.40, 3.69, 3.49 [18:13:21] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 0.611 second response time [18:13:43] PROBLEM - cp21 Stunnel Http for mw13 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:13:56] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:14:22] PROBLEM - cp21 Stunnel Http for mw11 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:14:35] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:14:41] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.64, 20.77, 20.45 [18:14:48] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.82, 3.34, 3.38 [18:14:50] PROBLEM - cp30 Stunnel Http for mw11 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:14:54] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:15:01] PROBLEM - cp31 Stunnel Http for mw11 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:15:02] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:15:09] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:15:12] PROBLEM - cp20 Stunnel Http for mw11 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:15:14] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:15:17] PROBLEM - mw11 MediaWiki Rendering on mw11 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:15:35] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:15:58] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 4.202 second response time [18:16:11] PROBLEM - cp20 Stunnel Http for mw13 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:16:19] PROBLEM - cp31 Stunnel Http for mw13 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:17:19] RECOVERY - cp30 Stunnel Http for mw13 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 1.158 second response time [18:17:41] RECOVERY - cp21 Stunnel Http for mw13 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.043 second response time [18:17:44] RECOVERY - mw13 MediaWiki Rendering on mw13 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 0.802 second response time [18:18:11] RECOVERY - cp20 Stunnel Http for mw13 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 4.269 second response time [18:18:23] RECOVERY - cp31 Stunnel Http for mw13 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 7.046 second response time [18:21:22] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 0.333 second response time [18:22:45] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 9.294 second response time [18:23:32] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 9.181 second response time [18:23:34] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 7.350 second response time [18:23:57] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 2.206 second response time [18:25:05] PROBLEM - cp20 Stunnel Http for mw10 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:25:22] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:25:23] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 5.501 second response time [18:25:24] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:25:35] PROBLEM - cp30 Stunnel Http for mw10 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:25:39] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:26:22] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.33, 3.69, 3.53 [18:27:07] RECOVERY - cp20 Stunnel Http for mw10 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 6.888 second response time [18:27:19] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.91, 7.33, 7.91 [18:27:23] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 3.451 second response time [18:27:24] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 3.442 second response time [18:27:37] RECOVERY - cp30 Stunnel Http for mw10 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 1.858 second response time [18:27:40] RECOVERY - cp31 Stunnel Http for mw10 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 1.095 second response time [18:28:26] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 19.29, 19.99, 20.38 [18:28:49] [02miraheze/MirahezeMagic] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-1 [+0/-0/±1] 13https://git.io/J9c5b [18:28:50] [02miraheze/MirahezeMagic] 07RhinosF1 0332f540b - Fix VE [18:28:52] [02MirahezeMagic] 07RhinosF1 synchronize pull request 03#313: Recaptcha: use enterprise version - 13https://git.io/J9cPd [18:29:34] CosmicAlpha: ^ [18:29:42] Should be {{done}} [18:29:42] https://meta.miraheze.org/wiki/Template:done [18:29:53] RECOVERY - cp30 Stunnel Http for mw11 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 9.510 second response time [18:30:02] RECOVERY - cp20 Stunnel Http for mw11 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 8.816 second response time [18:30:13] RECOVERY - mw11 MediaWiki Rendering on mw11 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 7.193 second response time [18:31:19] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.88, 7.35, 7.71 [18:31:24] RECOVERY - cp21 Stunnel Http for mw11 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.015 second response time [18:31:31] RECOVERY - cp31 Stunnel Http for mw11 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.321 second response time [18:35:10] Thanks! [18:36:02] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:36:18] Np CosmicAlpha [18:36:20] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:36:26] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.33, 20.69, 20.60 [18:36:29] We just need Reception123 to sort the bank account [18:36:54] miraheze/MirahezeMagic - RhinosF1 the build passed. [18:37:03] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:37:19] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.24, 7.67, 7.79 [18:37:56] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.55, 4.01, 3.73 [18:37:57] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:37:57] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:38:39] PROBLEM - mw11 MediaWiki Rendering on mw11 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:39:13] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 18.66, 19.33, 20.26 [18:39:19] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.52, 8.15, 7.96 [18:39:46] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:39:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.71, 3.80, 3.69 [18:39:57] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:40:08] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:40:14] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:40:17] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:40:41] RECOVERY - mw11 MediaWiki Rendering on mw11 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 4.131 second response time [18:44:30] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.12, 22.31, 21.34 [18:46:26] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.26, 21.87, 21.31 [18:46:57] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 7.406 second response time [18:47:25] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 4.598 second response time [18:49:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.13, 3.55, 3.55 [18:50:26] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.72, 23.28, 21.99 [18:51:19] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:51:47] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:52:26] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.41, 22.68, 21.93 [18:53:26] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 8.896 second response time [18:53:51] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 8.554 second response time [18:53:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 1.99, 3.16, 3.42 [18:54:39] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 2.751 second response time [18:54:46] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 5.280 second response time [18:55:20] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23579 bytes in 7.599 second response time [18:55:55] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.36, 3.09, 3.38 [18:57:48] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:58:13] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:58:44] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 5.779 second response time [18:59:03] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:59:04] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 8.913 second response time [18:59:14] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 8.208 second response time [18:59:22] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 9.687 second response time [18:59:30] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 7.199 second response time [18:59:47] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 2.028 second response time [19:00:13] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 5.169 second response time [19:01:04] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 5.141 second response time [19:02:30] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 20.75, 19.84, 19.43 [19:04:04] PROBLEM - cp20 Stunnel Http for mw10 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:04:25] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 19.25, 19.70, 19.43 [19:04:44] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:04:57] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:05:00] PROBLEM - cp30 Stunnel Http for mw10 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:05:08] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:05:39] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:06:00] RECOVERY - cp20 Stunnel Http for mw10 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.167 second response time [19:06:39] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.62, 5.62, 4.84 [19:06:48] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 4.014 second response time [19:06:55] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 1.212 second response time [19:07:01] RECOVERY - cp30 Stunnel Http for mw10 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.326 second response time [19:07:08] RECOVERY - cp31 Stunnel Http for mw10 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.420 second response time [19:08:09] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:08:17] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:08:34] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [19:08:36] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.46, 5.11, 4.74 [19:08:39] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:08:42] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:08:46] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.18, 3.66, 3.46 [19:09:51] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 7.874 second response time [19:10:16] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 8.612 second response time [19:10:23] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 9.346 second response time [19:10:33] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 4.67, 5.07, 4.78 [19:10:35] [02miraheze/MirahezeMagic] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/J9cNq [19:10:37] [02miraheze/MirahezeMagic] 07Universal-Omega 036c18f81 - Update ReCaptchaNoCaptcha.php [19:10:38] [02MirahezeMagic] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://git.io/fQRGX [19:10:41] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.01, 3.37, 3.37 [19:10:41] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 7.378 second response time [19:10:45] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 7.200 second response time [19:10:46] [02MirahezeMagic] 07Universal-Omega opened pull request 03#314: Return user information - 13https://git.io/J9cN3 [19:11:20] [02miraheze/MirahezeMagic] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/J9cNG [19:11:22] [02miraheze/MirahezeMagic] 07Universal-Omega 03fc5b3f5 - Update en.json [19:11:23] [02MirahezeMagic] 07Universal-Omega synchronize pull request 03#314: Return user information - 13https://git.io/J9cN3 [19:12:14] RhinosF1: ^ just a concept, not great formatted error message. [19:12:34] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::1b80/cpweb [19:13:28] CosmicAlpha: don't you need to call the create Assesments api? [19:13:45] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:14:16] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:14:31] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.70, 3.62, 3.47 [19:14:40] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:15:02] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.06, 6.86, 7.71 [19:15:03] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:15:07] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:15:46] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 3.634 second response time [19:16:26] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 1.85, 2.96, 3.25 [19:16:52] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:19:03] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.52, 7.71, 7.85 [19:19:28] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.82, 1.22, 0.82 [19:20:10] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:20:53] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:21:05] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:21:09] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 9.782 second response time [19:21:15] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:21:26] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.82, 1.10, 0.82 [19:21:30] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:25:26] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 6.702 second response time [19:25:32] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:25:35] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:25:40] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 0.321 second response time [19:26:22] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 0.362 second response time [19:27:03] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 0.940 second response time [19:27:23] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23577 bytes in 1.718 second response time [19:27:37] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 2.432 second response time [19:28:04] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.04, 2.67, 1.46 [19:29:06] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 8.722 second response time [19:29:38] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 9.354 second response time [19:29:40] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 7.820 second response time [19:29:44] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 7.495 second response time [19:29:45] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 7.785 second response time [19:30:04] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 0.50, 1.88, 1.31 [19:30:49] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [19:32:04] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.89, 1.54, 1.26 [19:33:32] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:34:34] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [19:35:31] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.548 second response time [19:36:26] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 18.47, 19.61, 20.33 [19:41:21] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.38, 3.38, 3.09 [19:42:34] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 7 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [19:42:49] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 4 datacenters are down: 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [19:43:16] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.13, 3.16, 3.04 [19:44:34] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [19:44:49] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [19:47:07] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.03, 4.07, 3.42 [19:48:35] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 8 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [19:48:49] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 5 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::5ebc/cpweb [19:49:02] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.72, 3.83, 3.40 [19:50:57] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.25, 3.79, 3.42 [19:50:58] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:51:04] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:51:05] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:51:15] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:51:17] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:51:23] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.07, 21.05, 20.48 [19:51:32] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:51:32] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:51:38] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:51:55] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:52:50] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:52:52] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.95, 3.90, 3.51 [19:54:46] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.67, 4.18, 3.66 [19:54:54] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 7.650 second response time [19:55:15] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 4.163 second response time [19:55:16] [02miraheze/MirahezeMagic] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/J9cpa [19:55:17] [02miraheze/MirahezeMagic] 07Universal-Omega 036ce1476 - Update ReCaptchaNoCaptcha.php [19:55:19] [02MirahezeMagic] 07Universal-Omega synchronize pull request 03#314: Return user information - 13https://git.io/J9cN3 [19:55:27] RhinosF1: ^ [19:55:41] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 6.455 second response time [19:55:45] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 6.399 second response time [19:55:51] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 5.300 second response time [19:57:43] [02puppet] 07RhinosF1 opened pull request 03#2292: rdns: allow flat CNAME for dreamhost too - 13https://git.io/J9cp6 [19:57:55] paladox, Reception123: ^ [19:58:26] [02miraheze/MirahezeMagic] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/J9cpX [19:58:27] [02miraheze/MirahezeMagic] 07Universal-Omega 03ef3086d - Update extension.json [19:58:29] [02MirahezeMagic] 07Universal-Omega synchronize pull request 03#314: Return user information - 13https://git.io/J9cN3 [19:58:30] [02puppet] 07paladox closed pull request 03#2292: rdns: allow flat CNAME for dreamhost too - 13https://git.io/J9cp6 [19:58:32] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/J9cp1 [19:58:33] [02miraheze/puppet] 07RhinosF1 03cc501f4 - rdns: allow flat CNAME for dreamhost too (#2292) [19:58:36] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.47, 3.60, 3.56 [19:58:37] CosmicAlpha: thanks [19:59:17] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:59:19] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 9.373 second response time [19:59:26] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 7.434 second response time [19:59:34] [02MirahezeMagic] 07Universal-Omega edited pull request 03#314: Use enterprise assessment - 13https://git.io/J9cN3 [19:59:38] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:59:47] No problem RhinosF1! [19:59:49] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 9.950 second response time [20:00:05] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:00:12] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:00:21] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:00:40] RhinosF1: I wasn't sure is ?key in enterprise secret key or a new key? [20:01:01] CosmicAlpha: I think that one is unique to the individual [20:01:50] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 7.333 second response time [20:02:13] No [20:02:20] RhinosF1: its something you have to specify in post request: "API key associated with the current project" but I wasn't sure if it was secret key or one unique to enterprise? [20:02:23] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 2.905 second response time [20:02:24] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 4.72, 6.56, 7.67 [20:02:34] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [20:02:37] CosmicAlpha: unique to enterprise [20:02:51] Set it as a config variable [20:02:55] Alright good so I did that right then. [20:03:02] I can get someone to add it to PS.php [20:03:05] When I make it [20:03:10] Yeah it is a configuration variable. [20:03:26] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 8.063 second response time [20:03:32] But it should be a public key, not in PS, or is it a secret key for enterprises? [20:03:47] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 4.643 second response time [20:04:08] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 0.038 second response time [20:04:08] CosmicAlpha: secret [20:04:13] It's like a login [20:04:20] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.31, 3.56, 3.51 [20:04:20] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 0.660 second response time [20:04:30] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 1.032 second response time [20:04:50] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [20:04:58] Ah alright, so it just replaces $wgReCaptchaSecretKey then so we can just use that variable still but change it, as that currently would be unused now I think. [20:06:15] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.57, 3.21, 3.39 [20:06:17] It's easier then just messing with PS stuff. [20:06:23] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 11.34, 8.51, 8.12 [20:06:25] CosmicAlpha: yes [20:06:38] [02miraheze/MirahezeMagic] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/J9chE [20:06:40] [02miraheze/MirahezeMagic] 07Universal-Omega 03e8c2444 - Update ReCaptchaNoCaptcha.php [20:06:41] [02MirahezeMagic] 07Universal-Omega synchronize pull request 03#314: Use enterprise assessment - 13https://git.io/J9cN3 [20:06:53] [02miraheze/MirahezeMagic] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/J9chz [20:06:55] [02miraheze/MirahezeMagic] 07Universal-Omega 03d7a2912 - Update extension.json [20:06:56] [02MirahezeMagic] 07Universal-Omega synchronize pull request 03#314: Use enterprise assessment - 13https://git.io/J9cN3 [20:08:09] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:08:19] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:08:34] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 198.244.148.90/cpweb [20:08:47] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:08:49] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:08:53] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:10:38] [02miraheze/MirahezeMagic] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/J9chy [20:10:40] [02miraheze/MirahezeMagic] 07Universal-Omega 03eb532b8 - Update ReCaptchaNoCaptcha.php [20:10:41] [02MirahezeMagic] 07Universal-Omega synchronize pull request 03#314: Use enterprise assessment - 13https://git.io/J9cN3 [20:11:50] RhinosF1: should be good I think. [20:12:05] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 21.24, 20.23, 19.59 [20:12:44] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 26.46, 22.43, 21.31 [20:14:20] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 7.878 second response time [20:14:34] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 0.338 second response time [20:14:41] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.62, 21.98, 21.27 [20:14:44] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.008 second response time [20:14:53] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.28, 5.18, 4.54 [20:15:04] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 1.417 second response time [20:15:06] PROBLEM - cp20 Stunnel Http for mw10 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:15:06] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:15:09] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 2.345 second response time [20:15:33] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:15:33] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:15:40] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:15:41] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 4 datacenters are down: 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::5ebc/cpweb [20:15:43] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:15:44] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:15:56] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 19.24, 19.99, 19.67 [20:16:00] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:16:21] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:16:51] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.65, 4.65, 4.43 [20:17:09] RECOVERY - cp20 Stunnel Http for mw10 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 7.324 second response time [20:17:15] PROBLEM - cp30 Stunnel Http for mw13 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:17:32] PROBLEM - mw13 MediaWiki Rendering on mw13 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:17:35] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 5.916 second response time [20:17:39] RECOVERY - cp31 Stunnel Http for mw10 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 5.435 second response time [20:17:49] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 6.236 second response time [20:19:11] RECOVERY - cp30 Stunnel Http for mw13 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.348 second response time [20:19:29] RECOVERY - mw13 MediaWiki Rendering on mw13 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 0.843 second response time [20:20:09] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 1.258 second response time [20:20:26] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 0.323 second response time [20:21:23] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 0.321 second response time [20:21:33] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:21:51] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 2.862 second response time [20:21:54] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 3.422 second response time [20:21:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.89, 3.71, 3.39 [20:22:40] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:23:03] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:23:04] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:23:20] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:30:26] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 19.36, 19.54, 20.23 [20:30:34] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [20:31:00] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [20:31:24] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 9.164 second response time [20:31:41] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 8.190 second response time [20:31:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.26, 3.38, 3.45 [20:32:05] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 8.629 second response time [20:33:08] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.008 second response time [20:33:35] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 1.713 second response time [20:34:26] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.18, 20.70, 20.54 [20:36:49] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb [20:37:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.35, 3.71, 3.55 [20:38:26] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 19.78, 20.23, 20.39 [20:38:34] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 149.56.141.75/cpweb, 2607:5300:201:3100::5ebc/cpweb [20:41:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.99, 3.42, 3.47 [20:42:27] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.20, 20.99, 20.61 [20:42:56] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:43:44] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:43:50] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:43:55] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.07, 3.02, 3.32 [20:44:09] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 1595 bytes in 0.008 second response time [20:44:52] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.447 second response time [20:45:14] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:45:18] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:45:19] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:45:19] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:45:41] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.029 second response time [20:45:47] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.665 second response time [20:46:18] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 9.725 second response time [20:47:24] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 8.719 second response time [20:47:24] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 8.764 second response time [20:47:28] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 9.632 second response time [20:50:04] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:50:15] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:50:35] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:50:37] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:50:48] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:51:13] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:51:47] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:51:48] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:51:50] [02puppet] 07Universal-Omega opened pull request 03#2293: CSP: Add recaptchaenterprise.googleapis.com - 13https://git.io/J9CvM [20:51:58] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:53:16] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 6.872 second response time [20:56:40] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [20:57:38] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:58:20] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 5.573 second response time [20:58:55] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 6.544 second response time [20:59:02] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23579 bytes in 3.015 second response time [20:59:17] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 4.248 second response time [20:59:33] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.315 second response time [20:59:42] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 9.467 second response time [21:00:42] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 7.742 second response time [21:00:46] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 6.907 second response time [21:02:19] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 9.924 second response time [21:02:23] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 9.029 second response time [21:03:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:03:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.83, 3.97, 3.46 [21:04:26] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 19.00, 19.74, 20.26 [21:05:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.75, 3.88, 3.49 [21:06:34] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [21:06:49] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [21:08:58] [02puppet] 07RhinosF1 opened pull request 03#2294: rdns: fix dreamhost rules - 13https://git.io/J9CJG [21:09:07] paladox: ^ [21:09:10] That + CSP [21:09:21] [02puppet] 07paladox closed pull request 03#2294: rdns: fix dreamhost rules - 13https://git.io/J9CJG [21:09:22] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/J9CJW [21:09:24] [02miraheze/puppet] 07RhinosF1 034a4d018 - fix dreamhost rules (#2294) [21:09:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.31, 4.03, 3.64 [21:10:19] Ty paladox [21:10:34] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [21:10:49] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 149.56.140.43/cpweb, 149.56.141.75/cpweb [21:11:23] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:11:32] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:11:40] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:11:45] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:12:45] paladox: can you also merge https://github.com/miraheze/puppet/pull/2293 please? [21:12:45] [url] CSP: Add recaptchaenterprise.googleapis.com by Universal-Omega · Pull Request #2293 · miraheze/puppet · GitHub | github.com [21:13:06] has that been approved? [21:13:26] Google domains have been. [21:13:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.08, 3.53, 3.57 [21:14:10] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:14:26] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.26, 20.65, 20.38 [21:14:42] [02puppet] 07paladox closed pull request 03#2293: CSP: Add recaptchaenterprise.googleapis.com - 13https://git.io/J9CvM [21:14:43] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/J9CJy [21:14:44] ok [21:14:45] [02miraheze/puppet] 07Universal-Omega 038f14e1b - CSP: Add recaptchaenterprise.googleapis.com (#2293) [21:14:50] Thanks! [21:16:26] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 19.95, 20.33, 20.29 [21:17:52] [02MatomoAnalytics] 07Universal-Omega closed pull request 03#66: Remove dummyProcess - 13https://git.io/JDulj [21:17:53] [02miraheze/MatomoAnalytics] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/J9CJA [21:17:55] [02miraheze/MatomoAnalytics] 07Universal-Omega 037bb72ad - Remove dummyProcess (#66) [21:17:56] [02miraheze/MatomoAnalytics] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [21:17:58] [02MatomoAnalytics] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://git.io/fN4LT [21:18:34] PROBLEM - cp30 Stunnel Http for mw13 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:19:02] PROBLEM - cp31 Stunnel Http for mw13 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:19:24] PROBLEM - cp21 Stunnel Http for mw13 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:19:33] PROBLEM - mw13 MediaWiki Rendering on mw13 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:19:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.07, 3.52, 3.52 [21:20:19] PROBLEM - cp20 Stunnel Http for mw13 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:20:26] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.73, 20.77, 20.49 [21:21:03] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:21:09] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:21:11] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:21:11] PROBLEM - cp20 Stunnel Http for mw10 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:21:25] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:21:33] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:21:35] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:21:40] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.025 second response time [21:21:53] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:21:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.16, 3.41, 3.48 [21:21:58] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 1.611 second response time [21:22:01] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 1.467 second response time [21:22:15] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:22:41] RECOVERY - cp30 Stunnel Http for mw13 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 5.637 second response time [21:23:11] miraheze/MatomoAnalytics - Universal-Omega the build passed. [21:23:13] PROBLEM - cp30 Stunnel Http for mw10 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:23:13] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 3.619 second response time [21:23:37] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 5.659 second response time [21:23:40] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 6.455 second response time [21:23:44] PROBLEM - cp30 Stunnel Http for mw11 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:23:56] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.10, 3.05, 3.35 [21:23:59] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 9.721 second response time [21:25:50] RECOVERY - cp30 Stunnel Http for mw11 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 6.683 second response time [21:26:02] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:26:24] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:26:26] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:26:51] RECOVERY - housing.wiki - reverse DNS on sslhost is OK: SSL OK - housing.wiki reverse DNS resolves to cp31.miraheze.org - CNAME FLAT [21:27:03] PROBLEM - cp30 Stunnel Http for mw13 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:27:29] RECOVERY - cp20 Stunnel Http for mw10 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 9.567 second response time [21:27:31] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 9.839 second response time [21:27:33] RECOVERY - cp30 Stunnel Http for mw10 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 8.090 second response time [21:27:43] RECOVERY - cp31 Stunnel Http for mw10 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 8.401 second response time [21:27:44] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 8.066 second response time [21:28:27] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 2.258 second response time [21:28:35] RECOVERY - cp20 Stunnel Http for mw13 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 4.101 second response time [21:28:59] RECOVERY - housing.wiki - reverse DNS on sslhost is OK: /usr/lib/nagios/plugins/check_reverse_dns.py:101: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead resolved_ip_addr = str(dns_resolver.query(hostname, 'AAAA')[0])/usr/lib/nagios/plugins/check_reverse_dns.py:103: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead rev_host = str(dns_resolver.query(ptr_record, "PTR")[0]).rstrip('.')/usr/lib/nagios [21:28:59] ns/check_reverse_dns.py:66: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead nameserversans = dns_resolver.query(root_domain, 'NS')/usr/lib/nagios/plugins/check_reverse_dns.py:79: DeprecationWarning: please use dns.resolver.Resolver.resolve() instead cname = str(dns_resolver.query(hostname, 'CNAME')[0])SSL OK - housing.wiki reverse DNS resolves to cp20.miraheze.org - CNAME FLAT [21:29:07] RECOVERY - cp30 Stunnel Http for mw13 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 9.132 second response time [21:29:28] RECOVERY - cp31 Stunnel Http for mw13 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 4.097 second response time [21:29:56] RECOVERY - mw13 MediaWiki Rendering on mw13 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 3.711 second response time [21:30:04] RECOVERY - cp21 Stunnel Http for mw13 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 4.783 second response time [21:30:07] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.034 second response time [21:30:26] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 18.35, 19.82, 20.33 [21:30:28] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.355 second response time [21:30:28] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.327 second response time [21:30:48] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 0.312 second response time [21:30:49] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.007 second response time [21:31:53] PROBLEM - cp20 Stunnel Http for mw10 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:32:00] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:32:06] PROBLEM - cp30 Stunnel Http for mw10 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:32:10] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:32:17] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:33:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.41, 3.32, 3.31 [21:33:58] RECOVERY - cp20 Stunnel Http for mw10 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 9.015 second response time [21:34:09] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 9.161 second response time [21:34:15] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 8.877 second response time [21:34:16] RECOVERY - cp30 Stunnel Http for mw10 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 9.212 second response time [21:35:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.06, 3.55, 3.40 [21:36:34] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [21:37:13] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 20.83, 20.43, 19.80 [21:37:32] PROBLEM - cp30 Stunnel Http for mw13 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:37:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.26, 3.64, 3.46 [21:38:37] RECOVERY - cp31 Stunnel Http for mw10 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.310 second response time [21:39:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.46, 3.97, 3.60 [21:40:26] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.60, 20.55, 20.08 [21:40:34] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 4 datacenters are down: 51.195.220.68/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb [21:41:36] RECOVERY - cp30 Stunnel Http for mw13 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 3.434 second response time [21:43:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.07, 3.92, 3.70 [21:44:06] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.025 second response time [21:44:42] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:45:13] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 18.53, 20.13, 19.99 [21:45:19] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:45:25] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:45:42] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:45:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.06, 4.03, 3.77 [21:47:11] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:47:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.51, 3.46, 3.60 [21:48:41] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:48:49] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:48:57] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:49:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.20, 3.70, 3.66 [21:50:37] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.329 second response time [21:50:47] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.309 second response time [21:50:53] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.013 second response time [21:51:18] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 1.236 second response time [21:51:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.15, 3.03, 3.41 [21:52:37] PROBLEM - mw13 MediaWiki Rendering on mw13 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:52:44] PROBLEM - cp20 Stunnel Http for mw13 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:52:47] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 0.80, 1.95, 1.32 [21:54:36] RECOVERY - mw13 MediaWiki Rendering on mw13 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 3.182 second response time [21:54:41] RECOVERY - cp20 Stunnel Http for mw13 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 1.691 second response time [21:54:45] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.69, 1.52, 1.24 [21:55:00] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:55:13] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:55:15] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:55:42] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:55:55] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.74, 3.13, 3.38 [21:56:23] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.82, 7.18, 7.96 [21:56:51] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:59:18] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 6.429 second response time [22:00:25] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.38, 7.12, 7.69 [22:00:55] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 7.897 second response time [22:01:51] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.88, 5.25, 4.56 [22:02:22] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 3.974 second response time [22:02:24] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.95, 7.37, 7.71 [22:02:35] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 8.261 second response time [22:03:05] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.515 second response time [22:03:13] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.340 second response time [22:03:27] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.008 second response time [22:03:31] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.565 second response time [22:03:40] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 20.51, 20.38, 20.10 [22:03:48] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.53, 5.63, 4.78 [22:04:11] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 7.262 second response time [22:04:23] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.51, 7.90, 7.85 [22:04:41] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 1.008 second response time [22:04:49] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [22:05:35] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 20.22, 20.09, 20.01 [22:05:45] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.19, 5.27, 4.74 [22:07:42] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.79, 4.72, 4.60 [22:08:34] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [22:13:56] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:14:36] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:14:43] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.33, 4.02, 3.53 [22:15:28] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:15:32] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:15:41] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:16:34] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 51.195.220.68/cpweb, 2001:41d0:801:2000::4c25/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [22:16:38] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.54, 3.87, 3.54 [22:17:24] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 5 datacenters are down: 51.195.220.68/cpweb, 2001:41d0:801:2000::4c25/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [22:18:33] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.15, 4.06, 3.65 [22:22:23] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.50, 3.75, 3.62 [22:22:52] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:23:03] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:23:08] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:23:17] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:23:28] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:25:32] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 7.971 second response time [22:29:16] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 8.046 second response time [22:29:25] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 7.860 second response time [22:29:26] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 8.115 second response time [22:29:46] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 9.305 second response time [22:30:02] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.12, 3.10, 3.36 [22:30:11] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 9.787 second response time [22:30:18] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 8.607 second response time [22:30:20] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 7.566 second response time [22:30:46] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 6.426 second response time [22:31:37] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 4.041 second response time [22:33:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.12, 3.72, 3.56 [22:34:34] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [22:34:49] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [22:37:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.83, 3.56, 3.57 [22:38:34] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 5 datacenters are down: 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::5ebc/cpweb [22:38:49] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 4 datacenters are down: 198.244.148.90/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [22:43:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.41, 3.61, 3.56 [22:45:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.36, 3.59, 3.56 [22:49:00] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:49:00] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:49:04] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:49:26] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:49:31] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:49:39] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.74, 5.47, 4.61 [22:50:18] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:50:39] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:50:58] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 3.456 second response time [22:50:59] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 3.831 second response time [22:51:04] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 2.113 second response time [22:51:24] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.394 second response time [22:51:30] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 0.294 second response time [22:51:36] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 4.12, 5.09, 4.58 [22:52:16] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 0.355 second response time [22:52:39] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 0.322 second response time [22:53:55] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 1.90, 2.98, 3.32 [22:58:26] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.39, 21.61, 21.21 [22:59:58] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:00:02] !log [@test101] starting deploy of {'l10nupdate': True} to skip [23:00:03] !log [@mwtask111] starting deploy of {'l10nupdate': True} to scsvg [23:00:04] !log [@test3] starting deploy of {'l10nupdate': True} to skip [23:00:05] !log [@mw11] starting deploy of {'l10nupdate': True} to ovlon [23:00:26] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.47, 21.45, 21.19 [23:00:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:01:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:01:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:01:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:02:02] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23579 bytes in 6.037 second response time [23:06:26] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:07:40] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 21.54, 20.26, 19.58 [23:07:45] PROBLEM - cp20 Stunnel Http for mw13 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:08:29] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 4.945 second response time [23:08:44] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.023 second response time [23:09:04] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:09:05] PROBLEM - cp20 Stunnel Http for mw10 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:09:15] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:09:24] PROBLEM - cp30 Stunnel Http for mw10 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:09:35] PROBLEM - cloud5 Current Load on cloud5 is CRITICAL: CRITICAL - load average: 24.28, 21.48, 20.10 [23:10:40] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.021 second response time [23:10:54] PROBLEM - test3 Current Load on test3 is CRITICAL: CRITICAL - load average: 12.45, 6.22, 2.61 [23:11:00] RECOVERY - cp20 Stunnel Http for mw10 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.009 second response time [23:11:04] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 1.202 second response time [23:11:15] PROBLEM - mw13 MediaWiki Rendering on mw13 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 1595 bytes in 0.082 second response time [23:11:16] PROBLEM - test3 APT on test3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:11:16] PROBLEM - cp31 Stunnel Http for mw13 on cp31 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.258 second response time [23:11:20] RECOVERY - cp31 Stunnel Http for mw10 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 4.761 second response time [23:11:28] RECOVERY - cp30 Stunnel Http for mw10 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 3.735 second response time [23:11:34] PROBLEM - cp30 Stunnel Http for mw13 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:12:07] PROBLEM - cp20 Stunnel Http for mw11 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:12:21] PROBLEM - cp31 Stunnel Http for mw11 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:12:53] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:13:11] RECOVERY - mw13 MediaWiki Rendering on mw13 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 0.379 second response time [23:13:13] RECOVERY - cp31 Stunnel Http for mw13 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.321 second response time [23:13:14] RECOVERY - test3 APT on test3 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [23:13:14] PROBLEM - mw11 MediaWiki Rendering on mw11 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:13:18] PROBLEM - cp30 Stunnel Http for mw11 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:13:31] RECOVERY - cp30 Stunnel Http for mw13 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.651 second response time [23:13:33] PROBLEM - cp21 Stunnel Http for mw11 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:13:52] RECOVERY - cp20 Stunnel Http for mw13 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 1.253 second response time [23:14:05] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:14:45] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:15:01] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:15:01] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:15:10] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:15:21] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 19.41, 22.04, 21.04 [23:15:25] !log [@test101] finished deploy of {'l10nupdate': True} to skip - SUCCESS in 923s [23:15:31] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:15:53] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:16:00] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:16:12] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 9.662 second response time [23:16:16] RECOVERY - cp20 Stunnel Http for mw11 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 5.151 second response time [23:16:26] RECOVERY - cp31 Stunnel Http for mw11 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 5.244 second response time [23:16:30] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:16:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:16:54] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 9.801 second response time [23:17:11] !log [@mwtask111] finished deploy of {'l10nupdate': True} to scsvg - SUCCESS in 1028s [23:17:28] RECOVERY - mw11 MediaWiki Rendering on mw11 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 7.012 second response time [23:17:33] RECOVERY - cp30 Stunnel Http for mw11 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 7.064 second response time [23:17:46] RECOVERY - cp21 Stunnel Http for mw11 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 4.720 second response time [23:18:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:19:00] RECOVERY - test3 Current Load on test3 is OK: OK - load average: 0.37, 2.85, 2.81 [23:20:34] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:21:20] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:22:26] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 26.85, 23.48, 21.87 [23:23:45] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 9.293 second response time [23:24:40] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 0.629 second response time [23:25:23] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 4.475 second response time [23:25:27] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14570 bytes in 5.076 second response time [23:25:30] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 3.092 second response time [23:25:59] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 9.677 second response time [23:26:17] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 2.955 second response time [23:26:26] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.16, 23.40, 22.21 [23:26:30] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 1.179 second response time [23:26:45] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.23, 3.52, 3.27 [23:26:59] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 1.900 second response time [23:27:16] PROBLEM - cloud5 Current Load on cloud5 is CRITICAL: CRITICAL - load average: 26.92, 23.54, 21.84 [23:27:50] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 2.931 second response time [23:29:00] PROBLEM - test3 Current Load on test3 is CRITICAL: CRITICAL - load average: 5.97, 4.12, 3.11 [23:29:14] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 23.97, 23.82, 22.17 [23:29:30] PROBLEM - cp31 Stunnel Http for mw13 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:29:32] PROBLEM - test3 APT on test3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:29:36] PROBLEM - mw13 MediaWiki Rendering on mw13 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:29:41] PROBLEM - cp30 Stunnel Http for mw13 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:30:05] PROBLEM - cp21 Stunnel Http for mw13 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:30:07] PROBLEM - cp20 Stunnel Http for mw13 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:31:29] RECOVERY - test3 APT on test3 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [23:32:11] RECOVERY - cp20 Stunnel Http for mw13 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 8.261 second response time [23:32:13] RECOVERY - cp21 Stunnel Http for mw13 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 8.180 second response time [23:32:49] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [23:32:59] PROBLEM - test3 Current Load on test3 is WARNING: WARNING - load average: 1.10, 3.51, 3.20 [23:33:35] RECOVERY - cp31 Stunnel Http for mw13 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 3.130 second response time [23:33:40] RECOVERY - mw13 MediaWiki Rendering on mw13 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 2.340 second response time [23:33:42] RECOVERY - cp30 Stunnel Http for mw13 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.322 second response time [23:34:34] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [23:34:58] RECOVERY - test3 Current Load on test3 is OK: OK - load average: 0.45, 2.48, 2.86 [23:38:34] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2607:5300:201:3100::929a/cpweb [23:38:49] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 3 datacenters are down: 51.195.220.68/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb [23:42:05] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.61, 3.80, 3.77 [23:42:37] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:42:42] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:43:32] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:43:55] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:44:20] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:44:34] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [23:44:40] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 7.719 second response time [23:44:42] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 3.137 second response time [23:44:49] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [23:45:29] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.314 second response time [23:45:51] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 0.033 second response time [23:46:19] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 1.788 second response time [23:50:27] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:50:34] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 2607:5300:201:3100::929a/cpweb [23:50:49] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 149.56.141.75/cpweb [23:50:57] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:50:59] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:51:02] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:51:13] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 17.87, 19.05, 20.33 [23:51:21] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:51:32] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:52:00] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:52:16] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:52:17] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:52:50] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:53:04] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 9.066 second response time [23:53:07] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 8.458 second response time [23:54:00] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14571 bytes in 3.785 second response time [23:54:19] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 5.595 second response time [23:54:53] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 5.539 second response time [23:55:44] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 9.623 second response time [23:55:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.20, 3.64, 3.56 [23:56:24] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 23576 bytes in 1.040 second response time [23:56:47] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 3.020 second response time [23:57:14] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14564 bytes in 3.249 second response time [23:57:20] PROBLEM - cp30 Stunnel Http for mw10 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:57:30] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 1.782 second response time [23:57:35] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:57:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.91, 3.58, 3.54 [23:59:21] RECOVERY - cp30 Stunnel Http for mw10 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.403 second response time [23:59:37] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 23578 bytes in 1.783 second response time