[00:00:08] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.72, 4.24, 3.66 [00:00:38] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.36, 1.65, 1.68 [00:00:55] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.25, 4.19, 3.69 [00:00:56] PROBLEM - mwtask111 APT on mwtask111 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [00:02:55] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.90, 3.93, 3.65 [00:03:01] RECOVERY - mwtask111 APT on mwtask111 is OK: APT OK: 58 packages available for upgrade (0 critical updates). [00:04:07] PROBLEM - mw101 Puppet on mw101 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:04:31] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.17, 3.49, 2.56 [00:04:55] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.12, 3.89, 3.66 [00:06:06] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.72, 3.62, 3.60 [00:08:55] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.83, 3.55, 3.58 [00:10:18] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 0.59, 1.50, 1.94 [00:10:55] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.81, 3.74, 3.63 [00:12:55] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.45, 3.19, 3.44 [00:14:05] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.07, 2.76, 3.26 [00:14:09] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.14, 1.58, 1.85 [00:14:55] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 3.20, 3.09, 3.36 [00:15:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:16:05] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.13, 1.39, 1.75 [00:18:14] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 0.56, 0.99, 1.97 [00:19:56] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.81, 1.28, 1.64 [00:24:21] PROBLEM - cp30 Stunnel HTTP for mw122 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:26:19] RECOVERY - cp30 Stunnel HTTP for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.308 second response time [00:27:33] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.39, 3.40, 3.96 [00:27:42] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 1.26, 2.04, 1.90 [00:28:14] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.28, 1.29, 1.68 [00:31:06] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.10, 6.97, 5.96 [00:31:38] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 0.92, 1.67, 1.81 [00:32:07] RECOVERY - mw101 Puppet on mw101 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [00:33:05] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 4.68, 6.18, 5.80 [00:35:38] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.88, 1.33, 1.64 [00:39:17] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.78, 4.01, 3.93 [00:41:59] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.31, 6.88, 6.22 [00:43:59] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 5.50, 6.29, 6.08 [00:45:53] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.97, 3.53, 3.15 [00:47:10] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.80, 3.95, 3.99 [00:47:51] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.57, 3.36, 3.05 [00:47:52] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.57, 3.16, 3.06 [00:49:09] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 3.54, 3.99, 4.01 [00:49:46] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.97, 3.33, 3.07 [00:51:09] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 1.72, 3.37, 3.79 [00:51:17] [02mw-config] 07AgentIsai opened pull request 03#4547: T9025: Disable Cosmos rail for shrqwiki + alphabetize Cosmos settings - 13https://github.com/miraheze/mw-config/pull/4547 [00:51:54] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 9.72, 7.58, 6.74 [00:52:24] miraheze/mw-config - AgentIsai the build passed. [00:53:53] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 5.82, 6.81, 6.56 [00:57:48] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.09, 3.16, 2.94 [00:57:51] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.13, 7.34, 6.80 [00:59:09] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.45, 3.96, 3.75 [00:59:47] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.86, 3.07, 2.94 [00:59:51] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 5.34, 6.62, 6.61 [01:01:18] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.27, 3.78, 3.27 [01:03:13] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.78, 3.78, 3.33 [01:08:59] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.10, 3.05, 3.19 [01:11:48] PROBLEM - cloud10 Puppet on cloud10 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [01:14:43] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.75, 6.89, 6.59 [01:16:42] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 6.50, 6.43, 6.45 [01:17:09] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.80, 3.26, 3.97 [01:24:14] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.04, 1.59, 1.14 [01:25:09] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.08, 3.71, 3.80 [01:26:14] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 0.98, 1.28, 1.07 [01:28:34] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.64, 3.40, 2.91 [01:29:09] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.00, 3.36, 3.64 [01:30:33] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.96, 3.23, 2.91 [01:33:09] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.22, 2.86, 3.37 [01:39:44] RECOVERY - cloud10 Puppet on cloud10 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [01:41:39] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.50, 6.97, 6.24 [01:43:38] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 5.16, 6.46, 6.15 [01:47:36] [02mw-config] 07Universal-Omega reviewed pull request 03#4547 commit - 13https://github.com/miraheze/mw-config/pull/4547#discussion_r843392254 [01:49:34] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.56, 6.98, 6.41 [01:50:32] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 6.35, 2.42, 1.46 [01:51:12] [02mw-config] 07Universal-Omega reviewed pull request 03#4547 commit - 13https://github.com/miraheze/mw-config/pull/4547#discussion_r843392254 [01:51:33] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 5.17, 6.49, 6.31 [01:52:27] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.76, 1.89, 1.38 [01:54:26] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 5.30, 2.47, 1.61 [01:55:31] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.73, 6.95, 6.56 [01:56:25] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.26, 1.84, 1.48 [01:58:25] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.44, 1.56, 1.41 [02:01:28] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.80, 7.48, 6.86 [02:05:26] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 5.25, 7.02, 6.87 [02:07:26] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.66, 7.45, 7.03 [02:09:25] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 5.89, 6.82, 6.85 [02:10:40] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.91, 1.56, 1.21 [02:11:24] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 4.89, 6.11, 6.58 [02:12:36] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.28, 1.44, 1.20 [02:13:15] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.41, 3.09, 2.78 [02:17:12] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.16, 3.81, 3.35 [02:17:13] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.44, 3.01, 2.84 [02:17:37] [02mw-config] 07AgentIsai opened pull request 03#4548: T9025: Add wgCosmosRailBlacklistedNamespaces to ManageWiki/namespaces - 13https://github.com/miraheze/mw-config/pull/4548 [02:18:47] miraheze/mw-config - AgentIsai the build passed. [02:19:02] [02mw-config] 07AgentIsai synchronize pull request 03#4547: T9025: Disable Cosmos rail for shrqwiki + alphabetize Cosmos settings - 13https://github.com/miraheze/mw-config/pull/4547 [02:19:09] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.09, 3.29, 3.22 [02:20:03] miraheze/mw-config - AgentIsai the build passed. [02:20:56] [02mw-config] 07AgentIsai synchronize pull request 03#4547: T9025: Disable Cosmos rail for shrqwiki + alphabetize Cosmos settings - 13https://github.com/miraheze/mw-config/pull/4547 [02:22:02] miraheze/mw-config - AgentIsai the build passed. [02:23:09] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.28, 3.66, 3.36 [02:23:54] [02mw-config] 07AgentIsai reviewed pull request 03#4547 commit - 13https://github.com/miraheze/mw-config/pull/4547#discussion_r843405028 [02:24:09] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.69, 3.51, 3.14 [02:24:14] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.94, 2.16, 1.58 [02:24:16] PROBLEM - mwtask111 APT on mwtask111 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [02:25:07] [02mw-config] 07AgentIsai edited pull request 03#4547: T9025: Disable Cosmos rail for shrqwiki + alphabetize Cosmos settings - 13https://github.com/miraheze/mw-config/pull/4547 [02:25:09] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.45, 3.48, 3.32 [02:25:51] [02mw-config] 07AgentIsai edited pull request 03#4547: T9025:Set wgCosmosRailBlacklistedNamespaces + alphabetize Cosmos settings - 13https://github.com/miraheze/mw-config/pull/4547 [02:26:08] [02mw-config] 07AgentIsai edited pull request 03#4547: T9025: Set wgCosmosRailBlacklistedNamespaces + alphabetize Cosmos settings - 13https://github.com/miraheze/mw-config/pull/4547 [02:26:22] RECOVERY - mwtask111 APT on mwtask111 is OK: APT OK: 58 packages available for upgrade (0 critical updates). [02:27:09] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.44, 3.71, 3.42 [02:27:47] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.08, 3.53, 2.92 [02:28:08] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.75, 4.14, 3.45 [02:29:09] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.00, 3.46, 3.36 [02:29:24] [02mw-config] 07Universal-Omega reviewed pull request 03#4548 commit - 13https://github.com/miraheze/mw-config/pull/4548#discussion_r843405160 [02:29:25] [02mw-config] 07Universal-Omega reviewed pull request 03#4548 commit - 13https://github.com/miraheze/mw-config/pull/4548#discussion_r843406510 [02:29:42] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.73, 3.17, 2.86 [02:30:07] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.90, 3.85, 3.42 [02:30:18] [02mw-config] 07Universal-Omega reviewed pull request 03#4547 commit - 13https://github.com/miraheze/mw-config/pull/4547#discussion_r843407017 [02:31:09] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.89, 3.18, 3.27 [02:32:06] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.47, 3.87, 3.46 [02:34:14] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.92, 1.91, 1.78 [02:34:19] [02mw-config] 07Universal-Omega reviewed pull request 03#4548 commit - 13https://github.com/miraheze/mw-config/pull/4548#discussion_r843406510 [02:37:13] [02mw-config] 07AgentIsai synchronize pull request 03#4548: T9025: Add wgCosmosRailBlacklistedNamespaces to ManageWiki/namespaces - 13https://github.com/miraheze/mw-config/pull/4548 [02:37:40] [02mw-config] 07AgentIsai reviewed pull request 03#4548 commit - 13https://github.com/miraheze/mw-config/pull/4548#discussion_r843409586 [02:37:55] [02mw-config] 07AgentIsai reviewed pull request 03#4548 commit - 13https://github.com/miraheze/mw-config/pull/4548#discussion_r843409671 [02:38:05] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.42, 3.22, 3.33 [02:38:14] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.24, 1.89, 1.79 [02:38:25] miraheze/mw-config - AgentIsai the build passed. [02:39:26] [02mw-config] 07AgentIsai synchronize pull request 03#4547: T9025: Set wgCosmosRailBlacklistedNamespaces + alphabetize Cosmos settings - 13https://github.com/miraheze/mw-config/pull/4547 [02:39:40] [02mw-config] 07AgentIsai reviewed pull request 03#4547 commit - 13https://github.com/miraheze/mw-config/pull/4547#discussion_r843410317 [02:40:18] [02mw-config] 07Universal-Omega reviewed pull request 03#4548 commit - 13https://github.com/miraheze/mw-config/pull/4548#discussion_r843410513 [02:40:27] miraheze/mw-config - AgentIsai the build passed. [02:40:39] [02mw-config] 07AgentIsai edited pull request 03#4547: T9025: Set wgCosmosRailBlacklistedNamespaces + alphabetize Cosmos settings - 13https://github.com/miraheze/mw-config/pull/4547 [02:42:21] PROBLEM - mwtask111 APT on mwtask111 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [02:43:09] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.97, 3.58, 3.30 [02:43:10] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.89, 7.51, 6.31 [02:44:17] Thanks for that PR Agent, I'll test it tomorrow since I don't know how the special namespace will work here, I don't feel comfortable merging without testing I don't think. Though I could, it wouldn't cause fatal, just confusion if it were to start appearing on special namespace. [02:44:53] CosmicAlpha: sounds good, thank you for your patience :) [02:45:11] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.50, 7.74, 6.53 [02:45:57] [02mw-config] 07Universal-Omega edited pull request 03#4547: Alphabetize Cosmos settings - 13https://github.com/miraheze/mw-config/pull/4547 [02:46:06] [02mw-config] 07Universal-Omega edited pull request 03#4547: Alphabetize Cosmos settings in LocalSettings.php - 13https://github.com/miraheze/mw-config/pull/4547 [02:46:17] [02mw-config] 07Universal-Omega edited pull request 03#4547: Alphabetize Cosmos settings in LocalSettings.php - 13https://github.com/miraheze/mw-config/pull/4547 [02:46:36] RECOVERY - mwtask111 APT on mwtask111 is OK: APT OK: 58 packages available for upgrade (0 critical updates). [02:46:57] [02mw-config] 07Universal-Omega edited pull request 03#4547: Alphabetise Cosmos settings in LocalSettings.php - 13https://github.com/miraheze/mw-config/pull/4547 [02:47:09] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.93, 3.30, 3.26 [02:48:05] [02mw-config] 07Universal-Omega closed pull request 03#4547: Alphabetise Cosmos settings in LocalSettings.php - 13https://github.com/miraheze/mw-config/pull/4547 [02:48:06] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/10be844ceda7...4ae36e9bb183 [02:48:08] [02miraheze/mw-config] 07AgentIsai 034ae36e9 - Alphabetise Cosmos settings in LocalSettings.php (#4547) [02:49:11] miraheze/mw-config - Universal-Omega the build passed. [02:49:57] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/4ae36e9bb183...f7a315889680 [02:49:58] [02miraheze/mw-config] 07Universal-Omega 03f7a3158 - Add newline after Cosmos configs [02:51:00] miraheze/mw-config - Universal-Omega the build passed. [02:53:01] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.20, 3.48, 3.32 [02:53:09] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.06, 3.88, 3.57 [02:54:18] PROBLEM - mwtask111 APT on mwtask111 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [02:55:00] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.40, 3.40, 3.30 [02:56:15] !log [@mwtask111] starting deploy of {'config': True} to all [02:56:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:56:32] !log [@mwtask111] finished deploy of {'config': True} to all - SUCCESS in 16s [02:56:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:57:09] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.88, 4.28, 3.78 [02:58:32] RECOVERY - mwtask111 APT on mwtask111 is OK: APT OK: 58 packages available for upgrade (0 critical updates). [02:58:58] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.93, 4.51, 3.77 [02:59:59] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 6.41, 6.86, 6.54 [03:00:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:01:59] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 4.97, 6.32, 6.39 [03:04:55] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.01, 3.89, 3.75 [03:05:04] PROBLEM - mwtask111 APT on mwtask111 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [03:06:55] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.45, 4.01, 3.80 [03:07:09] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.10, 3.97, 3.99 [03:08:54] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.21, 3.55, 3.65 [03:09:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:11:26] RECOVERY - mwtask111 APT on mwtask111 is OK: APT OK: 58 packages available for upgrade (0 critical updates). [03:12:52] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.03, 3.47, 3.57 [03:14:51] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.34, 3.39, 3.53 [03:15:09] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.65, 4.27, 4.01 [03:16:14] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.28, 1.52, 1.91 [03:17:09] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.72, 3.98, 3.93 [03:18:50] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.43, 2.81, 3.27 [03:19:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:22:14] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.40, 1.87, 1.91 [03:25:10] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.05, 3.85, 3.72 [03:26:01] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:26:14] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.47, 1.83, 1.91 [03:26:55] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.52, 3.19, 2.92 [03:27:03] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.54, 1.95, 1.53 [03:27:09] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.58, 3.30, 3.53 [03:28:55] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.12, 2.97, 2.88 [03:28:59] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.89, 1.60, 1.45 [03:31:01] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:35:09] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.85, 3.02, 3.35 [03:38:14] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.52, 1.44, 1.70 [03:39:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:44:26] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 73, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') File "/usr/l [03:44:26] n3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query bellinrattin.it. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [03:44:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:45:58] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.45, 3.58, 3.40 [03:46:14] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.23, 1.81, 1.72 [03:47:55] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.17, 3.43, 3.37 [03:48:14] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.68, 1.76, 1.71 [03:49:53] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.89, 3.82, 3.51 [03:50:14] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.21, 1.94, 1.79 [03:51:50] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.27, 3.19, 3.31 [03:52:14] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.88, 1.95, 1.81 [03:54:14] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.40, 2.18, 1.92 [03:54:41] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.13, 7.02, 6.40 [03:58:14] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.72, 1.94, 1.89 [03:58:41] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.43, 7.80, 6.86 [04:00:41] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.58, 7.69, 6.93 [04:02:41] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 5.40, 6.77, 6.69 [04:03:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:08:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:10:14] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.43, 1.85, 1.82 [04:14:11] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp20.miraheze.org - CNAME OK [04:14:14] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.47, 1.81, 1.82 [04:22:14] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 3.06, 2.15, 1.92 [04:34:51] PROBLEM - mwtask111 APT on mwtask111 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [04:36:57] RECOVERY - mwtask111 APT on mwtask111 is OK: APT OK: 58 packages available for upgrade (0 critical updates). [04:41:25] PROBLEM - mwtask111 APT on mwtask111 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [04:50:00] RECOVERY - mwtask111 APT on mwtask111 is OK: APT OK: 58 packages available for upgrade (0 critical updates). [04:54:27] PROBLEM - mwtask111 APT on mwtask111 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [05:00:53] RECOVERY - mwtask111 APT on mwtask111 is OK: APT OK: 58 packages available for upgrade (0 critical updates). [05:08:50] PROBLEM - db121 Current Load on db121 is CRITICAL: CRITICAL - load average: 8.93, 7.24, 5.92 [05:10:49] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 7.09, 7.08, 6.02 [05:12:49] RECOVERY - db121 Current Load on db121 is OK: OK - load average: 5.38, 6.52, 5.95 [05:14:53] PROBLEM - mwtask111 APT on mwtask111 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [05:27:01] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.99, 3.51, 2.66 [05:29:00] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.09, 3.21, 2.64 [05:29:54] RECOVERY - mwtask111 APT on mwtask111 is OK: APT OK: 58 packages available for upgrade (0 critical updates). [05:31:23] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.34, 5.99, 5.51 [05:33:22] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 5.22, 5.75, 5.49 [05:34:22] PROBLEM - mwtask111 APT on mwtask111 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [05:40:54] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 5.35, 4.51, 2.36 [05:45:06] RECOVERY - mwtask111 APT on mwtask111 is OK: APT OK: 58 packages available for upgrade (0 critical updates). [05:45:15] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.17, 7.45, 6.40 [05:46:40] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 0.99, 1.96, 1.86 [05:47:14] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 6.67, 7.19, 6.43 [05:49:33] PROBLEM - mwtask111 APT on mwtask111 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [05:50:30] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.03, 1.52, 1.70 [05:51:13] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 5.85, 6.59, 6.36 [05:56:20] PROBLEM - wiki.valkyrienskies.org - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 73, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') File "/u [05:56:20] ython3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query valkyrienskies.org. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [05:59:03] [02puppet] 07JohnFLewis commented on pull request 03#2465: jobs: move around - 13https://github.com/miraheze/puppet/pull/2465#issuecomment-1089853036 [06:17:42] [02puppet] 07RhinosF1 commented on pull request 03#2465: jobs: move around - 13https://github.com/miraheze/puppet/pull/2465#issuecomment-1089864081 [06:20:05] [02puppet] 07RhinosF1 commented on pull request 03#2465: jobs: move around - 13https://github.com/miraheze/puppet/pull/2465#issuecomment-1089865521 [06:24:55] RECOVERY - wiki.valkyrienskies.org - reverse DNS on sslhost is OK: SSL OK - wiki.valkyrienskies.org reverse DNS resolves to cp20.miraheze.org - CNAME OK [06:33:09] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.65, 3.12, 2.87 [06:37:09] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.70, 2.96, 2.88 [06:43:30] PROBLEM - cloud10 Puppet on cloud10 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [07:09:30] RECOVERY - cloud10 Puppet on cloud10 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [07:15:34] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 6.89, 6.55, 5.87 [07:16:33] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 73, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') File "/usr/l [07:16:33] n3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query bellinrattin.it. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [07:17:33] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 5.35, 6.13, 5.81 [07:23:30] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.08, 6.76, 6.17 [07:25:29] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 6.45, 6.47, 6.12 [07:30:40] RECOVERY - mwtask111 APT on mwtask111 is OK: APT OK: 58 packages available for upgrade (0 critical updates). [07:37:14] PROBLEM - mwtask111 APT on mwtask111 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [07:41:30] RECOVERY - mwtask111 APT on mwtask111 is OK: APT OK: 58 packages available for upgrade (0 critical updates). [07:45:58] PROBLEM - mwtask111 APT on mwtask111 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [08:00:59] RECOVERY - mwtask111 APT on mwtask111 is OK: APT OK: 58 packages available for upgrade (0 critical updates). [08:03:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:08:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:15:32] PROBLEM - conlang.tk - LetsEncrypt on sslhost is CRITICAL: connect to address conlang.tk and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [08:16:04] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp20.miraheze.org - CNAME OK [08:17:47] PROBLEM - conlang.tk - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for conlang.tk could not be found [08:26:01] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:31:01] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:13:38] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 6.53, 6.85, 6.17 [09:15:37] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 6.33, 6.49, 6.11 [09:27:31] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.35, 1.92, 1.42 [09:29:29] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.48, 1.42, 1.29 [09:38:14] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.26, 1.49, 1.92 [09:46:14] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 0.81, 1.17, 1.61 [09:49:22] PROBLEM - mwtask111 APT on mwtask111 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [09:50:14] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.16, 1.67, 1.71 [09:56:14] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.75, 1.99, 1.88 [09:57:56] RECOVERY - mwtask111 APT on mwtask111 is OK: APT OK: 58 packages available for upgrade (0 critical updates). [10:02:14] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.07, 1.96, 1.88 [10:04:14] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.70, 1.84, 1.84 [10:08:14] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.50, 2.05, 1.91 [10:10:00] PROBLEM - mwtask111 APT on mwtask111 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [10:14:16] RECOVERY - mwtask111 APT on mwtask111 is OK: APT OK: 58 packages available for upgrade (0 critical updates). [10:19:53] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 73, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') File "/usr/l [10:19:53] n3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query bellinrattin.it. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [10:29:47] !log [rhinos@mwtask111] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/w/maintenance/runJobs.php --type=htmlCacheUpdate (START) [10:29:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:32:29] PROBLEM - mwtask111 APT on mwtask111 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [10:33:31] !log [rhinos@mwtask111] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/w/maintenance/runJobs.php --type=htmlCacheUpdate (END - exit=2) [10:33:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:34:36] RECOVERY - mwtask111 APT on mwtask111 is OK: APT OK: 58 packages available for upgrade (0 critical updates). [10:42:14] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 0.85, 1.45, 1.93 [10:48:54] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.53, 6.85, 6.29 [10:49:38] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - dnd.bellinrattin.it All nameservers failed to answer the query. [10:52:14] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.04, 1.73, 1.78 [10:56:50] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 4.89, 6.00, 6.22 [10:56:55] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 11.35, 6.27, 3.52 [10:57:50] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.29, 5.32, 3.57 [10:57:51] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.45, 4.63, 3.22 [11:00:14] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.62, 2.00, 1.96 [11:01:49] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.06, 3.87, 3.23 [11:02:14] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 1.97, 2.01, 1.98 [11:02:55] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.60, 3.79, 3.27 [11:03:42] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.51, 3.99, 3.55 [11:03:48] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.36, 3.32, 3.11 [11:04:55] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.86, 3.13, 3.09 [11:05:39] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.16, 3.04, 3.25 [11:18:14] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.30, 1.70, 1.96 [11:19:24] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 86, in check_records cname = str(dns_resolver.resolve(hostname, 'CNAME')[0]) File "/usr/li [11:19:24] 3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query dnd.bellinrattin.it. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [11:22:14] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.31, 1.99, 2.02 [11:26:56] PROBLEM - mwtask111 APT on mwtask111 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [11:29:29] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.70, 8.00, 7.06 [11:31:07] RECOVERY - mwtask111 APT on mwtask111 is OK: APT OK: 58 packages available for upgrade (0 critical updates). [11:32:14] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.04, 1.58, 1.90 [11:33:29] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 5.08, 6.82, 6.82 [11:42:14] PROBLEM - wiki.ssangyongsports.eu.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.ssangyongsports.eu.org could not be found [11:42:14] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.45, 1.50, 1.70 [11:48:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:49:09] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp20.miraheze.org - CNAME OK [11:53:10] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.59, 7.98, 6.78 [11:54:14] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.78, 1.77, 1.71 [11:56:14] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.56, 1.67, 1.68 [11:57:10] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.54, 7.80, 6.99 [12:00:14] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.11, 1.90, 1.77 [12:03:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:03:56] PROBLEM - mwtask111 APT on mwtask111 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [12:09:29] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 4.13, 2.86, 1.54 [12:11:57] RECOVERY - wiki.ssangyongsports.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.ssangyongsports.eu.org reverse DNS resolves to cp21.miraheze.org - CNAME OK [12:13:29] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.69, 1.55, 1.29 [12:20:39] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.15, 8.76, 7.73 [12:22:39] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.90, 8.64, 7.82 [12:24:33] !log [rhinos@mwtask111] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/w/maintenance/runJobs.php (END - exit=0) [12:24:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:25:10] Heh [12:25:26] RECOVERY - mwtask111 APT on mwtask111 is OK: APT OK: 58 packages available for upgrade (0 critical updates). [12:25:27] It's very high but I'm not sure what with [12:28:39] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.35, 8.47, 8.00 [12:29:11] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.96, 3.17, 2.72 [12:29:13] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.48, 3.10, 2.53 [12:31:09] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.86, 3.04, 2.73 [12:31:12] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.02, 2.74, 2.47 [12:31:20] The red line is the paging limit [12:31:35] It's got to stay above that for 90 minutes at the moment [12:34:01] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.65, 9.36, 8.06 [12:34:39] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.57, 8.86, 8.33 [12:35:07] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 9.68, 7.69, 6.50 [12:36:00] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.72, 8.35, 7.85 [12:36:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:36:39] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.63, 9.23, 8.51 [12:37:06] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.45, 7.47, 6.56 [12:38:39] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.99, 9.22, 8.60 [12:41:10] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.92, 9.72, 8.46 [12:45:07] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.01, 9.15, 8.59 [12:48:39] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.22, 8.88, 8.75 [12:51:00] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.77, 7.72, 7.14 [12:51:03] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.05, 9.01, 8.62 [12:52:39] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.60, 9.49, 9.11 [12:52:59] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.91, 7.84, 7.26 [12:53:02] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.48, 8.96, 8.66 [13:00:05] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.26, 3.60, 2.94 [13:02:05] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.34, 3.21, 2.88 [13:04:53] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.98, 7.85, 8.36 [13:04:54] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 5.73, 6.18, 6.69 [13:08:06] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.21, 3.46, 3.11 [13:08:39] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.76, 9.21, 8.93 [13:08:52] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.81, 8.26, 7.81 [13:10:05] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.83, 3.26, 3.08 [13:10:39] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 6.69, 8.34, 8.66 [13:10:52] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.10, 7.85, 7.72 [13:11:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:12:41] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.09, 3.44, 3.20 [13:12:44] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.77, 9.12, 8.71 [13:14:43] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.46, 8.37, 8.48 [13:16:36] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.00, 3.98, 3.44 [13:18:39] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.19, 9.96, 9.19 [13:19:55] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.72, 3.56, 3.18 [13:20:01] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:20:05] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.33, 4.11, 3.40 [13:20:39] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.25, 9.76, 9.22 [13:22:16] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - dnd.bellinrattin.it All nameservers failed to answer the query. [13:23:46] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.05, 2.98, 3.05 [13:24:05] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.37, 3.36, 3.28 [13:24:34] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.60, 8.92, 8.32 [13:24:39] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.05, 9.27, 9.10 [13:26:31] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.30, 9.40, 8.88 [13:26:46] .mh SAL [13:26:46] https://meta.miraheze.org/wiki/SAL [13:28:19] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.18, 3.39, 3.63 [13:30:26] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.49, 7.94, 8.14 [13:32:27] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.25, 9.23, 8.86 [13:32:39] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.85, 9.47, 9.35 [13:33:26] !log [rhinos@mwtask111] sudo -u www-data php /srv/mediawiki/w/extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=zhinternetpedia --logwiki=metawiki TheBurningPrincess Eytirth (END - exit=65280) [13:33:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:33:42] !log [rhinos@mwtask111] sudo -u www-data php /srv/mediawiki/w/extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=zhinternetpediawiki --logwiki=metawiki TheBurningPrincess Eytirth (END - exit=256) [13:33:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:34:11] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.21, 2.92, 3.34 [13:34:26] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.47, 9.52, 9.02 [13:34:34] !log [rhinos@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/sql.php --wiki=zhinternetpediawiki /srv/mediawiki/w/extensions/VoteNY/sql/vote.mysql (END - exit=0) [13:34:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:34:43]