[00:00:07] [02mw-config] 07Universal-Omega synchronize pull request 03#5147: Try parser cache sharding - 13https://github.com/miraheze/mw-config/pull/5147 [00:00:10] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03parsercache [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/48ef3387e2c5...f62d1013fde7 [00:00:11] [02miraheze/mw-config] 07Universal-Omega 03f62d101 - Update GlobalCache.php [00:01:12] miraheze/mw-config - Universal-Omega the build passed. [00:03:59] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.98, 3.85, 3.81 [00:04:25] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.74, 3.23, 2.80 [00:05:38] !log create tables parsercache.pc0, and parsercache.pc1 (try sharding parsercache) [00:05:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:06:24] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.62, 3.25, 2.87 [00:06:48] [02mw-config] 07Universal-Omega edited pull request 03#5147: Try sharding parsercache - 13https://github.com/miraheze/mw-config/pull/5147 [00:07:04] [02mw-config] 07Universal-Omega edited pull request 03#5147: Enable sharding on parsercache - 13https://github.com/miraheze/mw-config/pull/5147 [00:07:22] [02mw-config] 07Universal-Omega synchronize pull request 03#5147: Enable sharding on parsercache - 13https://github.com/miraheze/mw-config/pull/5147 [00:07:25] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03parsercache [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/f62d1013fde7...169e00f0d719 [00:07:28] [02miraheze/mw-config] 07Universal-Omega 03169e00f - Update GlobalCache.php [00:07:57] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.62, 4.44, 4.05 [00:08:32] miraheze/mw-config - Universal-Omega the build passed. [00:09:56] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.19, 3.59, 3.79 [00:10:18] Create tables parsercache.pc2 - parsercache.pc9 on db121 (sharding) [00:10:20] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.73, 3.75, 3.16 [00:10:31] !log Create tables parsercache.pc2 - parsercache.pc9 on db121 (sharding) [00:10:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:12:19] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.22, 3.78, 3.23 [00:12:34] PROBLEM - cp22 Current Load on cp22 is CRITICAL: CRITICAL - load average: 4.48, 2.39, 1.51 [00:14:34] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 1.15, 1.85, 1.42 [00:15:35] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - uk.religiononfire.mar.in.ua All nameservers failed to answer the query. [00:16:17] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.60, 3.15, 3.14 [00:16:18] [02mw-config] 07Universal-Omega synchronize pull request 03#5147: Enable sharding on parsercache - 13https://github.com/miraheze/mw-config/pull/5147 [00:16:20] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03parsercache [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/169e00f0d719...af0c47625ddf [00:16:23] [02miraheze/mw-config] 07Universal-Omega 03af0c476 - Update GlobalCache.php [00:17:21] miraheze/mw-config - Universal-Omega the build passed. [00:17:52] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03parsercache [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/af0c47625ddf...348e6405e3ac [00:17:55] [02miraheze/mw-config] 07Universal-Omega 03348e640 - Update Database.php [00:17:58] [02mw-config] 07Universal-Omega synchronize pull request 03#5147: Enable sharding on parsercache - 13https://github.com/miraheze/mw-config/pull/5147 [00:18:52] miraheze/mw-config - Universal-Omega the build passed. [00:25:50] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 7.12, 4.46, 3.88 [00:27:30] !log [@test131] starting deploy of {'config': True} to all [00:27:31] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [00:27:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:27:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:28:08] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [00:28:36] !log create parsercache databases on db131 and db142, and create pc0-pc9 tables for parsercache multi-server sharding [00:28:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:29:48] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.09, 3.66, 3.72 [00:30:49] PROBLEM - es141 Current Load on es141 is CRITICAL: CRITICAL - load average: 4.45, 3.13, 2.24 [00:31:53] [02mw-config] 07Universal-Omega edited pull request 03#5147: Enable sharding and multi-server on parsercache - 13https://github.com/miraheze/mw-config/pull/5147 [00:32:00] [02mw-config] 07Universal-Omega closed pull request 03#5147: Enable sharding and multi-server on parsercache - 13https://github.com/miraheze/mw-config/pull/5147 [00:32:03] [02miraheze/mw-config] 07Universal-Omega deleted branch 03parsercache [00:32:04] [02mw-config] 07Universal-Omega deleted branch 03parsercache - 13https://github.com/miraheze/mw-config [00:32:07] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/mw-config/compare/5864258272f3...d712763c8d05 [00:32:09] [02miraheze/mw-config] 07Universal-Omega 03d712763 - Enable sharding and multi-server on parsercache (#5147) [00:32:14] !log [universalomega@mwtask141] starting deploy of {'pull': 'config', 'config': True} to all [00:32:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:32:48] RECOVERY - es141 Current Load on es141 is OK: OK - load average: 3.34, 3.15, 2.36 [00:32:53] !log [universalomega@mwtask141] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 39s [00:32:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:33:03] !log DROP parsercache.pc table on db121 [00:33:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:35:05] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.51, 3.56, 3.06 [00:35:45] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.75, 2.63, 3.25 [00:37:04] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.65, 3.66, 3.16 [00:38:34] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.58, 11.12, 11.92 [00:39:03] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.08, 3.75, 3.25 [00:41:01] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.72, 3.00, 3.03 [00:44:54] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.00, 11.02, 11.98 [00:48:34] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.81, 10.52, 11.10 [00:48:54] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.10, 11.31, 11.80 [00:50:54] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.92, 11.00, 11.62 [00:51:10] RECOVERY - mwtask141 Puppet on mwtask141 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:52:35] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.97, 10.77, 11.17 [00:52:58] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.09, 11.03, 11.52 [00:53:37] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.79, 3.34, 2.87 [00:55:09] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.63, 10.91, 11.41 [00:57:08] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.67, 11.35, 11.50 [00:57:35] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.09, 3.31, 3.05 [00:58:51] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.88, 12.34, 11.59 [00:58:51] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.72, 11.47, 11.52 [00:59:24] !log [universalomega@test131] starting deploy of {'pull': 'config', 'config': True} to all [00:59:25] !log [universalomega@test131] DEPLOY ABORTED: Non-Zero Exit Code in prep, see output. [00:59:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:59:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:00:09] PROBLEM - test131 Puppet on test131 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki config] [01:00:11] PROBLEM - removededm.ml - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for removededm.ml could not be found [01:00:27] !log [universalomega@test131] starting deploy of {'pull': 'config', 'config': True} to all [01:00:28] !log [universalomega@test131] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 0s [01:00:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:00:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:00:51] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.26, 12.17, 11.78 [01:02:57] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.09, 11.84, 11.72 [01:04:07] RECOVERY - test131 Puppet on test131 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:08:29] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.95, 11.72, 11.57 [01:08:45] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.89, 3.29, 2.73 [01:10:44] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.93, 3.12, 2.73 [01:13:36] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mar.in.ua. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [01:14:35] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.71, 11.34, 11.85 [01:16:26] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.15, 11.10, 11.72 [01:25:22] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.22, 3.44, 3.12 [01:26:34] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 14.08, 11.85, 11.69 [01:28:22] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 15.12, 12.87, 12.03 [01:28:34] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.59, 11.74, 11.68 [01:29:20] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.34, 4.06, 3.44 [01:30:34] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.19, 11.87, 11.73 [01:32:34] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.39, 11.66, 11.66 [01:33:18] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.51, 3.80, 3.52 [01:34:34] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 16.11, 12.95, 12.11 [01:35:17] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.38, 3.39, 3.40 [01:36:28] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.59, 3.55, 3.06 [01:38:27] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 3.11, 3.36, 3.06 [01:43:13] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.49, 3.18, 3.19 [01:43:36] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [01:45:12] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 6.13, 4.10, 3.51 [01:47:11] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.28, 3.36, 3.32 [01:48:15] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.05, 11.38, 11.89 [01:54:09] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.21, 10.84, 11.43 [01:55:08] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 3.79, 4.10, 3.60 [01:56:07] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.37, 10.10, 11.08 [01:56:16] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.93, 3.48, 2.95 [01:57:07] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.80, 3.21, 3.33 [01:58:05] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.51, 11.37, 11.97 [01:59:59] PROBLEM - es141 PowerDNS Recursor on es141 is CRITICAL: CRITICAL - Plugin timed out while executing system call [02:01:56] RECOVERY - es141 PowerDNS Recursor on es141 is OK: DNS OK: 0.321 seconds response time. miraheze.org returns 109.228.51.216,217.174.247.33,2a00:da00:1800:326::1,2a00:da00:1800:328::1 [02:02:01] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.28, 10.92, 11.03 [02:02:13] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.42, 3.67, 3.34 [02:04:13] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.81, 3.19, 3.19 [02:05:06] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.82, 3.52, 3.34 [02:05:57] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.65, 11.55, 11.29 [02:07:05] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.21, 3.05, 3.19 [02:09:53] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.16, 11.22, 11.17 [02:11:51] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.23, 11.06, 11.11 [02:12:00] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.13, 11.44, 11.34 [02:13:59] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.14, 11.23, 11.27 [02:18:04] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.11, 3.75, 3.18 [02:19:43] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.03, 11.10, 10.94 [02:20:03] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.45, 3.59, 3.19 [02:21:41] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.11, 10.00, 10.59 [02:22:02] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.83, 3.92, 3.35 [02:23:39] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.60, 8.97, 10.15 [02:24:01] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.67, 3.44, 3.24 [02:25:02] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.83, 3.74, 3.45 [02:25:55] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.14, 10.70, 10.71 [02:25:59] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.45, 3.09, 3.13 [02:27:54] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.11, 11.04, 10.85 [02:29:00] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.13, 2.93, 3.20 [02:29:54] RECOVERY - removededm.ml - reverse DNS on sslhost is OK: SSL OK - removededm.ml reverse DNS resolves to cp23.miraheze.org - NS RECORDS OK [02:30:48] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - en.religiononfire.mar.in.ua All nameservers failed to answer the query. [02:33:52] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.04, 9.14, 10.05 [02:47:50] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.83, 3.12, 2.94 [02:49:49] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.69, 2.95, 2.90 [02:55:06] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.47, 10.29, 9.72 [02:55:46] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.77, 4.05, 3.33 [02:57:04] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.17, 10.82, 9.98 [02:59:02] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.25, 10.79, 10.09 [03:00:29] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [03:01:29] RECOVERY - db131 Backups SQL on db131 is OK: FILE_AGE OK: /var/log/sql-backup.log is 87 seconds old and 273 bytes [03:02:18] PROBLEM - db112 Backups SQL on db112 is WARNING: FILE_AGE WARNING: /var/log/sql-backup.log is 864136 seconds old and 365 bytes [03:02:20] PROBLEM - db112 Backups SQL on db112 is WARNING: FILE_AGE WARNING: /var/log/sql-backup.log is 864139 seconds old and 365 bytes [03:02:58] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.84, 9.91, 9.90 [03:09:40] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 1.67, 3.65, 3.92 [03:15:36] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - uk.religiononfire.mar.in.ua All nameservers failed to answer the query. [03:15:37] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.24, 2.23, 3.20 [03:18:33] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.10, 10.50, 9.80 [03:20:32] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.74, 10.57, 9.90 [03:20:37] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.64, 11.00, 9.82 [03:22:35] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.61, 11.12, 10.00 [03:24:31] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.99, 9.97, 9.82 [03:25:33] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.80, 4.07, 3.47 [03:26:34] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.17, 10.08, 9.84 [03:27:22] PROBLEM - cp33 APT on cp33 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [03:29:18] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/d712763c8d05...7b230dd2072b [03:29:20] [02miraheze/mw-config] 07Universal-Omega 037b230dd - Database: remove pc* from external loads [03:29:21] RECOVERY - cp33 APT on cp33 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [03:30:25] miraheze/mw-config - Universal-Omega the build passed. [03:31:30] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.32, 3.66, 3.59 [03:32:00] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/203b3f905508 [03:32:02] [02miraheze/mw-config] 07Universal-Omega 03203b3f9 - Switch main cache to mem141 [03:32:03] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-2 - 13https://github.com/miraheze/mw-config [03:32:04] [02mw-config] 07Universal-Omega opened pull request 03#5148: Switch main cache to mem141 - 13https://github.com/miraheze/mw-config/pull/5148 [03:32:27] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/203b3f905508...63797c1a7c73 [03:32:30] [02miraheze/mw-config] 07Universal-Omega 0363797c1 - Update SemanticMediaWiki.php [03:32:32] [02mw-config] 07Universal-Omega synchronize pull request 03#5148: Switch main cache to mem141 - 13https://github.com/miraheze/mw-config/pull/5148 [03:33:09] miraheze/mw-config - Universal-Omega the build passed. [03:33:18] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.95, 3.54, 3.23 [03:33:30] miraheze/mw-config - Universal-Omega the build passed. [03:35:17] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.00, 4.02, 3.43 [03:35:26] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.35, 11.33, 10.34 [03:35:28] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.63, 2.68, 3.20 [03:37:24] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.54, 9.48, 9.78 [03:39:14] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.51, 3.61, 3.44 [03:40:39] !log [@mwtask141] starting deploy of {'config': True} to all [03:40:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:40:46] !log [@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 7s [03:40:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:41:57] [02mw-config] 07Universal-Omega closed pull request 03#5148: Switch main cache to mem141 - 13https://github.com/miraheze/mw-config/pull/5148 [03:41:59] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/mw-config/compare/7b230dd2072b...684eabbf41ca [03:42:01] !log [universalomega@mwtask141] starting deploy of {'pull': 'config', 'config': True} to all [03:42:01] [02miraheze/mw-config] 07Universal-Omega 03684eabb - Switch main cache to mem141 (#5148) [03:42:03] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 [03:42:06] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 - 13https://github.com/miraheze/mw-config [03:42:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:42:13] !log [universalomega@mwtask141] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 11s [03:42:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:42:54] miraheze/mw-config - Universal-Omega the build passed. [03:43:12] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.01, 2.97, 3.24 [03:43:43] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03revert-5148-Universal-Omega-patch-2 [+0/-0/±2] 13https://github.com/miraheze/mw-config/commit/a4c065188275 [03:43:44] [02miraheze/mw-config] 07Universal-Omega 03a4c0651 - Revert "Switch main cache to mem141 (#5148)" [03:43:46] [02mw-config] 07Universal-Omega created branch 03revert-5148-Universal-Omega-patch-2 - 13https://github.com/miraheze/mw-config [03:43:47] [02mw-config] 07Universal-Omega opened pull request 03#5149: Revert "Switch main cache to mem141" - 13https://github.com/miraheze/mw-config/pull/5149 [03:43:52] [02mw-config] 07Universal-Omega closed pull request 03#5149: Revert "Switch main cache to mem141" - 13https://github.com/miraheze/mw-config/pull/5149 [03:43:53] [02miraheze/mw-config] 07Universal-Omega deleted branch 03revert-5148-Universal-Omega-patch-2 [03:43:54] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/mw-config/compare/684eabbf41ca...fea892459896 [03:43:56] [02miraheze/mw-config] 07Universal-Omega 03fea8924 - Revert "Switch main cache to mem141" (#5149) [03:43:58] [02mw-config] 07Universal-Omega deleted branch 03revert-5148-Universal-Omega-patch-2 - 13https://github.com/miraheze/mw-config [03:44:00] !log [universalomega@mwtask141] starting deploy of {'pull': 'config', 'config': True} to all [03:44:11] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [03:44:15] !log [universalomega@mwtask141] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 15s [03:44:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:44:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:44:46] miraheze/mw-config - Universal-Omega the build passed. [03:45:12] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.20, 12.02, 10.72 [03:47:10] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.28, 11.62, 10.73 [03:53:04] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.67, 9.88, 10.18 [03:55:20] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.31, 4.32, 3.47 [03:56:43] !log [@test131] starting deploy of {'config': True} to all [03:56:44] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [03:56:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:56:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:57:19] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.60, 3.97, 3.44 [04:01:17] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.13, 3.18, 3.26 [04:06:58] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 6.31, 4.29, 3.30 [04:10:55] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.65, 3.22, 3.12 [04:15:35] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - uk.religiononfire.mar.in.ua All nameservers failed to answer the query. [04:18:09] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.32, 10.53, 9.94 [04:19:09] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.43, 3.84, 3.23 [04:21:08] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.12, 3.17, 3.06 [04:21:29] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.93, 10.32, 9.80 [04:22:08] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.61, 9.64, 9.77 [04:23:28] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.06, 9.37, 9.51 [04:25:10] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 15.71, 7.04, 4.44 [04:28:37] PROBLEM - cp33 APT on cp33 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [04:38:38] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.48, 3.81, 3.15 [04:39:30] RECOVERY - cp33 APT on cp33 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [04:42:36] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.94, 2.94, 2.97 [04:44:11] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [04:48:59] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.49, 9.91, 9.10 [04:50:57] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.70, 9.72, 9.15 [04:54:28] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.73, 3.97, 3.21 [04:54:55] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.20, 11.23, 9.70 [04:56:27] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.82, 3.39, 3.08 [04:56:55] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.95, 10.42, 9.59 [04:58:54] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 9.01, 9.88, 9.49 [05:03:05] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.23, 2.77, 3.84 [05:06:21] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.55, 3.51, 3.08 [05:08:20] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 3.14, 3.20, 3.01 [05:13:05] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.21, 2.36, 3.16 [05:19:24] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 16.33, 11.84, 10.18 [05:21:22] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.85, 10.32, 9.84 [05:23:07] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 7.34, 4.16, 3.51 [05:25:18] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.14, 9.87, 9.80 [05:29:05] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.40, 3.74, 3.68 [05:33:05] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.31, 2.72, 3.25 [05:35:06] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.01, 10.22, 9.79 [05:39:02] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.15, 10.15, 9.91 [05:50:48] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.80, 10.16, 9.86 [05:52:46] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.05, 9.44, 9.62 [05:53:36] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.92, 10.39, 9.56 [05:55:35] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 10.00, 10.03, 9.52 [06:05:06] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.91, 3.65, 2.71 [06:06:30] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.89, 10.40, 9.79 [06:06:34] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.16, 10.31, 10.13 [06:07:05] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.58, 3.42, 2.76 [06:08:30] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 9.15, 9.82, 9.65 [06:08:34] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.14, 9.70, 9.91 [06:09:05] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.58, 2.80, 2.61 [06:12:28] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.62, 10.79, 10.11 [06:12:34] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.88, 11.31, 10.52 [06:14:27] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 9.70, 10.16, 9.94 [06:18:25] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.70, 11.22, 10.44 [06:24:01] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.61, 3.02, 2.68 [06:24:23] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.47, 9.96, 10.19 [06:24:34] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.02, 10.90, 11.10 [06:24:40] PROBLEM - cp33 APT on cp33 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:26:00] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.50, 3.70, 2.98 [06:27:59] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.13, 3.47, 2.98 [06:28:33] PROBLEM - cp22 Current Load on cp22 is CRITICAL: CRITICAL - load average: 3.79, 5.17, 2.73 [06:28:55] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 4.22, 5.68, 3.08 [06:29:38] RECOVERY - cp33 APT on cp33 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [06:29:58] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.43, 3.14, 2.92 [06:30:33] PROBLEM - cp22 Current Load on cp22 is WARNING: WARNING - load average: 0.94, 3.63, 2.46 [06:30:34] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.03, 8.97, 10.14 [06:32:33] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 1.00, 2.78, 2.29 [06:32:55] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 0.78, 2.95, 2.54 [06:36:30] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/ff891e531f63...28a64f9e0d4d [06:36:33] [02miraheze/ssl] 07MirahezeSSLBot 0328a64f9 - Bot: Add SSL cert for cloud.neptune.wiki [06:39:45] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 7.55, 6.44, 3.74 [06:40:17] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.44, 10.37, 9.47 [06:42:16] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.14, 10.51, 9.65 [06:44:15] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.84, 9.43, 9.35 [06:45:42] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.22, 3.04, 3.06 [06:55:43] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.99, 3.43, 2.71 [06:58:29] PROBLEM - cp33 APT on cp33 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:00:25] RECOVERY - cp33 APT on cp33 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [07:01:28] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.50, 3.15, 2.88 [07:19:14] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.87, 9.88, 8.92 [07:21:13] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.78, 9.83, 9.01 [08:53:29] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.09, 10.27, 9.45 [08:57:26] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.61, 9.97, 9.59 [09:06:02] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query en.religiononfire.mar.in.ua. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [09:35:34] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [09:36:02] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.32, 10.19, 8.93 [09:38:01] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.73, 9.67, 8.90 [10:22:04] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.93, 10.38, 9.42 [10:22:35] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.39, 10.57, 9.21 [10:24:03] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.98, 9.95, 9.39 [10:24:34] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.38, 9.97, 9.15 [10:25:05] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 15.37, 5.78, 2.92 [10:28:59] PROBLEM - cp33 APT on cp33 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:30:56] RECOVERY - cp33 APT on cp33 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [10:32:44] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 0.79, 2.97, 2.81 [10:33:38] PROBLEM - removededm.ml - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for removededm.ml could not be found [10:42:51] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.80, 10.45, 9.87 [10:44:50] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.65, 9.59, 9.61 [10:48:47] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.07, 10.53, 9.96 [10:50:29] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mar.in.ua. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [10:50:46] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.62, 9.83, 9.79 [10:55:11] PROBLEM - mon141 NTP time on mon141 is WARNING: NTP WARNING: Offset 0.1039134264 secs [11:03:31] RECOVERY - mon141 NTP time on mon141 is OK: NTP OK: Offset 0.09489780664 secs [11:20:28] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [11:30:19] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 6.63, 4.95, 2.90 [11:32:19] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 2.09, 3.93, 2.79 [11:34:19] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.74, 3.10, 2.61 [11:53:19] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.78, 10.18, 8.73 [11:55:18] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.84, 9.49, 8.68 [12:01:38] RECOVERY - removededm.ml - reverse DNS on sslhost is OK: SSL OK - removededm.ml reverse DNS resolves to cp22.miraheze.org - NS RECORDS OK [12:08:37] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.59, 10.40, 9.23 [12:11:07] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.89, 11.85, 10.19 [12:13:06] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.77, 11.22, 10.16 [12:14:33] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.23, 9.79, 9.41 [12:20:28] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.57, 11.60, 10.35 [12:20:33] PROBLEM - cp23 NTP time on cp23 is WARNING: NTP WARNING: Offset 0.135899663 secs [12:23:09] PROBLEM - graylog121 Current Load on graylog121 is WARNING: WARNING - load average: 3.91, 3.02, 2.36 [12:24:19] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 8.76, 4.66, 2.63 [12:25:09] RECOVERY - graylog121 Current Load on graylog121 is OK: OK - load average: 3.38, 3.15, 2.49 [12:25:20] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 7.10, 5.80, 3.38 [12:26:59] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.73, 9.67, 10.07 [12:27:20] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 2.08, 4.63, 3.27 [12:30:19] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.39, 3.26, 2.79 [12:34:19] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 1.67, 3.69, 3.19 [12:34:20] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 15.58, 12.14, 10.93 [12:36:19] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.43, 2.81, 2.92 [12:36:53] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.55, 10.28, 10.09 [12:37:49] PROBLEM - en.famepedia.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:38:52] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.01, 10.78, 10.29 [12:40:33] RECOVERY - cp23 NTP time on cp23 is OK: NTP OK: Offset 0.07692724466 secs [12:48:12] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.46, 11.55, 11.70 [12:48:46] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.83, 11.16, 11.11 [12:54:43] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.89, 9.02, 10.09 [12:56:19] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.41, 3.46, 2.59 [12:58:19] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.55, 2.70, 2.42 [13:00:05] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.11, 8.34, 9.87 [13:06:23] RECOVERY - en.famepedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'en.famepedia.org' will expire on Thu 01 Jun 2023 19:24:13 GMT +0000. [13:10:16] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.92, 3.66, 2.33 [13:12:16] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.71, 2.83, 2.18 [13:12:33] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.05, 11.25, 10.00 [13:13:55] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.13, 10.84, 10.19 [13:15:53] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.95, 10.54, 10.17 [13:16:31] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.89, 11.86, 10.61 [13:17:52] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.65, 11.37, 10.49 [13:21:50] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.47, 11.45, 10.73 [13:23:49] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.39, 11.32, 10.74 [13:24:29] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 16.21, 12.60, 11.27 [13:25:47] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.34, 11.25, 10.78 [13:26:29] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.31, 11.82, 11.16 [13:28:10] PROBLEM - cp33 APT on cp33 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:29:47] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.50, 11.24, 10.83 [13:30:06] RECOVERY - cp33 APT on cp33 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [13:30:29] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.26, 11.97, 11.31 [13:31:47] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.78, 11.01, 10.79 [13:32:30] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.93, 10.33, 10.79 [13:33:47] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 14.69, 11.94, 11.13 [13:40:29] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.96, 11.20, 11.00 [13:42:29] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.08, 11.56, 11.19 [13:43:47] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.44, 11.74, 11.53 [13:44:33] PROBLEM - cp23 NTP time on cp23 is WARNING: NTP WARNING: Offset -0.1139574051 secs [13:45:47] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.95, 12.13, 11.69 [13:46:29] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 16.93, 13.43, 11.95 [13:46:33] RECOVERY - cp23 NTP time on cp23 is OK: NTP OK: Offset -0.09572070837 secs [13:54:19] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.01, 3.41, 2.43 [13:55:41] PROBLEM - cp22 NTP time on cp22 is WARNING: NTP WARNING: Offset 0.3020961583 secs [13:58:29] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.54, 10.84, 11.63 [13:59:47] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.03, 11.06, 11.82 [14:01:41] RECOVERY - cp22 NTP time on cp22 is OK: NTP OK: Offset -0.004317075014 secs [14:01:47] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.38, 11.32, 11.80 [14:02:19] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.37, 3.45, 2.98 [14:04:19] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.56, 2.70, 2.76 [14:08:16] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.12, 2.94, 2.09 [14:08:29] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.55, 12.07, 11.58 [14:10:16] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.15, 2.60, 2.07 [14:36:29] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.24, 11.33, 11.86 [14:38:29] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.62, 11.55, 11.86 [14:42:42] PROBLEM - cp33 APT on cp33 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:44:29] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.67, 11.41, 11.92 [14:44:48] RECOVERY - cp33 APT on cp33 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [14:46:29] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.41, 12.19, 12.14 [14:50:19] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 3.10, 4.01, 3.14 [14:52:19] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.34, 3.33, 2.99 [14:59:10] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.95, 4.29, 3.52 [15:00:02] PROBLEM - cp33 APT on cp33 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:00:29] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.20, 11.47, 11.96 [15:01:05] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.48, 3.71, 3.41 [15:01:58] RECOVERY - cp33 APT on cp33 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [15:03:00] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 0.96, 2.76, 3.10 [15:03:15] RECOVERY - cloud11 IPMI Sensors on cloud11 is OK: IPMI Status: OK [15:03:39] PROBLEM - removededm.ml - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['columbus.aternos.org.', 'galileo.aternos.org.'], 'CNAME': None} [15:04:29] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.51, 11.59, 11.87 [15:06:29] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.81, 11.14, 11.67 [15:07:15] PROBLEM - cloud11 IPMI Sensors on cloud11 is CRITICAL: IPMI Status: Critical [Cntlr 2 Bay 7 = Critical] [15:08:29] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.08, 11.58, 11.74 [15:23:06] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.37, 3.41, 2.92 [15:25:01] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.86, 2.94, 2.82 [15:31:47] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.76, 10.40, 11.86 [15:32:17] RECOVERY - removededm.ml - reverse DNS on sslhost is OK: SSL OK - removededm.ml reverse DNS resolves to cp22.miraheze.org - NS RECORDS OK [15:32:29] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.08, 10.82, 11.91 [15:33:50] PROBLEM - wiki.zulaclub.net - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.zulaclub.net' expires in 15 day(s) (Mon 20 Mar 2023 15:31:18 GMT +0000). [15:34:47] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.55, 2.95, 2.24 [15:36:45] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.08, 3.85, 2.68 [15:38:29] [02miraheze/puppet] 07paladox pushed 031 commit to 03opensearch [+102/-0/±0] 13https://github.com/miraheze/puppet/compare/823c7085a647...fff183c89fab [15:38:30] [url] Comparing 823c7085a647...fff183c89fab · miraheze/puppet · GitHub | github.com [15:38:31] [02miraheze/puppet] 07paladox 03fff183c - Introduce opensearch puppet module [15:38:34] [02puppet] 07paladox synchronize pull request 03#3110: Introduce opensearch puppet module - 13https://github.com/miraheze/puppet/pull/3110 [15:38:35] [url] Page not found · GitHub · GitHub | github.com [15:38:43] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.48, 2.94, 2.49 [15:39:36] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/28a64f9e0d4d...3fd363acc9f7 [15:39:36] [url] Comparing 28a64f9e0d4d...3fd363acc9f7 · miraheze/ssl · GitHub | github.com [15:39:39] [02miraheze/ssl] 07MirahezeSSLBot 033fd363a - Bot: Update SSL cert for wiki.zulaclub.net [15:41:47] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.93, 11.99, 11.76 [15:43:47] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.76, 11.50, 11.61 [15:45:47] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.45, 12.10, 11.83 [15:46:29] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.51, 11.29, 11.31 [15:47:47] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.16, 11.67, 11.70 [15:48:29] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.98, 11.49, 11.39 [15:51:47] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.87, 12.40, 11.95 [15:52:29] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 16.32, 13.12, 11.98 [15:54:19] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 6.97, 4.39, 2.98 [15:54:21] PROBLEM - dutki.kro.kr - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'dutki.kro.kr' expires in 15 day(s) (Mon 20 Mar 2023 15:32:24 GMT +0000). [15:55:26] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/3fd363acc9f7...f4e4ce6f74b0 [15:55:27] [url] Comparing 3fd363acc9f7...f4e4ce6f74b0 · miraheze/ssl · GitHub | github.com [15:55:29] [02miraheze/ssl] 07MirahezeSSLBot 03f4e4ce6 - Bot: Update SSL cert for dutki.kro.kr [15:58:08] PROBLEM - cp33 APT on cp33 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:00:09] RECOVERY - cp33 APT on cp33 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [16:02:41] RECOVERY - wiki.zulaclub.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.zulaclub.net' will expire on Fri 02 Jun 2023 14:39:28 GMT +0000. [16:03:38] PROBLEM - removededm.ml - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['columbus.aternos.org.', 'galileo.aternos.org.'], 'CNAME': None} [16:04:19] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.44, 3.69, 3.81 [16:04:29] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.36, 11.62, 12.00 [16:10:19] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.80, 2.50, 3.24 [16:10:29] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.97, 11.43, 11.69 [16:24:12] RECOVERY - dutki.kro.kr - LetsEncrypt on sslhost is OK: OK - Certificate 'dutki.kro.kr' will expire on Fri 02 Jun 2023 14:55:19 GMT +0000. [16:26:19] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.89, 3.35, 3.04 [16:28:19] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.35, 2.83, 2.88 [16:29:49] PROBLEM - test131 MediaWiki Rendering on test131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:30:40] PROBLEM - test131 Current Load on test131 is CRITICAL: CRITICAL - load average: 40.69, 19.86, 8.71 [16:30:50] PROBLEM - test131 SSH on test131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:31:20] PROBLEM - test131 PowerDNS Recursor on test131 is CRITICAL: CRITICAL - Plugin timed out while executing system call [16:33:09] PROBLEM - test131 Puppet on test131 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:33:13] PROBLEM - test131 APT on test131 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:34:29] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.64, 10.26, 11.58 [16:34:54] RECOVERY - test131 SSH on test131 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [16:35:07] RECOVERY - test131 Puppet on test131 is OK: OK: Puppet is currently enabled, last run 37 minutes ago with 0 failures [16:36:03] PROBLEM - test131 Redis Process on test131 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [16:37:57] RECOVERY - test131 PowerDNS Recursor on test131 is OK: DNS OK: 0.554 seconds response time. miraheze.org returns 109.228.51.216,217.174.247.33,2a00:da00:1800:326::1,2a00:da00:1800:328::1 [16:38:00] RECOVERY - test131 Redis Process on test131 is OK: PROCS OK: 1 process with args 'redis-server' [16:40:19] RECOVERY - test131 MediaWiki Rendering on test131 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.440 second response time [16:40:29] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.27, 11.49, 11.60 [16:41:00] RECOVERY - test131 APT on test131 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [16:51:35] PROBLEM - wiki.zulaclub.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.zulaclub.net All nameservers failed to answer the query. [16:57:47] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.53, 11.31, 11.95 [17:00:40] PROBLEM - test131 MediaWiki Rendering on test131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:01:38] RECOVERY - removededm.ml - reverse DNS on sslhost is OK: SSL OK - removededm.ml reverse DNS resolves to cp23.miraheze.org - NS RECORDS OK [17:01:47] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.34, 11.86, 12.01 [17:03:23] PROBLEM - test131 PowerDNS Recursor on test131 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:03:24] PROBLEM - test131 APT on test131 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:04:48] RECOVERY - test131 MediaWiki Rendering on test131 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.978 second response time [17:05:21] RECOVERY - test131 PowerDNS Recursor on test131 is OK: DNS OK: 0.121 seconds response time. miraheze.org returns 109.228.51.216,217.174.247.33,2a00:da00:1800:326::1,2a00:da00:1800:328::1 [17:05:23] RECOVERY - test131 APT on test131 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [17:07:27] PROBLEM - test131 Puppet on test131 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/ssl/localcerts/sparabel.wiki.crt] [17:07:47] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.36, 11.21, 11.94 [17:09:47] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.06, 11.77, 12.07 [17:10:29] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.34, 11.76, 11.95 [17:12:29] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 17.34, 13.92, 12.71 [17:20:11] PROBLEM - wiki.zulaclub.net - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.zulaclub.net. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [17:22:19] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.31, 3.22, 2.62 [17:24:19] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.76, 3.13, 2.67 [17:28:19] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.12, 4.37, 3.29 [17:30:19] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 1.99, 3.46, 3.09 [17:31:39] PROBLEM - test131 APT on test131 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:32:19] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.13, 2.68, 2.84 [17:33:35] RECOVERY - test131 APT on test131 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [17:34:41] RECOVERY - test131 Puppet on test131 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [17:46:29] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.67, 11.17, 11.82 [17:47:08] PROBLEM - test131 Current Load on test131 is WARNING: WARNING - load average: 1.11, 1.71, 3.78 [17:49:36] RECOVERY - wiki.zulaclub.net - reverse DNS on sslhost is OK: SSL OK - wiki.zulaclub.net reverse DNS resolves to cp22.miraheze.org - CNAME OK [17:51:02] RECOVERY - test131 Current Load on test131 is OK: OK - load average: 1.11, 1.39, 3.18 [17:58:57] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.51, 3.98, 3.20 [17:59:47] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.57, 10.31, 11.64 [18:00:27] PROBLEM - cp23 APT on cp23 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:00:52] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.93, 3.55, 3.13 [18:02:26] RECOVERY - cp23 APT on cp23 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [18:02:47] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.73, 2.89, 2.94 [18:05:47] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.45, 11.18, 11.55 [18:07:47] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.13, 9.70, 10.96 [18:08:29] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.78, 9.35, 10.11 [18:08:33] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.66, 2.99, 2.90 [18:09:47] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 14.08, 11.20, 11.34 [18:10:27] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.39, 2.80, 2.84 [18:11:47] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.42, 11.08, 11.28 [18:12:29] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.19, 11.21, 10.67 [18:13:47] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.63, 11.49, 11.40 [18:14:29] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.15, 11.04, 10.69 [18:15:47] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.63, 10.59, 11.07 [18:23:00] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.84, 3.26, 2.90 [18:23:47] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.09, 11.59, 11.31 [18:24:55] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.03, 3.84, 3.15 [18:25:47] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.81, 10.74, 11.02 [18:26:50] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.18, 3.51, 3.11 [18:28:29] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.57, 9.50, 10.07 [18:29:15] PROBLEM - test131 MediaWiki Rendering on test131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:29:59] PROBLEM - test131 PowerDNS Recursor on test131 is CRITICAL: CRITICAL - Plugin timed out while executing system call [18:30:11] PROBLEM - test131 Current Load on test131 is CRITICAL: CRITICAL - load average: 27.16, 14.04, 6.88 [18:30:39] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.93, 2.99, 3.00 [18:30:50] PROBLEM - test131 SSH on test131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:31:36] PROBLEM - test131 APT on test131 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:33:47] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 14.95, 11.96, 11.31 [18:34:29] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.54, 11.32, 10.65 [18:34:39] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03performance [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/4699d7ef2d87 [18:34:41] [02miraheze/mw-config] 07Universal-Omega 034699d7e - Set some performance and session cache variables [18:34:43] [02mw-config] 07Universal-Omega created branch 03performance - 13https://github.com/miraheze/mw-config [18:34:43] [url] Page not found · GitHub · GitHub | github.com [18:34:46] [02mw-config] 07Universal-Omega opened pull request 03#5150: Set some performance and session cache variables - 13https://github.com/miraheze/mw-config/pull/5150 [18:34:46] [url] Page not found · GitHub · GitHub | github.com [18:35:03] RECOVERY - test131 SSH on test131 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [18:35:13] [02miraheze/mw-config] 07github-actions[bot] pushed 031 commit to 03performance [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/4699d7ef2d87...b303c26c6109 [18:35:13] [url] Comparing 4699d7ef2d87...b303c26c6109 · miraheze/mw-config · GitHub | github.com [18:35:14] [02miraheze/mw-config] 07github-actions 03b303c26 - CI: lint code to MediaWiki standards [18:35:17] [02mw-config] 07github-actions[bot] synchronize pull request 03#5150: Set some performance and session cache variables - 13https://github.com/miraheze/mw-config/pull/5150 [18:35:17] [url] Page not found · GitHub · GitHub | github.com [18:35:50] PROBLEM - wariopedia.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wariopedia.com' expires in 15 day(s) (Mon 20 Mar 2023 18:17:30 GMT +0000). [18:35:50] miraheze/mw-config - Universal-Omega the build passed. [18:36:29] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.89, 11.63, 10.85 [18:36:31] PROBLEM - test131 Puppet on test131 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:36:55] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03performance [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/b303c26c6109...2b935cbcee88 [18:36:56] [url] Comparing b303c26c6109...2b935cbcee88 · miraheze/mw-config · GitHub | github.com [18:36:57] [02miraheze/mw-config] 07Universal-Omega 032b935cb - Update Database.php [18:36:58] [02mw-config] 07Universal-Omega synchronize pull request 03#5150: Set some performance and session cache variables - 13https://github.com/miraheze/mw-config/pull/5150 [18:36:59] [url] Page not found · GitHub · GitHub | github.com [18:37:47] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.08, 11.90, 11.52 [18:37:53] miraheze/mw-config - Universal-Omega the build passed. [18:38:41] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03performance [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/2b935cbcee88...ac422e0f4ade [18:38:41] [url] Comparing 2b935cbcee88...ac422e0f4ade · miraheze/mw-config · GitHub | github.com [18:38:43] [02miraheze/mw-config] 07Universal-Omega 03ac422e0 - Update LocalSettings.php [18:38:45] [02mw-config] 07Universal-Omega synchronize pull request 03#5150: Set some performance and session cache variables - 13https://github.com/miraheze/mw-config/pull/5150 [18:38:45] [url] Page not found · GitHub · GitHub | github.com [18:39:04] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/f4e4ce6f74b0...5822d3699735 [18:39:05] [url] Comparing f4e4ce6f74b0...5822d3699735 · miraheze/ssl · GitHub | github.com [18:39:07] [02miraheze/ssl] 07MirahezeSSLBot 035822d36 - Bot: Update SSL cert for wariopedia.com [18:39:22] PROBLEM - test131 SSH on test131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:39:36] miraheze/mw-config - Universal-Omega the build passed. [18:39:47] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.33, 12.22, 11.67 [18:40:13] PROBLEM - test131 php-fpm on test131 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:40:29] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.90, 11.27, 10.80 [18:40:54] PROBLEM - test131 JobChron Service on test131 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:41:24] PROBLEM - test131 NTP time on test131 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:41:30] PROBLEM - test131 JobRunner Service on test131 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:41:42] PROBLEM - test131 Redis Process on test131 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:41:51] PROBLEM - test131 nutcracker process on test131 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:41:57] PROBLEM - test131 conntrack_table_size on test131 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:42:03] PROBLEM - test131 ferm_active on test131 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [18:42:16] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.85, 3.59, 2.88 [18:42:29] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.17, 10.22, 10.49 [18:43:18] PROBLEM - test131 HTTPS on test131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:43:47] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.31, 11.29, 11.42 [18:44:02] PROBLEM - emmytherobot.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'emmytherobot.wiki' expires in 15 day(s) (Mon 20 Mar 2023 18:16:33 GMT +0000). [18:44:16] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.44, 2.81, 2.68 [18:44:27] RECOVERY - test131 nutcracker process on test131 is OK: PROCS OK: 1 process with UID = 114 (nutcracker), command name 'nutcracker' [18:44:28] RECOVERY - test131 Redis Process on test131 is OK: PROCS OK: 1 process with args 'redis-server' [18:44:28] RECOVERY - test131 ferm_active on test131 is OK: OK ferm input default policy is set [18:44:28] RECOVERY - test131 conntrack_table_size on test131 is OK: OK: nf_conntrack is 0 % full [18:44:29] RECOVERY - test131 Puppet on test131 is OK: OK: Puppet is currently enabled, last run 46 minutes ago with 0 failures [18:44:32] RECOVERY - test131 APT on test131 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [18:44:44] RECOVERY - test131 PowerDNS Recursor on test131 is OK: DNS OK: 1.034 second response time. miraheze.org returns 109.228.51.216,217.174.247.33,2a00:da00:1800:326::1,2a00:da00:1800:328::1 [18:45:01] RECOVERY - test131 php-fpm on test131 is OK: PROCS OK: 9 processes with command name 'php-fpm7.4' [18:45:15] RECOVERY - test131 HTTPS on test131 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 549 bytes in 0.012 second response time [18:45:32] RECOVERY - test131 SSH on test131 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [18:45:43] RECOVERY - test131 JobChron Service on test131 is OK: PROCS OK: 1 process with args 'redisJobChronService' [18:46:12] RECOVERY - test131 MediaWiki Rendering on test131 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 1.463 second response time [18:46:18] RECOVERY - test131 JobRunner Service on test131 is OK: PROCS OK: 1 process with args 'redisJobRunnerService' [18:46:22] RECOVERY - test131 NTP time on test131 is OK: NTP OK: Offset 0.001186609268 secs [18:46:29] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.16, 10.53, 10.46 [18:47:38] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/5822d3699735...6fa9f8860b9f [18:47:39] [url] Comparing 5822d3699735...6fa9f8860b9f · miraheze/ssl · GitHub | github.com [18:47:41] [02miraheze/ssl] 07MirahezeSSLBot 036fa9f88 - Bot: Update SSL cert for emmytherobot.wiki [18:48:28] PROBLEM - test131 Puppet on test131 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/srv/mediawiki/stopforumspam/listed_ip_30_ipv46_all.txt] [18:49:47] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.33, 11.84, 11.57 [18:50:29] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.45, 11.95, 11.14 [18:51:47] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.77, 11.26, 11.39 [18:52:29] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.15, 11.73, 11.13 [18:53:47] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.22, 11.41, 11.41 [18:54:29] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.21, 11.54, 11.14 [18:56:19] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 3.24, 4.20, 3.33 [18:56:29] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.41, 12.24, 11.42 [18:57:47] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.80, 11.22, 11.39 [18:58:15] RECOVERY - test131 Puppet on test131 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [19:00:19] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.19, 3.58, 3.29 [19:02:19] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.10, 3.02, 3.11 [19:03:47] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.18, 11.54, 11.46 [19:04:29] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.54, 11.97, 11.86 [19:05:47] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.22, 11.72, 11.57 [19:07:56] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.00, 3.53, 2.91 [19:09:55] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.36, 3.00, 2.79 [19:10:29] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.39, 11.58, 11.68 [19:16:29] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.90, 10.97, 11.59 [19:20:29] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.53, 11.49, 11.64 [19:21:16] PROBLEM - test131 Current Load on test131 is WARNING: WARNING - load average: 1.14, 1.19, 3.80 [19:21:41] PROBLEM - cp22 NTP time on cp22 is WARNING: NTP WARNING: Offset 0.1450250447 secs [19:22:29] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.60, 10.35, 11.21 [19:23:47] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.53, 11.25, 10.94 [19:24:19] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 9.39, 4.88, 3.45 [19:24:29] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.01, 11.34, 11.43 [19:25:07] RECOVERY - test131 Current Load on test131 is OK: OK - load average: 1.08, 1.10, 3.18 [19:27:37] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.49, 3.23, 2.79 [19:28:29] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.07, 11.68, 11.62 [19:29:34] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.32, 3.01, 2.77 [19:30:29] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.44, 12.12, 11.77 [19:32:19] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.07, 3.94, 3.77 [19:33:41] RECOVERY - cp22 NTP time on cp22 is OK: NTP OK: Offset 0.07495057583 secs [19:35:44] RECOVERY - wariopedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wariopedia.com' will expire on Fri 02 Jun 2023 17:38:53 GMT +0000. [19:36:19] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.52, 2.85, 3.33 [19:37:25] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.28, 3.90, 3.21 [19:37:47] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.80, 11.93, 11.79 [19:39:23] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.00, 3.25, 3.05 [19:39:47] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.06, 11.91, 11.79 [19:41:47] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.67, 11.26, 11.58 [19:42:02] RECOVERY - emmytherobot.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'emmytherobot.wiki' will expire on Fri 02 Jun 2023 17:47:31 GMT +0000. [19:43:14] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - en.religiononfire.mar.in.ua All nameservers failed to answer the query. [19:49:47] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.83, 12.55, 11.97 [19:54:50] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.66, 3.54, 3.03 [19:56:45] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.82, 3.30, 3.00 [19:57:50] PROBLEM - cp33 APT on cp33 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:59:46] RECOVERY - cp33 APT on cp33 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [20:03:38] PROBLEM - removededm.ml - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['columbus.aternos.org.', 'galileo.aternos.org.'], 'CNAME': None} [20:08:52] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 3.86, 4.46, 3.62 [20:11:47] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.82, 10.84, 11.93 [20:13:47] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.27, 11.97, 12.21 [20:14:29] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.32, 11.60, 11.98 [20:14:46] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.02, 3.46, 3.49 [20:16:44] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.26, 2.93, 3.28 [20:19:47] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.05, 11.24, 11.92 [20:24:33] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.51, 3.29, 2.81 [20:26:27] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.75, 3.13, 2.80 [20:29:47] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.38, 11.52, 11.56 [20:30:29] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.33, 11.28, 11.31 [20:31:47] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.94, 11.07, 11.38 [20:32:29] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.77, 11.25, 11.28 [20:33:47] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 15.42, 12.41, 11.82 [20:35:47] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.77, 11.71, 11.66 [20:40:29] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.27, 10.75, 10.83 [20:42:18] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [20:44:29] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.53, 11.53, 11.21 [20:48:29] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 15.92, 13.07, 11.83 [20:51:47] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.12, 10.38, 10.58 [20:52:29] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.66, 11.81, 11.65 [20:53:47] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.02, 10.56, 10.65 [20:56:19] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.48, 4.38, 3.43 [20:56:29] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.51, 12.14, 11.80 [20:57:47] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.17, 11.65, 11.08 [20:58:59] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - uk.religiononfire.mar.in.ua All nameservers failed to answer the query. [20:59:47] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.34, 11.15, 10.97 [21:00:19] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.86, 3.28, 3.23 [21:00:29] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.59, 11.47, 11.62 [21:01:38] RECOVERY - removededm.ml - reverse DNS on sslhost is OK: SSL OK - removededm.ml reverse DNS resolves to cp23.miraheze.org - NS RECORDS OK [21:06:51] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.79, 3.95, 2.93 [21:08:49] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.79, 3.44, 2.86 [21:10:29] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.52, 8.44, 9.98 [21:10:47] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 3.06, 3.28, 2.86 [21:15:47] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.25, 9.60, 10.06 [21:19:37] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.65, 4.11, 3.26 [21:19:47] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.43, 10.53, 10.40 [21:20:29] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.54, 10.67, 10.54 [21:21:32] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.06, 3.21, 3.03 [21:21:47] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.83, 11.76, 10.89 [21:23:47] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.47, 10.77, 10.63 [21:25:22] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.54, 3.77, 3.30 [21:27:17] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.60, 3.29, 3.18 [21:28:29] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 9.63, 9.59, 10.05 [21:28:57] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [21:31:07] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.64, 3.63, 3.39 [21:33:02] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.07, 3.06, 3.21 [21:36:20] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 7.88, 4.64, 3.32 [21:38:18] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.44, 3.87, 3.18 [21:38:47] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.35, 3.42, 3.34 [21:42:16] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.06, 2.85, 2.92 [21:42:37] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.94, 2.93, 3.18 [21:45:47] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.42, 10.59, 10.57 [21:47:47] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.65, 9.65, 10.25 [21:49:47] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.82, 9.43, 10.09 [21:53:41] PROBLEM - cp22 NTP time on cp22 is WARNING: NTP WARNING: Offset 0.1541395783 secs [21:54:19] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.42, 3.80, 3.32 [21:58:23] PROBLEM - cp33 APT on cp33 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [22:00:50] RECOVERY - cp33 APT on cp33 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [22:03:38] PROBLEM - removededm.ml - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for removededm.ml could not be found [22:06:19] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 1.93, 3.40, 3.97 [22:08:16] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.36, 4.90, 3.45 [22:10:16] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.45, 3.87, 3.24 [22:12:16] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.23, 3.34, 3.13 [22:12:19] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.21, 2.45, 3.31 [22:18:19] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.70, 3.49, 3.51 [22:20:19] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.81, 2.95, 3.31 [22:24:19] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.85, 3.66, 3.55 [22:27:41] RECOVERY - cp22 NTP time on cp22 is OK: NTP OK: Offset 0.08376300335 secs [22:27:58] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.10, 10.26, 9.27 [22:30:19] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.89, 2.83, 3.24 [22:32:16] RECOVERY - removededm.ml - reverse DNS on sslhost is OK: SSL OK - removededm.ml reverse DNS resolves to cp22.miraheze.org - NS RECORDS OK [22:33:55] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.92, 9.39, 9.25 [22:35:08] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.11, 3.27, 2.79 [22:36:39] PROBLEM - wiki.aridia.space - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.aridia.space could not be found [22:37:07] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 3.07, 3.21, 2.84 [22:41:50] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.92, 10.87, 9.79 [22:43:49] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.27, 10.83, 9.92 [22:51:44] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.79, 9.91, 9.95 [22:57:19] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.42, 3.53, 3.15 [22:59:14] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.14, 3.20, 3.09 [23:02:19] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 3.99, 4.10, 2.46 [23:03:38] PROBLEM - removededm.ml - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for removededm.ml could not be found [23:04:19] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.60, 3.19, 2.33 [23:04:59] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.10, 3.60, 3.27 [23:05:20] PROBLEM - wiki.aridia.space - LetsEncrypt on sslhost is CRITICAL: connect to address wiki.aridia.space and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [23:06:35] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.46, 3.04, 2.60 [23:06:54] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.53, 3.14, 3.14 [23:08:51] [02mw-config] 07Universal-Omega synchronize pull request 03#5150: Set some performance and session cache variables - 13https://github.com/miraheze/mw-config/pull/5150 [23:08:51] [url] Page not found · GitHub · GitHub | github.com [23:08:54] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03performance [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/ac422e0f4ade...47a0a5dd6ff9 [23:08:54] [url] Comparing ac422e0f4ade...47a0a5dd6ff9 · miraheze/mw-config · GitHub | github.com [23:08:55] [02miraheze/mw-config] 07Universal-Omega 0347a0a5d - -Miser mode [23:09:49] miraheze/mw-config - Universal-Omega the build passed. [23:11:24] [02mw-config] 07Universal-Omega synchronize pull request 03#5150: Set some performance and session cache variables - 13https://github.com/miraheze/mw-config/pull/5150 [23:11:24] [url] Page not found · GitHub · GitHub | github.com [23:11:26] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03performance [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/47a0a5dd6ff9...aa133b9211cc [23:11:26] [url] Comparing 47a0a5dd6ff9...aa133b9211cc · miraheze/mw-config · GitHub | github.com [23:11:27] [02miraheze/mw-config] 07Universal-Omega 03aa133b9 - Update LocalSettings.php [23:12:14] [02mw-config] 07Universal-Omega synchronize pull request 03#5150: Set some performance and session cache variables - 13https://github.com/miraheze/mw-config/pull/5150 [23:12:14] [url] Page not found · GitHub · GitHub | github.com [23:12:16] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03performance [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/aa133b9211cc...6e963f1de0d2 [23:12:16] [url] Comparing aa133b9211cc...6e963f1de0d2 · miraheze/mw-config · GitHub | github.com [23:12:19] [02miraheze/mw-config] 07Universal-Omega 036e963f1 - Update GlobalCache.php [23:12:25] miraheze/mw-config - Universal-Omega the build passed. [23:12:29] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.53, 3.13, 2.82 [23:13:12] miraheze/mw-config - Universal-Omega the build passed. [23:13:53] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - en.religiononfire.mar.in.ua All nameservers failed to answer the query. [23:15:06] [02miraheze/mw-config] 07The-Voidwalker pushed 031 commit to 03The-Voidwalker-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/cdfb11e49315 [23:15:09] [02miraheze/mw-config] 07The-Voidwalker 03cdfb11e - add viewsuppressed to oversight [23:15:12] [02mw-config] 07The-Voidwalker created branch 03The-Voidwalker-patch-1 - 13https://github.com/miraheze/mw-config [23:15:12] [url] Page not found · GitHub · GitHub | github.com [23:15:15] [02mw-config] 07The-Voidwalker opened pull request 03#5151: add viewsuppressed to oversight - 13https://github.com/miraheze/mw-config/pull/5151 [23:15:15] [url] Page not found · GitHub · GitHub | github.com [23:16:12] miraheze/mw-config - The-Voidwalker the build passed. [23:17:29] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.88, 10.04, 9.47 [23:18:24] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.85, 3.37, 3.14 [23:19:28] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.81, 9.40, 9.32 [23:20:19] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.74, 3.20, 3.11 [23:25:04] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.56, 11.10, 9.86 [23:25:32] [02mw-config] 07Universal-Omega synchronize pull request 03#5150: Set some performance and session cache variables - 13https://github.com/miraheze/mw-config/pull/5150 [23:25:33] [url] Page not found · GitHub · GitHub | github.com [23:25:35] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03performance [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/6e963f1de0d2...2493f8daef77 [23:25:36] [url] Comparing 6e963f1de0d2...2493f8daef77 · miraheze/mw-config · GitHub | github.com [23:25:37] [02miraheze/mw-config] 07Universal-Omega 032493f8d - Update GlobalCache.php [23:26:18] !log [universalomega@test131] starting deploy of {'pull': 'config', 'branch': 'performance', 'config': True} to all [23:26:19] !log [universalomega@test131] finished deploy of {'pull': 'config', 'branch': 'performance', 'config': True} to all - SUCCESS in 1s [23:26:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:26:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:26:35] miraheze/mw-config - Universal-Omega the build passed. [23:27:03] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.58, 11.25, 10.07 [23:27:33] !log [@test131] starting deploy of {'config': True} to all [23:27:34] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [23:27:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:27:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:31:01] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.06, 9.91, 9.77 [23:32:16] RECOVERY - removededm.ml - reverse DNS on sslhost is OK: SSL OK - removededm.ml reverse DNS resolves to cp23.miraheze.org - NS RECORDS OK [23:35:41] PROBLEM - cp22 NTP time on cp22 is WARNING: NTP WARNING: Offset -0.1137450337 secs [23:37:35] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03performance [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/2493f8daef77...2b762cfefc43 [23:37:36] [url] Comparing 2493f8daef77...2b762cfefc43 · miraheze/mw-config · GitHub | github.com [23:37:37] [02miraheze/mw-config] 07Universal-Omega 032b762cf - Update GlobalCache.php [23:37:40] [02mw-config] 07Universal-Omega synchronize pull request 03#5150: Set some performance and session cache variables - 13https://github.com/miraheze/mw-config/pull/5150 [23:37:41] [url] Page not found · GitHub · GitHub | github.com [23:37:41] !log [universalomega@test131] starting deploy of {'pull': 'config', 'branch': 'performance', 'config': True} to all [23:37:42] !log [universalomega@test131] finished deploy of {'pull': 'config', 'branch': 'performance', 'config': True} to all - SUCCESS in 0s [23:37:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:37:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:39:30] [02mw-config] 07The-Voidwalker commented on pull request 03#5151: add viewsuppressed to oversight - 13https://github.com/miraheze/mw-config/pull/5151#issuecomment-1454923505 [23:39:30] [url] add viewsuppressed to oversight by The-Voidwalker · Pull Request #5151 · miraheze/mw-config · GitHub | github.com [23:42:15] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.88, 10.12, 9.20 [23:42:53] [02mw-config] 07Universal-Omega closed pull request 03#5150: Set some performance and session cache variables - 13https://github.com/miraheze/mw-config/pull/5150 [23:42:53] [url] Page not found · GitHub · GitHub | github.com [23:42:56] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/mw-config/compare/fea892459896...df30796611c6 [23:42:56] [url] Comparing fea892459896...df30796611c6 · miraheze/mw-config · GitHub | github.com [23:42:57] [02miraheze/mw-config] 07Universal-Omega 03df30796 - Set some performance and session cache variables (#5150) [23:43:00] [02miraheze/mw-config] 07Universal-Omega deleted branch 03performance [23:43:02] [02mw-config] 07Universal-Omega deleted branch 03performance - 13https://github.com/miraheze/mw-config [23:43:02] [url] Page not found · GitHub · GitHub | github.com [23:43:02] !log [universalomega@test131] starting deploy of {'pull': 'config', 'config': True} to all [23:43:05] !log [universalomega@test131] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 2s [23:43:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:43:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:43:11] !log [universalomega@mwtask141] starting deploy of {'pull': 'config', 'config': True} to all [23:43:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:43:21] !log [universalomega@mwtask141] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 9s [23:43:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:43:26] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [23:44:05] miraheze/mw-config - Universal-Omega the build passed. [23:44:14] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.12, 9.36, 9.03 [23:47:41] RECOVERY - cp22 NTP time on cp22 is OK: NTP OK: Offset -0.08438962698 secs [23:48:11] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.33, 10.38, 9.50 [23:48:49] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.72, 10.25, 9.55 [23:48:59] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/df30796611c6...4e9f20e7035c [23:49:00] [url] Comparing df30796611c6...4e9f20e7035c · miraheze/mw-config · GitHub | github.com [23:49:00] [02miraheze/mw-config] 07Universal-Omega 034e9f20e - Still use Memcached for $wgMWOAuthSessionCacheType on betaheze [23:49:06] !log [universalomega@test131] starting deploy of {'pull': 'config', 'config': True} to all [23:49:08] !log [universalomega@test131] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 2s [23:49:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:49:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:49:13] !log [universalomega@mwtask141] starting deploy of {'pull': 'config', 'config': True} to all [23:49:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:49:22] !log [universalomega@mwtask141] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 8s [23:49:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:50:10] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.53, 9.45, 9.26 [23:50:48] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.34, 9.75, 9.45 [23:54:15] PROBLEM - mw122 NTP time on mw122 is WARNING: NTP WARNING: Offset 0.1038698554 secs [23:55:03] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 6.95, 4.52, 3.34