[00:00:10] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.94, 3.48, 3.24 [00:00:16] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/cceac20b0346...8071ae556c44 [00:00:18] [02miraheze/ImportDump] 07Universal-Omega 038071ae5 - Update en.json [00:00:19] [02ImportDump] 07Universal-Omega synchronize pull request 03#1: Create extension - 13https://github.com/miraheze/ImportDump/pull/1 [00:00:21] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.47, 3.51, 4.00 [00:01:20] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/8071ae556c44...b615ae616747 [00:01:21] [02miraheze/ImportDump] 07Universal-Omega 03b615ae6 - Update qqq.json [00:01:23] [02ImportDump] 07Universal-Omega synchronize pull request 03#1: Create extension - 13https://github.com/miraheze/ImportDump/pull/1 [00:01:35] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.15, 3.74, 3.55 [00:02:07] [02ImportDump] 07Universal-Omega edited pull request 03#1: Create extension - 13https://github.com/miraheze/ImportDump/pull/1 [00:02:20] [02ImportDump] 07Universal-Omega edited pull request 03#1: Create extension - 13https://github.com/miraheze/ImportDump/pull/1 [00:02:21] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 3.94, 3.78, 4.04 [00:03:17] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/b615ae616747...88c3b1ad18da [00:03:18] [02miraheze/ImportDump] 07Universal-Omega 0388c3b1a - Update ImportDumpRequestManager.php [00:03:20] [02ImportDump] 07Universal-Omega synchronize pull request 03#1: Create extension - 13https://github.com/miraheze/ImportDump/pull/1 [00:03:29] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.71, 3.58, 3.51 [00:04:22] [02ImportDump] 07Universal-Omega edited pull request 03#1: Create extension - 13https://github.com/miraheze/ImportDump/pull/1 [00:05:23] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 6.06, 4.62, 3.91 [00:05:45] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.44, 11.03, 10.37 [00:06:08] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.02, 3.94, 3.47 [00:07:45] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.94, 10.83, 10.39 [00:08:01] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:11:45] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 14.21, 11.86, 10.85 [00:12:05] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.48, 3.80, 3.65 [00:12:39] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.10, 1.21, 1.94 [00:13:45] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.55, 10.85, 10.60 [00:14:04] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 7.19, 5.01, 4.10 [00:14:47] miraheze/ImportDump - Universal-Omega the build passed. [00:17:57] PROBLEM - es101 Current Load on es101 is WARNING: WARNING - load average: 3.96, 3.47, 2.51 [00:18:01] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.97, 1.98, 1.46 [00:19:57] RECOVERY - es101 Current Load on es101 is OK: OK - load average: 2.82, 3.30, 2.57 [00:19:59] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.49, 1.77, 1.44 [00:20:39] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.01, 1.66, 1.83 [00:22:39] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.51, 1.64, 1.80 [00:23:45] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.45, 11.59, 10.94 [00:23:55] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.14, 1.62, 1.46 [00:26:39] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.07, 1.33, 1.63 [00:27:45] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.38, 11.32, 11.01 [00:27:53] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.99, 9.93, 8.75 [00:29:53] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.16, 9.21, 8.63 [00:31:57] PROBLEM - es101 Current Load on es101 is WARNING: WARNING - load average: 3.52, 3.16, 2.78 [00:33:01] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:33:57] RECOVERY - es101 Current Load on es101 is OK: OK - load average: 3.05, 3.15, 2.83 [00:35:45] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.10, 11.12, 10.92 [00:37:45] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.64, 10.68, 10.79 [00:38:40] PROBLEM - es121 Disk Space on es121 is WARNING: DISK WARNING - free space: / 28429 MB (10% inode=99%); [00:39:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:41:00] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.28, 10.45, 9.16 [00:42:48] PROBLEM - cloud10 Puppet on cloud10 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [00:43:01] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.02, 10.26, 9.25 [00:44:22] PROBLEM - es121 Current Load on es121 is CRITICAL: CRITICAL - load average: 4.19, 3.32, 2.60 [00:45:01] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.81, 9.35, 9.04 [00:46:21] PROBLEM - es121 Current Load on es121 is WARNING: WARNING - load average: 3.48, 3.34, 2.70 [00:47:45] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.70, 11.05, 10.79 [00:48:21] RECOVERY - es121 Current Load on es121 is OK: OK - load average: 3.14, 3.23, 2.74 [00:49:45] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.46, 10.70, 10.69 [00:49:57] PROBLEM - es101 Current Load on es101 is WARNING: WARNING - load average: 3.80, 3.55, 3.11 [00:51:07] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.28, 1.99, 1.69 [00:51:45] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.96, 11.36, 10.92 [00:51:49] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.93, 1.70, 1.46 [00:51:57] RECOVERY - es101 Current Load on es101 is OK: OK - load average: 2.87, 3.35, 3.09 [00:53:04] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.63, 1.84, 1.67 [00:53:46] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.12, 1.98, 1.60 [00:54:36] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:55:00] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.28, 1.64, 1.61 [00:55:43] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.12, 1.63, 1.52 [00:56:36] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14634 bytes in 2.535 second response time [00:57:10] PROBLEM - es121 Current Load on es121 is WARNING: WARNING - load average: 3.52, 3.39, 3.03 [00:57:23] PROBLEM - es101 Disk Space on es101 is WARNING: DISK WARNING - free space: / 28555 MB (10% inode=99%); [00:57:45] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.32, 11.79, 11.30 [01:04:40] RECOVERY - es121 Disk Space on es121 is OK: DISK OK - free space: / 46303 MB (17% inode=99%); [01:04:59] RECOVERY - es121 Current Load on es121 is OK: OK - load average: 3.04, 3.30, 3.15 [01:07:45] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.16, 12.03, 11.52 [01:07:57] PROBLEM - es101 Current Load on es101 is WARNING: WARNING - load average: 3.63, 3.33, 3.07 [01:09:45] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.07, 11.66, 11.45 [01:09:53] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.62, 3.67, 3.99 [01:10:36] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 1.83, 3.27, 3.91 [01:10:39] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.85, 1.70, 1.61 [01:10:47] RECOVERY - cloud10 Puppet on cloud10 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:11:45] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.61, 12.12, 11.65 [01:11:53] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.06, 3.80, 4.00 [01:11:57] RECOVERY - es101 Current Load on es101 is OK: OK - load average: 3.27, 3.36, 3.14 [01:13:45] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.28, 11.63, 11.54 [01:14:39] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.28, 1.84, 1.68 [01:15:53] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.73, 3.86, 3.98 [01:15:57] PROBLEM - es101 Current Load on es101 is CRITICAL: CRITICAL - load average: 5.31, 4.29, 3.56 [01:16:08] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.76, 1.63, 1.44 [01:17:53] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.11, 3.82, 3.94 [01:18:05] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.51, 1.54, 1.42 [01:18:36] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.01, 2.52, 3.32 [01:19:53] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.89, 3.52, 3.82 [01:21:22] RECOVERY - es101 Disk Space on es101 is OK: DISK OK - free space: / 49160 MB (18% inode=99%); [01:23:53] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.83, 4.08, 3.97 [01:23:57] PROBLEM - es101 Current Load on es101 is WARNING: WARNING - load average: 3.36, 3.84, 3.67 [01:24:39] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.77, 1.94, 1.88 [01:27:25] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.39, 3.56, 3.49 [01:27:45] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.76, 11.05, 10.87 [01:27:53] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.74, 3.92, 3.98 [01:28:17] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/88c3b1ad18da...17304ab64c10 [01:28:18] [02miraheze/ImportDump] 07Universal-Omega 0317304ab - Improve i18n [01:28:20] [02ImportDump] 07Universal-Omega synchronize pull request 03#1: Create extension - 13https://github.com/miraheze/ImportDump/pull/1 [01:29:53] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.47, 4.15, 4.06 [01:30:39] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.12, 1.93, 1.90 [01:30:44] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.73, 1.75, 1.56 [01:31:45] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.92, 11.04, 10.94 [01:32:39] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.76, 1.81, 1.86 [01:32:41] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.19, 1.52, 1.50 [01:33:11] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/17304ab64c10...2a0d40dfdc78 [01:33:12] [02miraheze/ImportDump] 07Universal-Omega 032a0d40d - Update ImportDumpRequestManager.php [01:33:14] [02ImportDump] 07Universal-Omega synchronize pull request 03#1: Create extension - 13https://github.com/miraheze/ImportDump/pull/1 [01:33:57] RECOVERY - es101 Current Load on es101 is OK: OK - load average: 2.94, 3.11, 3.39 [01:35:04] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/2a0d40dfdc78...e064164cc186 [01:35:06] [02miraheze/ImportDump] 07Universal-Omega 03e064164 - Update en.json [01:35:07] [02ImportDump] 07Universal-Omega synchronize pull request 03#1: Create extension - 13https://github.com/miraheze/ImportDump/pull/1 [01:36:32] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.25, 1.86, 1.64 [01:38:30] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.14, 1.56, 1.55 [01:38:51] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.22, 3.79, 3.63 [01:40:39] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.20, 1.46, 1.67 [01:40:46] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.63, 3.84, 3.68 [01:42:42] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.12, 10.56, 9.87 [01:45:45] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.62, 9.14, 10.09 [01:45:54] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.29, 3.53, 3.90 [01:46:30] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 14.48, 11.52, 10.32 [01:47:02] miraheze/ImportDump - Universal-Omega the build passed. [01:48:25] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.03, 9.67, 9.78 [01:50:25] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - dnd.bellinrattin.it All nameservers failed to answer the query. [01:50:39] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.29, 1.87, 1.74 [01:51:45] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.40, 10.14, 10.01 [01:51:53] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.03, 2.64, 3.38 [01:52:39] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.27, 1.64, 1.67 [01:53:45] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.73, 10.68, 10.24 [01:56:36] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.29, 2.95, 3.33 [01:57:45] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.83, 10.11, 10.10 [01:59:34] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/e064164cc186...e358719c7210 [01:59:36] [02miraheze/ImportDump] 07Universal-Omega 03e358719 - Update ImportDumpRequestViewer.php [01:59:38] [02ImportDump] 07Universal-Omega synchronize pull request 03#1: Create extension - 13https://github.com/miraheze/ImportDump/pull/1 [02:00:36] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.44, 4.32, 3.80 [02:02:36] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.90, 3.71, 3.64 [02:03:39] PROBLEM - ping6 on ns1 is WARNING: PING WARNING - Packet loss = 0%, RTA = 123.87 ms [02:03:39] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/e358719c7210...1a2d4058120f [02:03:40] [02miraheze/ImportDump] 07Universal-Omega 031a2d405 - Update en.json [02:03:42] [02ImportDump] 07Universal-Omega synchronize pull request 03#1: Create extension - 13https://github.com/miraheze/ImportDump/pull/1 [02:04:37] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.93, 4.26, 3.83 [02:05:16] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/1a2d4058120f...603cff0a8886 [02:05:18] [02miraheze/ImportDump] 07Universal-Omega 03603cff0 - Update qqq.json [02:05:19] [02ImportDump] 07Universal-Omega synchronize pull request 03#1: Create extension - 13https://github.com/miraheze/ImportDump/pull/1 [02:07:45] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.45, 10.60, 10.22 [02:07:47] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.89, 4.11, 3.60 [02:11:45] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.64, 3.25, 3.40 [02:12:36] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.05, 3.60, 3.74 [02:13:45] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.01, 9.46, 9.89 [02:13:51] RECOVERY - ping6 on ns1 is OK: PING OK - Packet loss = 0%, RTA = 115.82 ms [02:14:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:14:59] miraheze/ImportDump - Universal-Omega the build passed. [02:18:41] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.22, 1.73, 1.47 [02:20:21] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.21, 1.76, 1.51 [02:20:37] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.39, 2.61, 3.25 [02:22:21] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.19, 3.23, 3.93 [02:22:39] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.76, 1.85, 1.59 [02:24:15] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.30, 1.64, 1.52 [02:25:58] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/603cff0a8886...9fcbdb47979d [02:25:59] [02miraheze/ImportDump] 07Universal-Omega 039fcbdb4 - Update SpecialImportDumpRequestQueue.php [02:26:01] [02ImportDump] 07Universal-Omega synchronize pull request 03#1: Create extension - 13https://github.com/miraheze/ImportDump/pull/1 [02:26:39] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.44, 1.61, 1.54 [02:27:26] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/9fcbdb47979d...6bdb8fb1782e [02:27:27] [02miraheze/ImportDump] 07Universal-Omega 036bdb8fb - Update SpecialRequestImportDump.php [02:27:29] [02ImportDump] 07Universal-Omega synchronize pull request 03#1: Create extension - 13https://github.com/miraheze/ImportDump/pull/1 [02:28:36] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+1/-1/±0] 13https://github.com/miraheze/ImportDump/compare/6bdb8fb1782e...d9fdb4547148 [02:28:38] [02miraheze/ImportDump] 07Universal-Omega 03d9fdb45 - Update and rename includes/SpecialRequestImportDump.php to includes/Specials/SpecialRequestImportDump.php [02:28:40] [02ImportDump] 07Universal-Omega synchronize pull request 03#1: Create extension - 13https://github.com/miraheze/ImportDump/pull/1 [02:29:45] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.02, 10.52, 10.04 [02:29:57] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+1/-1/±0] 13https://github.com/miraheze/ImportDump/compare/d9fdb4547148...5afb9aff16a0 [02:29:58] [02miraheze/ImportDump] 07Universal-Omega 035afb9af - Update and rename includes/SpecialImportDumpRequestQueue.php to includes/Specials/SpecialImportDumpRequestQueue.php [02:30:00] [02ImportDump] 07Universal-Omega synchronize pull request 03#1: Create extension - 13https://github.com/miraheze/ImportDump/pull/1 [02:30:47] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/5afb9aff16a0...2ed65ba44ef0 [02:30:48] [02miraheze/ImportDump] 07Universal-Omega 032ed65ba - Update extension.json [02:30:50] [02ImportDump] 07Universal-Omega synchronize pull request 03#1: Create extension - 13https://github.com/miraheze/ImportDump/pull/1 [02:31:45] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.24, 10.13, 9.97 [02:32:21] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.60, 3.91, 3.78 [02:33:25] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/2ed65ba44ef0...2e537a97e695 [02:33:27] [02miraheze/ImportDump] 07Universal-Omega 032e537a9 - Update SpecialImportDumpRequestQueue.php [02:33:28] [02ImportDump] 07Universal-Omega synchronize pull request 03#1: Create extension - 13https://github.com/miraheze/ImportDump/pull/1 [02:34:21] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.97, 3.56, 3.67 [02:34:24] miraheze/ImportDump - Universal-Omega the build has errored. [02:35:45] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.75, 10.40, 10.11 [02:39:45] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 14.69, 11.75, 10.64 [02:41:29] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.15, 3.51, 3.09 [02:42:09] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/2e537a97e695...d4cf4c09f068 [02:42:11] [02miraheze/ImportDump] 07Universal-Omega 03d4cf4c0 - Update ImportDumpRequestViewer.php [02:42:12] [02ImportDump] 07Universal-Omega synchronize pull request 03#1: Create extension - 13https://github.com/miraheze/ImportDump/pull/1 [02:43:02] miraheze/ImportDump - Universal-Omega the build passed. [02:43:45] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.25, 10.60, 10.50 [02:45:27] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.82, 3.74, 3.27 [02:45:45] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.38, 9.58, 10.14 [02:47:49] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.45, 3.87, 3.42 [02:48:25] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp21.miraheze.org - CNAME OK [02:49:43] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.27, 3.28, 3.26 [02:50:39] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.03, 1.89, 1.66 [02:51:24] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.12, 3.04, 3.18 [02:51:52] miraheze/ImportDump - Universal-Omega the build passed. [02:52:39] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.80, 1.83, 1.66 [02:53:06] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/d4cf4c09f068...49df45893f72 [02:53:07] [02miraheze/ImportDump] 07Universal-Omega 0349df458 - Drop request_file [02:53:09] [02ImportDump] 07Universal-Omega synchronize pull request 03#1: Create extension - 13https://github.com/miraheze/ImportDump/pull/1 [02:54:21] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.25, 3.62, 3.55 [02:54:33] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/49df45893f72...9325723a9196 [02:54:35] [02miraheze/ImportDump] 07Universal-Omega 039325723 - Drop request_file [02:54:36] [02ImportDump] 07Universal-Omega synchronize pull request 03#1: Create extension - 13https://github.com/miraheze/ImportDump/pull/1 [02:55:09] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/9325723a9196...e25a447b3590 [02:55:11] [02miraheze/ImportDump] 07Universal-Omega 03e25a447 - Drop request_file [02:55:12] [02ImportDump] 07Universal-Omega synchronize pull request 03#1: Create extension - 13https://github.com/miraheze/ImportDump/pull/1 [02:57:45] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.43, 11.04, 10.40 [02:59:15] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.14, 3.75, 3.41 [02:59:19] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.34, 3.62, 3.31 [02:59:45] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.73, 10.31, 10.20 [03:01:10] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.98, 3.41, 3.32 [03:01:18] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.82, 3.35, 3.25 [03:03:04] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.11, 3.16, 3.25 [03:03:45] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.84, 10.16, 10.15 [03:04:21] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.72, 3.78, 3.82 [03:05:28] miraheze/ImportDump - Universal-Omega the build passed. [03:07:45] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.19, 10.49, 10.37 [03:13:21] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.97, 1.80, 1.46 [03:14:21] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.89, 3.90, 3.74 [03:15:21] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.11, 1.52, 1.39 [03:15:45] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.13, 9.15, 9.86 [03:16:21] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.88, 3.77, 3.71 [03:18:21] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.60, 4.41, 3.95 [03:18:39] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.45, 2.02, 1.86 [03:20:21] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.98, 3.86, 3.80 [03:21:21] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.49, 1.76, 1.48 [03:21:37] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/e25a447b3590...48e1f792eaa9 [03:21:39] [02miraheze/ImportDump] 07Universal-Omega 0348e1f79 - SECURITY: Additional permission checks for sending notifications [03:21:40] [02ImportDump] 07Universal-Omega synchronize pull request 03#1: Create extension - 13https://github.com/miraheze/ImportDump/pull/1 [03:22:39] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.36, 1.86, 1.86 [03:23:21] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.87, 1.79, 1.53 [03:23:26] [02miraheze/ImportDump] 07github-actions[bot] pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/48e1f792eaa9...8829d5c84c94 [03:23:27] [02miraheze/ImportDump] 07github-actions 038829d5c - CI: lint code to MediaWiki standards [03:23:29] [02ImportDump] 07github-actions[bot] synchronize pull request 03#1: Create extension - 13https://github.com/miraheze/ImportDump/pull/1 [03:24:21] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.41, 3.95, 3.79 [03:25:11] [02ImportDump] 07Universal-Omega edited pull request 03#1: Create extension - 13https://github.com/miraheze/ImportDump/pull/1 [03:25:21] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.66, 1.41, 1.42 [03:26:21] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.75, 3.88, 3.79 [03:26:39] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.15, 1.47, 1.69 [03:27:03] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.71, 3.74, 3.42 [03:27:06] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.17, 3.63, 3.39 [03:28:57] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.51, 3.35, 3.32 [03:29:01] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:29:05] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.56, 3.02, 3.20 [03:32:45] miraheze/ImportDump - Universal-Omega the build passed. [03:33:21] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.14, 9.95, 8.72 [03:35:20] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.21, 9.19, 8.59 [03:37:45] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.05, 10.46, 9.98 [03:38:21] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.76, 2.63, 3.22 [03:43:45] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.54, 9.44, 9.70 [03:44:01] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:44:21] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.45, 4.04, 3.65 [03:47:55] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.56, 4.01, 3.30 [03:48:36] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.61, 3.78, 3.42 [03:49:54] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.51, 3.44, 3.18 [03:50:39] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.25, 1.76, 1.52 [03:51:53] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.17, 3.32, 3.16 [03:52:39] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.30, 1.55, 1.47 [03:54:37] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.65, 3.34, 3.35 [03:57:53] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.50, 3.57, 3.31 [03:59:53] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.70, 3.12, 3.17 [04:00:21] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.03, 3.83, 3.94 [04:02:08] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 0.82, 2.01, 1.54 [04:04:06] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.01, 1.64, 1.46 [04:08:21] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.73, 2.84, 3.37 [04:12:21] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.93, 3.46, 3.56 [04:19:10] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.64, 1.83, 1.53 [04:20:21] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.03, 2.72, 3.21 [04:23:03] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.58, 1.76, 1.59 [04:23:53] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.43, 2.91, 2.83 [04:25:00] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.27, 1.60, 1.54 [04:25:45] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.90, 10.15, 9.34 [04:25:53] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.83, 2.52, 2.70 [04:29:45] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.04, 9.51, 9.29 [04:39:38] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.84, 1.69, 1.60 [04:43:20] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.78, 4.05, 3.39 [04:44:01] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.76, 3.70, 3.32 [04:44:45] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.62, 3.73, 3.30 [04:45:27] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.26, 1.84, 1.69 [04:45:58] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.91, 4.55, 3.67 [04:47:55] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.71, 3.97, 3.58 [04:49:03] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.37, 3.42, 3.35 [04:49:52] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.71, 3.30, 3.39 [04:50:25] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 73, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') File "/usr/l [04:50:25] n3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query bellinrattin.it. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [04:50:43] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.15, 3.00, 3.18 [04:50:57] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.71, 2.82, 3.14 [04:52:22] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 16.98, 12.90, 10.57 [04:53:01] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:53:37] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:53:44] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:53:44] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:53:49] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:55:10] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.84, 2.00, 1.90 [04:55:34] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14640 bytes in 0.329 second response time [04:55:40] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 22334 bytes in 0.292 second response time [04:55:42] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14640 bytes in 0.336 second response time [04:55:47] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14640 bytes in 0.016 second response time [04:56:24] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 14.55, 11.93, 9.02 [04:58:22] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.57, 10.32, 8.80 [05:00:08] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.32, 11.17, 10.90 [05:00:20] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.19, 9.33, 8.62 [05:00:36] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.53, 3.35, 3.17 [05:01:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.24, 3.42, 3.36 [05:02:37] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 3.04, 3.25, 3.16 [05:03:01] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:05:57] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.37, 11.15, 10.87 [05:06:36] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.03, 3.94, 3.46 [05:07:22] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.69, 3.08, 3.24 [05:07:54] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.20, 9.57, 10.32 [05:08:36] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.85, 3.43, 3.33 [05:08:46] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.25, 1.48, 1.67 [05:09:50] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.37, 8.52, 9.85 [05:10:36] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.78, 3.27, 3.28 [05:18:39] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.21, 1.67, 1.61 [05:20:39] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.62, 1.65, 1.62 [05:26:47] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.55, 3.53, 3.31 [05:27:08] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.52, 3.52, 3.27 [05:28:10] [02miraheze/MirahezeMagic] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/MirahezeMagic/compare/050420fcaf8c...c6d5234a170d [05:28:12] [02miraheze/MirahezeMagic] 07Reception123 03c6d5234 - change message for recaptcha errors [05:28:42] !log [reception@mwtask111] starting deploy of {'world': True, 'l10n': True} to all [05:28:44] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.38, 3.34, 3.25 [05:28:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:29:03] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.24, 2.98, 3.09 [05:30:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:34:32] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.58, 3.59, 3.45 [05:35:45] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.00, 10.49, 9.40 [05:36:29] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.83, 3.37, 3.38 [05:37:06] miraheze/MirahezeMagic - Reception123 the build passed. [05:38:33] !log [reception@mwtask111] finished deploy of {'world': True, 'l10n': True} to all - SUCCESS in 590s [05:38:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:39:45] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.23, 10.72, 9.76 [05:40:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:41:45] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.17, 10.06, 9.63 [05:50:39] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.29, 1.80, 1.49 [05:52:22] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/7be37672df78 [05:52:24] [02miraheze/mw-config] 07Universal-Omega 037be3767 - Update LocalSettings.php [05:52:25] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-2 - 13https://github.com/miraheze/mw-config [05:52:39] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.65, 1.74, 1.51 [05:54:34] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=sotfwiki /home/reception/sotf_pages_full.xml --report 1 --no-updates --username-prefix=wikia:sotf (START) [05:54:39] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.39, 1.61, 1.49 [05:54:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:55:53] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/7be37672df78...c3f6f43c5510 [05:55:54] [02miraheze/mw-config] 07Universal-Omega 03c3f6f43 - Update ManageWikiSettings.php [05:59:16] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/c3f6f43c5510...164ca3738919 [05:59:17] [02miraheze/mw-config] 07Universal-Omega 03164ca37 - Update Defines.php [05:59:35] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=criticalrolewiki /home/reception/criticalrole_pages_full.xml --report 1 --no-updates --username-prefix=wikia:criticalrole (START) [05:59:36] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.56, 4.77, 3.64 [05:59:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:00:36] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/164ca3738919...c0991e35d68d [06:00:37] [02miraheze/mw-config] 07Universal-Omega 03c0991e3 - Update Defines.php [06:01:15] [02mw-config] 07Universal-Omega opened pull request 03#4649: Add `$wgUploadWizardConfig['flickrApiKey']` to ManageWikiSettings - 13https://github.com/miraheze/mw-config/pull/4649 [06:02:50] miraheze/mw-config - Universal-Omega the build passed. [06:03:25] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 1.95, 3.43, 3.36 [06:03:36] [02ssl] 07AgentIsai opened pull request 03#507: Redirect mario.* to custom domain - 13https://github.com/miraheze/ssl/pull/507 [06:05:05] [02mw-config] 07Universal-Omega commented on pull request 03#4624: add UploadWizard Flickr configs for gpcommonswiki - 13https://github.com/miraheze/mw-config/pull/4624#issuecomment-1111782396 [06:05:07] [02mw-config] 07Universal-Omega closed pull request 03#4624: add UploadWizard Flickr configs for gpcommonswiki - 13https://github.com/miraheze/mw-config/pull/4624 [06:07:14] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.79, 2.88, 3.19 [06:08:27] [02ssl] 07AgentIsai synchronize pull request 03#507: Redirect mario.* to custom domain - 13https://github.com/miraheze/ssl/pull/507 [06:08:39] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 3.34, 2.72, 1.99 [06:10:02] miraheze/ssl - AgentIsai the build has errored. [06:10:42] [02ssl] 07AgentIsai synchronize pull request 03#507: Redirect mario.* to custom domain - 13https://github.com/miraheze/ssl/pull/507 [06:12:29] [02mw-config] 07Reception123 closed pull request 03#4649: Add `$wgUploadWizardConfig['flickrApiKey']` to ManageWikiSettings - 13https://github.com/miraheze/mw-config/pull/4649 [06:12:31] [02miraheze/mw-config] 07Reception123 deleted branch 03Universal-Omega-patch-2 [06:12:32] [02mw-config] 07Reception123 deleted branch 03Universal-Omega-patch-2 - 13https://github.com/miraheze/mw-config [06:12:34] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±3] 13https://github.com/miraheze/mw-config/compare/93a089e43757...59d3dbf719f3 [06:12:35] [02miraheze/mw-config] 07Universal-Omega 0359d3dbf - Add `$wgUploadWizardConfig['flickrApiKey']` to ManageWikiSettings (#4649) [06:12:51] !log [reception@mwtask111] starting deploy of {'config': True} to all [06:12:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:13:00] !log [reception@mwtask111] finished deploy of {'config': True} to all - SUCCESS in 9s [06:13:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:13:45] miraheze/mw-config - Reception123 the build passed. [06:17:34] [02mw-config] 07Universal-Omega closed pull request 03#4647: T9149: add "bw" to wgExtraLanguageNames for wikibenwiki - 13https://github.com/miraheze/mw-config/pull/4647 [06:17:36] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/59d3dbf719f3...51884a1a20e3 [06:17:37] [02miraheze/mw-config] 07ugochimobi 0351884a1 - T9149: add "bw" to wgExtraLanguageNames for wikibenwiki (#4647) [06:17:53] [02ssl] 07Reception123 closed pull request 03#507: Redirect mario.* to custom domain - 13https://github.com/miraheze/ssl/pull/507 [06:17:54] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/3aa7cf54f884...6d45e789cbdc [06:17:55] [02miraheze/ssl] 07AgentIsai 036d45e78 - Redirect mario.* to custom domain (#507) [06:18:41] miraheze/mw-config - Universal-Omega the build passed. [06:20:37] CosmicAlpha: why are we editing that [06:22:32] [02mw-config] 07RhinosF1 commented on pull request 03#4639: set wgExtraLanguageNames to "qqq" for gpcommonswiki and gratisdatawiki - 13https://github.com/miraheze/mw-config/pull/4639#issuecomment-1111793357 [06:23:13] !log [@test101] starting deploy of {'config': True} to all [06:23:14] !log [@test101] finished deploy of {'config': True} to all - SUCCESS in 1s [06:23:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:23:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:23:49] !log [@mwtask111] starting deploy of {'config': True} to all [06:23:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:23:59] !log [@mwtask111] finished deploy of {'config': True} to all - SUCCESS in 9s [06:24:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:24:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:28:07] Agent: lol, you conflicted with me on Phabricator 2 times (beat me to it), one for tags, one for closing a task... weird timing that we both went to do the same things on older tasks at the same time... [06:28:33] RhinosF1: editing what? [06:28:54] CosmicAlpha: extra language names [06:29:04] [02mw-config] 07ugochimobi commented on pull request 03#4639: set wgExtraLanguageNames to "qqq" for gpcommonswiki and gratisdatawiki - 13https://github.com/miraheze/mw-config/pull/4639#issuecomment-1111797606 [06:29:25] [02mw-config] 07RhinosF1 commented on pull request 03#4639: set wgExtraLanguageNames to "qqq" for gpcommonswiki and gratisdatawiki - 13https://github.com/miraheze/mw-config/pull/4639#issuecomment-1111797861 [06:29:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:29:42] What's the harm in editing it RhinosF1? [06:30:21] CosmicAlpha: if people want to make up languages I kinda get it but qqq which I saw makes 0 sense [06:30:48] How does it behave with global prefs [06:31:04] CosmicAlpha: lol yeah, weird timing [06:31:10] Because if you can set via global prefs on that wiki then there needs to be a fallback [06:34:11] RhinosF1: I mostly just wanted the tasks resolved, and PRs cleared, I'm kinda through worrying about the status and cleaness of our config, as it just became a mess when I was away for two months, so I gave up with it. I'm not to worried about what people do anymore on their config. I gave up on that. I tried for a year and a half to keep our config pretty clean, and most stuff in ManageWiki rather than our config, but now it [06:34:11] just seems a bit pointless, so I don't care anymore. If they have issues with them on their wikis because of config changes they make that is their problems, and if there were fatal, that's why I'm around to revert if necessary, but I can't account for everything. [06:34:41] CosmicAlpha: okay [06:36:36] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.12, 10.16, 9.47 [06:38:31] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.16, 9.75, 9.40 [06:43:00] [02ssl] 07AgentIsai opened pull request 03#508: T9129: Potential fix - 13https://github.com/miraheze/ssl/pull/508 [06:45:46] [02ssl] 07Reception123 closed pull request 03#508: T9129: Potential fix - 13https://github.com/miraheze/ssl/pull/508 [06:45:48] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/6d45e789cbdc...3e3862a33e50 [06:45:49] [02miraheze/ssl] 07AgentIsai 033e3862a - T9129: Potential fix (#508) [06:47:51] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=sotfwiki /home/reception/sotf_pages_full.xml --report 1 --no-updates --username-prefix=wikia:sotf (END - exit=0) [06:47:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:48:25] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp21.miraheze.org - CNAME OK [07:02:52] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.66, 3.16, 2.78 [07:08:47] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.40, 3.06, 2.88 [07:18:01] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.30, 3.22, 2.77 [07:18:37] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.81, 3.90, 3.29 [07:20:00] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.80, 3.46, 2.77 [07:30:00] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.44, 3.86, 3.60 [07:30:34] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.61, 3.97, 3.92 [07:32:35] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 3.77, 4.17, 4.02 [07:33:35] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.43, 3.74, 3.66 [07:34:34] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.30, 3.85, 3.91 [07:40:34] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.27, 3.86, 3.86 [07:42:34] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.00, 3.23, 3.63 [07:44:34] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.55, 3.58, 3.70 [07:45:14] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.49, 2.92, 3.31 [07:46:34] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.99, 3.43, 3.63 [07:48:01] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 3.15, 3.10, 3.38 [07:51:22] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.22, 10.53, 9.66 [07:53:17] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.58, 10.50, 9.75 [07:54:34] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.21, 2.97, 3.36 [07:55:12] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.47, 9.83, 9.60 [07:58:01] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 7.74, 4.12, 3.46 [08:00:35] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 3.84, 4.02, 3.70 [08:00:51] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.50, 3.72, 3.39 [08:02:00] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.14, 3.70, 3.46 [08:02:34] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.35, 3.70, 3.62 [08:02:51] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.42, 3.16, 3.22 [08:04:01] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.34, 3.90, 3.56 [08:06:01] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.44, 3.55, 3.46 [08:08:00] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.34, 3.17, 3.33 [08:08:34] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.80, 2.87, 3.35 [08:16:34] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.28, 3.53, 3.38 [08:18:34] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.99, 3.15, 3.25 [08:22:34] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.51, 3.89, 3.52 [08:24:34] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.96, 3.18, 3.31 [09:18:50] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=sotfwiki (START) [09:18:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:19:23] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=criticalrolewiki /home/reception/criticalrole_pages_full.xml --report 1 --no-updates --username-prefix=wikia:criticalrole (START) [09:19:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:21:25] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.58, 1.76, 1.98 [09:22:45] !log ALTER TABLE searchindex ADD FULLTEXT INDEX si_title (si_title), ADD FULLTEXT INDEX si_text (si_text); (mgqwiki) [09:22:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:24:34] PROBLEM - wiki.thunis.eu - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 73, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') File "/usr/lib/py [09:24:34] st-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query thunis.eu. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [09:26:22] PROBLEM - wiki.mineland.eu - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mineland.eu All nameservers failed to answer the query. [09:29:44] PROBLEM - fc.songcontests.eu - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 86, in check_records cname = str(dns_resolver.resolve(hostname, 'CNAME')[0]) File "/usr/lib [09:29:44] /dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query fc.songcontests.eu. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [09:37:25] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.40, 1.52, 1.67 [09:38:28] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=criticalrolewiki /home/reception/criticalrole_pages_full.xml --report 1 --no-updates --username-prefix=wikia:criticalrole (END - exit=0) [09:38:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:39:54] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=sotfwiki (END - exit=0) [09:40:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:40:28] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=criticalrolewiki (START) [09:40:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:45:37] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - dnd.bellinrattin.it All nameservers failed to answer the query. [09:47:25] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.09, 1.73, 1.67 [09:51:25] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.94, 1.97, 1.80 [09:53:38] RECOVERY - wiki.thunis.eu - reverse DNS on sslhost is OK: SSL OK - wiki.thunis.eu reverse DNS resolves to cp20.miraheze.org - CNAME OK [09:54:01] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.79, 3.42, 2.74 [09:56:01] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.87, 3.08, 2.69 [09:58:49] !log [reception@mwtask111] starting deploy of {'config': True, 'l10n': True} to all [09:58:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:00:21] !log [reception@mwtask111] finished deploy of {'config': True, 'l10n': True} to all - SUCCESS in 92s [10:00:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:13:09] PROBLEM - es111 Disk Space on es111 is WARNING: DISK WARNING - free space: / 28375 MB (10% inode=99%); [10:13:25] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.39, 1.56, 1.70 [10:14:14] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp20.miraheze.org - CNAME OK [10:15:41] [02miraheze/mediawiki] 07Reception123 pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/da6d4db96d3d...6c4954c7b2de [10:15:43] [02miraheze/mediawiki] 07Reception123 036c4954c - Update MirahezeMagic [10:15:44] !log [reception@mwtask111] starting deploy of {'config': True, 'l10n': True} to all [10:15:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:17:09] RECOVERY - es111 Disk Space on es111 is OK: DISK OK - free space: / 30008 MB (11% inode=99%); [10:17:18] !log [reception@mwtask111] finished deploy of {'config': True, 'l10n': True} to all - SUCCESS in 93s [10:17:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:19:16] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.55, 1.72, 1.36 [10:19:25] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.53, 1.98, 1.82 [10:21:09] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.04, 1.50, 1.32 [10:21:25] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.55, 1.85, 1.79 [10:23:25] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.17, 1.58, 1.70 [10:25:09] PROBLEM - es111 Disk Space on es111 is WARNING: DISK WARNING - free space: / 28635 MB (10% inode=99%); [10:25:18] RECOVERY - wiki.mineland.eu - reverse DNS on sslhost is OK: SSL OK - wiki.mineland.eu reverse DNS resolves to cp21.miraheze.org - CNAME OK [10:27:14] RECOVERY - fc.songcontests.eu - reverse DNS on sslhost is OK: SSL OK - fc.songcontests.eu reverse DNS resolves to cp20.miraheze.org - CNAME OK [10:29:42] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.02, 3.77, 3.17 [10:31:39] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.33, 3.30, 3.07 [10:34:29] !log [reception@mwtask111] starting deploy of {'world': True, 'l10n': True, 'ignoretime': True} to all [10:34:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:39:09] RECOVERY - es111 Disk Space on es111 is OK: DISK OK - free space: / 38655 MB (14% inode=99%); [10:42:20] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 5.21, 3.82, 2.84 [10:46:00] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 8.59, 5.49, 3.77 [10:46:34] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.30, 3.37, 2.89 [10:50:35] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.54, 3.20, 2.95 [10:52:12] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/findBadBlobs.php --wiki=trollpastawiki --revisions (END - exit=256) [10:52:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:52:20] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 2.95, 3.90, 3.60 [10:52:55] !log reception@mwtask111:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/findBadBlobs.php --wiki=trollpastawiki --scan-from 20200101000000 [10:52:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:54:01] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.46, 3.50, 3.62 [10:58:20] PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 4.04, 3.81, 3.63 [11:00:20] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 3.93, 3.88, 3.68 [11:04:01] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.63, 2.71, 3.25 [11:08:20] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 2.33, 3.02, 3.38 [11:09:51] [02miraheze/landing] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/landing/compare/e8a7264385d4...af2dcd357637 [11:09:53] [02miraheze/landing] 07translatewiki 03af2dcd3 - Localisation updates from https://translatewiki.net. [11:09:54] [02miraheze/IncidentReporting] 07translatewiki pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/IncidentReporting/compare/a7ef0d0c5acd...246c75f8f441 [11:09:56] [02miraheze/IncidentReporting] 07translatewiki 03246c75f - Localisation updates from https://translatewiki.net. [11:09:57] [02miraheze/CreateWiki] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/25158d1cca5f...673266d0a0a2 [11:09:59] [02miraheze/CreateWiki] 07translatewiki 03673266d - Localisation updates from https://translatewiki.net. [11:10:00] [02miraheze/ManageWiki] 07translatewiki pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/ManageWiki/compare/16819d4bfb18...0212fdbc1044 [11:10:02] [02miraheze/ManageWiki] 07translatewiki 030212fdb - Localisation updates from https://translatewiki.net. [11:10:03] [02miraheze/MirahezeMagic] 07translatewiki pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/MirahezeMagic/compare/c6d5234a170d...76b5990660db [11:10:05] [02miraheze/MirahezeMagic] 07translatewiki 0376b5990 - Localisation updates from https://translatewiki.net. [11:12:31] miraheze/landing - translatewiki the build passed. [11:15:39] PROBLEM - www.lab612.at - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 86, in check_records cname = str(dns_resolver.resolve(hostname, 'CNAME')[0]) File "/usr/lib/pyth [11:15:39] -packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query www.lab612.at. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [11:19:25] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.15, 1.84, 1.57 [11:20:52] miraheze/IncidentReporting - translatewiki the build passed. [11:21:25] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.62, 1.78, 1.59 [11:21:25] miraheze/ManageWiki - translatewiki the build passed. [11:21:28] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 13.39, 11.40, 8.71 [11:22:29] [02miraheze/mediawiki] 07Reception123 pushed 031 commit to 03REL1_37 [+1/-0/±1] 13https://github.com/miraheze/mediawiki/compare/6c4954c7b2de...d39953b8cee9 [11:22:31] [02miraheze/mediawiki] 07Reception123 03d39953b - Install InterwikiSorting (T9143) [11:23:25] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.56, 1.70, 1.58 [11:23:28] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.35, 9.70, 8.41 [11:23:41] PROBLEM - wiki.thunis.eu - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.thunis.eu All nameservers failed to answer the query. [11:24:05] miraheze/CreateWiki - translatewiki the build passed. [11:24:48] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.32, 1.62, 1.17 [11:26:10] PROBLEM - wiki.mineland.eu - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mineland.eu All nameservers failed to answer the query. [11:26:21] miraheze/MirahezeMagic - translatewiki the build passed. [11:26:48] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.08, 1.47, 1.18 [11:27:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:29:13] PROBLEM - fc.songcontests.eu - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 73, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') File "/usr/li [11:29:13] 3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query songcontests.eu. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [11:31:45] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03Reception123-patch-1 [+0/-0/±2] 13https://github.com/miraheze/mw-config/commit/13706d6b3e3c [11:31:46] [02miraheze/mw-config] 07Reception123 0313706d6 - Add InterwikiSorting ext. (T9143) [11:31:48] [02mw-config] 07Reception123 created branch 03Reception123-patch-1 - 13https://github.com/miraheze/mw-config [11:31:51] [02mw-config] 07Reception123 opened pull request 03#4650: Add InterwikiSorting ext. (T9143) - 13https://github.com/miraheze/mw-config/pull/4650 [11:32:22] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03Reception123-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/13706d6b3e3c...12f46df036cb [11:32:23] [02miraheze/mw-config] 07Reception123 0312f46df - rm space [11:32:25] [02mw-config] 07Reception123 synchronize pull request 03#4650: Add InterwikiSorting ext. (T9143) - 13https://github.com/miraheze/mw-config/pull/4650 [11:32:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:32:51] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03Reception123-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/12f46df036cb...0e023178923b [11:32:52] [02miraheze/mw-config] 07Reception123 030e02317 - space [11:32:54] [02mw-config] 07Reception123 synchronize pull request 03#4650: Add InterwikiSorting ext. (T9143) - 13https://github.com/miraheze/mw-config/pull/4650 [11:32:56] miraheze/mw-config - Reception123 the build has errored. [11:33:31] miraheze/mw-config - Reception123 the build has errored. [11:33:56] [02mw-config] 07Reception123 synchronize pull request 03#4650: Add InterwikiSorting ext. (T9143) - 13https://github.com/miraheze/mw-config/pull/4650 [11:33:57] miraheze/mw-config - Reception123 the build has errored. [11:33:57] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03Reception123-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/0e023178923b...f9ce452e96ef [11:33:59] [02miraheze/mw-config] 07Reception123 03f9ce452 - syntax [11:34:59] miraheze/mw-config - Reception123 the build passed. [11:35:00] [02mw-config] 07Reception123 commented on pull request 03#4650: Add InterwikiSorting ext. (T9143) - 13https://github.com/miraheze/mw-config/pull/4650#issuecomment-1112097281 [11:36:34] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 3.77, 3.64, 3.38 [11:37:09] !log [reception@mwtask111] finished deploy of {'world': True, 'l10n': True, 'ignoretime': True} to all - SUCCESS in 3759s [11:37:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:37:53] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.71, 1.68, 1.63 [11:38:29] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 2.80, 3.37, 3.31 [11:39:48] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.70, 1.69, 1.64 [11:45:17] PROBLEM - www.lab612.at - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - www.lab612.at All nameservers failed to answer the query. [11:45:33] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.10, 1.80, 1.69 [11:45:37] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 86, in check_records cname = str(dns_resolver.resolve(hostname, 'CNAME')[0]) File "/usr/li [11:45:37] 3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query dnd.bellinrattin.it. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [11:46:33] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.42, 3.52, 3.18 [11:47:29] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.87, 3.76, 3.26 [11:48:28] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 3.10, 3.33, 3.15 [11:49:28] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.25, 4.03, 3.42 [11:51:26] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.25, 3.75, 3.39 [11:55:23] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.68, 3.55, 3.35 [11:55:25] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.80, 1.92, 1.87 [11:55:38] PROBLEM - wiki.mineland.eu - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 73, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') File "/usr/lib/ [11:55:38] dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query mineland.eu. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [11:57:21] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.10, 3.22, 3.24 [11:57:56] RECOVERY - fc.songcontests.eu - reverse DNS on sslhost is OK: SSL OK - fc.songcontests.eu reverse DNS resolves to cp21.miraheze.org - CNAME OK [12:00:52] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.35, 3.45, 3.19 [12:00:53] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=criticalrolewiki (END - exit=0) [12:00:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:02:20] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.78, 1.83, 1.31 [12:02:46] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.69, 3.21, 3.13 [12:03:25] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.19, 1.38, 1.63 [12:04:20] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.04, 1.52, 1.25 [12:08:30] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.23, 3.50, 3.28 [12:09:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:10:24] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.34, 2.65, 2.99 [12:14:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:14:56] RECOVERY - www.lab612.at - reverse DNS on sslhost is OK: SSL OK - www.lab612.at reverse DNS resolves to cp20.miraheze.org - CNAME OK [12:16:02] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.44, 3.19, 3.12 [12:18:01] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.96, 2.88, 3.03 [12:19:25] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.20, 1.70, 1.56 [12:21:25] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.43, 1.62, 1.55 [12:21:48] RECOVERY - wiki.thunis.eu - reverse DNS on sslhost is OK: SSL OK - wiki.thunis.eu reverse DNS resolves to cp20.miraheze.org - CNAME OK [12:24:43] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.32, 3.52, 3.09 [12:25:51] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.32, 3.55, 3.32 [12:26:37] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.63, 3.24, 3.04 [12:27:50] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.17, 3.12, 3.19 [12:31:45] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.48, 3.94, 3.55 [12:33:43] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.37, 3.33, 3.37 [12:40:36] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.58, 3.50, 3.38 [12:42:35] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.53, 3.15, 3.27 [12:46:34] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.79, 3.79, 3.52 [12:47:25] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.91, 1.56, 1.42 [12:50:35] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.26, 4.15, 3.70 [12:52:31] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.30, 10.26, 9.06 [12:52:39] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.24, 3.48, 3.24 [12:54:34] RECOVERY - wiki.mineland.eu - reverse DNS on sslhost is OK: SSL OK - wiki.mineland.eu reverse DNS resolves to cp21.miraheze.org - CNAME OK [12:54:34] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.94, 3.86, 3.68 [12:55:25] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.45, 1.63, 1.54 [12:56:01] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:56:17] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.56, 3.28, 3.08 [12:56:27] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.50, 4.19, 3.54 [12:58:13] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.32, 3.20, 3.07 [13:00:16] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.37, 3.74, 3.49 [13:00:31] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.93, 9.62, 9.47 [13:00:34] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.62, 4.04, 3.75 [13:01:01] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:02:11] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.35, 3.22, 3.33 [13:02:34] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.26, 3.60, 3.62 [13:04:34] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.72, 3.94, 3.74 [13:06:31] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.71, 11.01, 10.07 [13:08:31] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.91, 11.02, 10.19 [13:08:34] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.81, 3.65, 3.70 [13:09:53] [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/f285967eb5e1...02ad56f9cf5e [13:09:55] [02miraheze/dns] 07Reception123 0302ad56f - add acme challenge for orain.org [13:10:01] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.48, 3.42, 3.32 [13:10:31] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.21, 9.98, 9.91 [13:11:34] [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/02ad56f9cf5e...2fc53f8ff8e1 [13:11:36] [02miraheze/dns] 07Reception123 032fc53f8 - add acme challenge for orain.org [13:12:01] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.29, 3.10, 3.23 [13:12:43] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/3e3862a33e50...d639279a3958 [13:12:45] [02miraheze/ssl] 07Reception123 03d639279 - renew *.orain.org [13:13:12] [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/2fc53f8ff8e1...23ab6a09cb48 [13:13:13] [02miraheze/dns] 07Reception123 0323ab6a0 - add acme challenge for miraheze.wiki [13:13:37] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp21.miraheze.org - CNAME OK [13:14:28] [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/23ab6a09cb48...49f3a65e4348 [13:14:30] [02miraheze/dns] 07Reception123 0349f3a65 - add acme challenge for miraheze.wiki [13:15:37] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/d639279a3958...8c4ff82a34c8 [13:15:38] [02miraheze/ssl] 07Reception123 038c4ff82 - renew *.miraheze.wiki [13:16:34] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.94, 3.99, 3.80 [13:18:34] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.47, 3.50, 3.65 [13:19:25] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.63, 1.97, 1.71 [13:22:34] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.82, 2.70, 3.29 [13:23:25] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.91, 2.00, 1.80 [13:28:10] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.74, 11.03, 10.09 [13:29:24] RECOVERY - miraheze.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'miraheze.wiki' will expire on Wed 27 Jul 2022 12:15:15 GMT +0000. [13:30:05] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.65, 10.22, 9.90 [13:30:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:32:00] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.06, 10.12, 9.91 [13:32:23] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.94, 10.22, 8.44 [13:32:34] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.94, 3.79, 3.47 [13:32:51] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.10, 3.78, 3.14 [13:34:18] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.65, 9.36, 8.34 [13:34:51] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.09, 3.55, 3.14 [13:35:25] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.67, 3.64, 3.24 [13:35:50] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.27, 10.60, 10.16 [13:36:10] RECOVERY - allthetropes.orain.org - LetsEncrypt on sslhost is OK: OK - Certificate 'orain.org' will expire on Wed 27 Jul 2022 12:12:22 GMT +0000. [13:36:35] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.37, 4.00, 3.67 [13:37:19] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.10, 3.68, 3.29 [13:37:46] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.89, 9.76, 9.94 [13:38:34] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.61, 4.30, 3.82 [13:39:14] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.63, 3.17, 3.15 [13:40:26] RECOVERY - isv.orain.org - LetsEncrypt on sslhost is OK: OK - Certificate 'orain.org' will expire on Wed 27 Jul 2022 12:12:22 GMT +0000. [13:40:34] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.23, 3.51, 3.58 [13:40:40] RECOVERY - poserdazfreebies.orain.org - LetsEncrypt on sslhost is OK: OK - Certificate 'orain.org' will expire on Wed 27 Jul 2022 12:12:22 GMT +0000. [13:40:51] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.78, 3.18, 3.13 [13:42:35] RECOVERY - orain.org - LetsEncrypt on sslhost is OK: OK - Certificate 'orain.org' will expire on Wed 27 Jul 2022 12:12:22 GMT +0000. [13:44:34] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.42, 3.92, 3.69 [13:46:52] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.57, 3.44, 3.21 [13:47:25] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.02, 1.73, 1.75 [13:47:46] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.98, 3.97, 3.44 [13:49:42] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.85, 3.64, 3.39 [13:50:40] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.05, 3.69, 3.37 [13:51:25] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.78, 1.98, 1.87 [13:51:39] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.15, 3.13, 3.23 [13:54:29] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.21, 3.83, 3.52 [13:57:25] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.14, 1.41, 1.63 [13:57:55] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.63, 10.36, 9.89 [13:59:50] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.78, 9.76, 9.73 [14:00:34] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.68, 3.60, 3.91 [14:04:01] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.84, 3.14, 3.33 [14:05:36] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.28, 10.87, 10.11 [14:07:31] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.93, 11.03, 10.24 [14:10:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:13:16] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.05, 10.11, 10.13 [14:14:34] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.21, 3.63, 3.66 [14:16:01] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.47, 3.77, 3.42 [14:16:34] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.21, 3.89, 3.79 [14:17:57] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.15, 3.43, 3.33 [14:20:56] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.55, 10.63, 10.17 [14:21:50] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.17, 2.92, 3.16 [14:22:51] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.33, 11.37, 10.48 [14:24:46] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.48, 11.43, 10.61 [14:28:34] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.92, 3.94, 3.72 [14:28:36] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.94, 9.61, 10.05 [14:30:35] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.46, 3.37, 3.54 [14:34:34] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.60, 3.07, 3.39 [14:44:01] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:47:28] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.05, 3.88, 3.50 [14:48:04] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.24, 1.76, 1.52 [14:48:44] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 5.16, 13.64, 7.18 [14:49:26] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.14, 3.27, 3.33 [14:50:31] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 15.93, 11.94, 10.44 [14:50:35] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.48, 9.58, 8.61 [14:51:53] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.95, 1.98, 1.68 [14:52:25] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.25, 1.90, 1.40 [14:52:31] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.37, 11.24, 10.40 [14:52:34] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.18, 8.99, 8.52 [14:54:19] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.86, 1.80, 1.42 [14:55:42] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.43, 1.67, 1.61 [14:58:06] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.02, 1.58, 1.43 [14:58:31] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.52, 11.70, 10.81 [14:59:01] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:00:31] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.99, 11.32, 10.77 [15:01:31] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.80, 10.63, 9.28 [15:03:13] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.76, 3.67, 3.23 [15:03:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:03:31] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.26, 9.20, 8.91 [15:05:11] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.05, 3.57, 3.26 [15:05:25] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.73, 1.64, 1.59 [15:07:09] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.25, 3.63, 3.30 [15:07:25] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.68, 1.65, 1.60 [15:11:06] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.95, 3.40, 3.29 [15:12:36] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 0.24, 0.43, 1.80 [15:13:46] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.24, 3.62, 3.10 [15:14:15] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.71, 1.72, 1.65 [15:14:31] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.03, 9.15, 10.03 [15:14:36] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.74, 0.61, 1.70 [15:15:01] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.15, 3.68, 3.49 [15:15:24] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.04, 10.26, 9.58 [15:15:42] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.24, 3.33, 3.04 [15:17:18] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.61, 9.58, 9.40 [15:18:04] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.15, 1.89, 1.73 [15:18:58] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.91, 4.02, 3.65 [15:22:31] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.10, 10.23, 10.05 [15:23:22] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.50, 3.80, 3.22 [15:23:48] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.69, 1.95, 1.85 [15:24:31] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.68, 10.57, 10.19 [15:25:17] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.87, 3.49, 3.17 [15:27:11] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.85, 3.83, 3.32 [15:28:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:28:49] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.85, 3.80, 3.85 [15:30:31] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.36, 9.87, 10.06 [15:31:59] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 13.27, 10.45, 9.32 [15:32:21] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.88, 3.90, 3.42 [15:32:46] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.40, 4.00, 3.92 [15:33:25] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.50, 1.58, 1.70 [15:33:54] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.26, 10.57, 9.52 [15:34:18] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.87, 3.44, 3.31 [15:34:44] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.06, 3.70, 3.82 [15:34:49] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.87, 3.57, 3.48 [15:35:49] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.34, 9.52, 9.27 [15:36:31] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.69, 11.84, 10.79 [15:36:43] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.56, 4.25, 3.99 [15:36:43] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.26, 3.71, 3.53 [15:38:01] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:38:11] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.78, 3.32, 3.29 [15:38:28] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 1.93, 2.10, 1.52 [15:38:31] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.23, 10.89, 10.55 [15:38:38] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.65, 3.24, 3.37 [15:39:40] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.94, 10.53, 9.76 [15:40:27] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.69, 1.55, 1.39 [15:41:36] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.78, 9.68, 9.53 [15:42:06] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.00, 4.28, 3.67 [15:42:09] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.76, 10.71, 9.72 [15:44:03] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 12.00, 11.17, 10.01 [15:44:21] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.12, 3.56, 3.46 [15:45:38] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - dnd.bellinrattin.it All nameservers failed to answer the query. [15:47:19] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.47, 11.04, 9.90 [15:49:19] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.72, 10.52, 9.85 [15:49:25] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.82, 1.49, 1.45 [15:49:46] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 9.23, 9.96, 9.82 [15:49:52] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.45, 3.66, 3.68 [15:50:05] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.87, 3.69, 3.64 [15:50:31] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.22, 11.51, 10.85 [15:51:18] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.04, 9.72, 9.65 [15:52:01] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.44, 4.16, 3.80 [15:53:25] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.32, 1.48, 1.46 [15:54:01] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.78, 3.86, 3.72 [15:54:31] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.09, 11.44, 10.99 [15:55:41] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.55, 2.61, 3.23 [15:56:34] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.08, 3.01, 3.79 [15:58:01] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:58:34] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.81, 3.97, 4.03 [15:59:37] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.37, 3.43, 3.41 [16:00:34] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.31, 3.64, 3.90 [16:01:33] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.74, 3.45, 3.42 [16:03:30] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.76, 3.07, 3.28 [16:03:30] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/8829d5c84c94...9ea52fee5e28 [16:03:32] [02miraheze/ImportDump] 07Universal-Omega 039ea52fe - Update extension.json [16:03:33] [02ImportDump] 07Universal-Omega synchronize pull request 03#1: Create extension - 13https://github.com/miraheze/ImportDump/pull/1 [16:03:35] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/9ea52fee5e28...e5b4dc185c78 [16:03:37] [02miraheze/ImportDump] 07Universal-Omega 03e5b4dc1 - Update en.json [16:03:38] [02ImportDump] 07Universal-Omega synchronize pull request 03#1: Create extension - 13https://github.com/miraheze/ImportDump/pull/1 [16:03:40] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/e5b4dc185c78...d0f40595f4bf [16:03:41] [02miraheze/ImportDump] 07Universal-Omega 03d0f4059 - Update qqq.json [16:03:43] [02ImportDump] 07Universal-Omega synchronize pull request 03#1: Create extension - 13https://github.com/miraheze/ImportDump/pull/1 [16:04:01] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.37, 2.94, 3.32 [16:04:31] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.22, 11.44, 10.98 [16:04:36] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/d0f40595f4bf...149bacc431fb [16:04:37] [02miraheze/ImportDump] 07Universal-Omega 03149bacc - Update ImportDumpRequestViewer.php [16:04:39] [02ImportDump] 07Universal-Omega synchronize pull request 03#1: Create extension - 13https://github.com/miraheze/ImportDump/pull/1 [16:05:01] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [16:06:17] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/149bacc431fb...43e5de4769a1 [16:06:18] [02miraheze/ImportDump] 07Universal-Omega 0343e5de4 - Update SpecialImportDumpRequestQueue.php [16:06:20] [02ImportDump] 07Universal-Omega synchronize pull request 03#1: Create extension - 13https://github.com/miraheze/ImportDump/pull/1 [16:06:31] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.23, 11.49, 11.07 [16:07:21] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/43e5de4769a1...2fecc0919c6f [16:07:23] [02miraheze/ImportDump] 07Universal-Omega 032fecc09 - Update SpecialRequestImportDump.php [16:07:24] [02ImportDump] 07Universal-Omega synchronize pull request 03#1: Create extension - 13https://github.com/miraheze/ImportDump/pull/1 [16:07:31] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.50, 10.15, 9.71 [16:08:46] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.38, 10.51, 9.90 [16:09:31] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.64, 9.44, 9.52 [16:09:32] [02ImportDump] 07Universal-Omega edited pull request 03#1: Create extension - 13https://github.com/miraheze/ImportDump/pull/1 [16:10:34] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.81, 4.00, 3.79 [16:10:40] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 13.41, 11.44, 10.31 [16:12:35] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.57, 3.77, 3.73 [16:12:36] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.69, 11.05, 10.32 [16:12:48] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/51884a1a20e3...daf5c016e970 [16:12:50] [02miraheze/mw-config] 07Universal-Omega 03daf5c01 - Update ImportDump rights [16:14:03] miraheze/mw-config - Universal-Omega the build passed. [16:14:34] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.08, 3.77, 3.73 [16:15:21] [02ImportDump] 07Universal-Omega synchronize pull request 03#1: Create extension - 13https://github.com/miraheze/ImportDump/pull/1 [16:15:22] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/2fecc0919c6f...62b3b09a893b [16:15:24] [02miraheze/ImportDump] 07Universal-Omega 0362b3b09 - Update extension.json [16:16:33] [02mw-config] 07ugochimobi synchronize pull request 03#4640: Wikibase configurations for gpcommonswiki - 13https://github.com/miraheze/mw-config/pull/4640 [16:17:09] [02mw-config] 07ugochimobi reviewed pull request 03#4640 commit - 13https://github.com/miraheze/mw-config/pull/4640#discussion_r861083104 [16:17:50] miraheze/mw-config - ugochimobi the build passed. [16:17:50] [02mw-config] 07ugochimobi commented on pull request 03#4640: Wikibase configurations for gpcommonswiki - 13https://github.com/miraheze/mw-config/pull/4640#issuecomment-1112402690 [16:19:12] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.55, 10.23, 9.59 [16:19:25] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.17, 1.82, 1.55 [16:19:56] !log [@test101] starting deploy of {'config': True} to all [16:19:57] !log [@test101] finished deploy of {'config': True} to all - SUCCESS in 1s [16:20:01] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [16:20:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:20:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:20:33] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.72, 3.73, 3.36 [16:20:35] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.52, 3.71, 3.70 [16:20:36] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 9.33, 10.06, 10.17 [16:21:25] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.95, 1.89, 1.61 [16:22:27] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.39, 3.54, 3.34 [16:22:35] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.06, 3.75, 3.71 [16:22:36] !log [@mwtask111] starting deploy of {'config': True} to all [16:22:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:22:49] !log [@mwtask111] finished deploy of {'config': True} to all - SUCCESS in 13s [16:22:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:23:11] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.26, 9.15, 9.34 [16:24:22] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.85, 3.02, 3.18 [16:24:28] miraheze/ImportDump - Universal-Omega the build passed. [16:24:35] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.87, 3.39, 3.59 [16:25:25] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.24, 1.58, 1.55 [16:27:31] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.24, 10.17, 9.73 [16:29:01] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [16:29:31] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.50, 9.27, 9.44 [16:29:48] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.70, 3.24, 3.08 [16:30:05] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.96, 3.48, 3.35 [16:30:34] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.44, 3.04, 3.37 [16:31:45] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.57, 2.99, 3.01 [16:32:01] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 3.10, 3.28, 3.29 [16:34:31] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 15.81, 12.21, 11.20 [16:39:03] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.78, 1.66, 1.48 [16:39:31] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.94, 10.31, 9.94 [16:40:57] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.53, 1.50, 1.43 [16:42:35] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.85, 4.32, 3.71 [16:43:31] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 9.21, 10.09, 9.95 [16:43:37] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 86, in check_records cname = str(dns_resolver.resolve(hostname, 'CNAME')[0]) File "/usr/li [16:43:37] 3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query dnd.bellinrattin.it. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [16:44:31] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.35, 11.66, 11.56 [16:46:31] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.61, 11.90, 11.65 [16:46:36] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.41, 10.47, 9.99 [16:47:25] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.33, 1.78, 1.61 [16:47:31] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.96, 10.98, 10.32 [16:48:36] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 9.72, 10.12, 9.92 [16:49:31] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.18, 11.14, 10.44 [16:51:02] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.24, 10.44, 9.68 [16:51:14] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.56, 3.37, 3.06 [16:51:31] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.98, 10.15, 10.16 [16:52:01] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.74, 3.88, 3.31 [16:53:11] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.36, 3.23, 3.04 [16:54:11] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:54:25] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:54:26] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:55:25] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.86, 1.97, 1.82 [16:56:29] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14626 bytes in 8.472 second response time [16:56:29] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14626 bytes in 8.555 second response time [16:57:00] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 13.37, 11.51, 10.35 [16:58:01] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.94, 3.64, 3.39 [16:58:18] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 22334 bytes in 6.660 second response time [16:58:34] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.46, 3.91, 3.98 [16:59:00] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.60, 10.61, 10.16 [17:00:01] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.81, 3.30, 3.29 [17:00:59] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.52, 9.95, 9.98 [17:02:31] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.57, 10.43, 11.68 [17:02:35] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.76, 4.69, 4.21 [17:03:31] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.46, 10.25, 9.79 [17:04:31] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.38, 11.78, 12.03 [17:06:34] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.70, 3.57, 3.85 [17:07:31] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 10.01, 10.06, 9.81 [17:08:31] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.94, 11.69, 11.98 [17:08:34] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.48, 4.33, 4.09 [17:09:47] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.26, 4.30, 3.53 [17:09:49] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/62b3b09a893b...455b6fa0ae14 [17:09:50] [02miraheze/ImportDump] 07Universal-Omega 03455b6fa - Improve schema [17:09:52] [02ImportDump] 07Universal-Omega synchronize pull request 03#1: Create extension - 13https://github.com/miraheze/ImportDump/pull/1 [17:10:01] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 6.64, 4.20, 3.55 [17:10:31] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.78, 12.13, 12.11 [17:11:43] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.76, 3.72, 3.41 [17:12:01] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.20, 3.78, 3.48 [17:12:36] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.73, 10.33, 9.66 [17:13:37] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp20.miraheze.org - CNAME OK [17:13:40] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.70, 3.06, 3.20 [17:14:01] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.25, 3.90, 3.56 [17:14:23] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.66, 10.87, 10.23 [17:14:31] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.50, 11.79, 12.00 [17:14:36] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 10.03, 10.10, 9.65 [17:14:55] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.76, 10.45, 9.80 [17:15:25] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.54, 1.57, 1.69 [17:16:01] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.61, 3.45, 3.44 [17:16:18] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.60, 10.14, 10.04 [17:16:34] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.91, 3.73, 3.93 [17:16:54] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 9.35, 9.99, 9.71 [17:19:25] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.05, 1.82, 1.76 [17:20:01] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.89, 3.33, 3.40 [17:20:25] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.89, 1.56, 1.23 [17:20:57] miraheze/ImportDump - Universal-Omega the build passed. [17:21:25] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.56, 1.80, 1.77 [17:22:25] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.68, 1.62, 1.29 [17:22:35] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.15, 3.84, 3.87 [17:23:25] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.33, 1.63, 1.70 [17:24:01] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.15, 3.41, 3.42 [17:24:31] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.74, 11.02, 11.26 [17:24:34] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.82, 3.99, 3.93 [17:26:01] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.34, 3.07, 3.30 [17:26:31] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.11, 11.16, 11.29 [17:26:35] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.45, 4.21, 4.00 [17:28:34] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.05, 4.00, 3.97 [17:30:01] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.89, 3.44, 3.44 [17:33:37] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.68, 11.05, 10.39 [17:34:01] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 3.10, 3.28, 3.39 [17:34:31] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 14.17, 11.64, 11.27 [17:35:34] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.43, 10.14, 10.14 [17:36:22] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/455b6fa0ae14...4565afe97fb9 [17:36:24] [02miraheze/ImportDump] 07Universal-Omega 034565afe - Update ImportDumpRequestQueuePager.php [17:36:25] [02ImportDump] 07Universal-Omega synchronize pull request 03#1: Create extension - 13https://github.com/miraheze/ImportDump/pull/1 [17:36:31] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.77, 11.30, 11.21 [17:38:01] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.18, 3.48, 3.44 [17:39:49] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/4565afe97fb9...f28a2cf06beb [17:39:50] [02miraheze/ImportDump] 07Universal-Omega 03f28a2cf - Update ImportDumpRequestQueuePager.php [17:39:52] [02ImportDump] 07Universal-Omega synchronize pull request 03#1: Create extension - 13https://github.com/miraheze/ImportDump/pull/1 [17:40:01] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 3.20, 3.38, 3.40 [17:40:12] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/f28a2cf06beb...cb1a16f05894 [17:40:14] [02miraheze/ImportDump] 07Universal-Omega 03cb1a16f - Update ImportDumpRequestQueuePager.php [17:40:15] [02ImportDump] 07Universal-Omega synchronize pull request 03#1: Create extension - 13https://github.com/miraheze/ImportDump/pull/1 [17:42:36] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.28, 10.57, 9.63 [17:42:38] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.33, 10.28, 9.76 [17:44:34] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.66, 3.77, 3.65 [17:44:36] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.71, 9.36, 9.29 [17:44:37] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.82, 9.60, 9.59 [17:45:31] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.93, 11.13, 10.38 [17:46:35] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.47, 3.41, 3.54 [17:47:31] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.74, 11.21, 10.50 [17:49:25] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.76, 1.50, 1.39 [17:50:31] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.56, 11.36, 11.02 [17:51:31] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.84, 9.85, 10.09 [17:52:08] miraheze/ImportDump - Universal-Omega the build passed. [17:54:01] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [17:54:31] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.48, 11.58, 11.18 [17:55:25] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.32, 1.53, 1.45 [17:56:31] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.70, 11.97, 11.38 [17:56:35] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.77, 3.05, 3.36 [18:01:40] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.26, 10.44, 9.95 [18:02:18] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.28, 11.42, 10.58 [18:03:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:04:13] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.50, 10.71, 10.42 [18:07:38] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.38, 9.73, 9.85 [18:08:09] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.57, 2.00, 1.66 [18:08:31] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.05, 11.81, 11.84 [18:09:33] [02mw-config] 07ugochimobi opened pull request 03#4651: set disabledDataTypes, entityDataFormats, taintedReferencesEnabled for gratispaideiawiki, gratisdatawiki, gpcommonswiki - 13https://github.com/miraheze/mw-config/pull/4651 [18:09:59] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 9.34, 9.82, 10.11 [18:10:34] miraheze/mw-config - ugochimobi the build passed. [18:11:36] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.26, 10.36, 10.11 [18:12:31] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.38, 11.94, 11.87 [18:12:34] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.78, 3.68, 3.38 [18:13:03] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.75, 3.79, 3.29 [18:13:36] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.26, 9.88, 9.98 [18:13:52] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.71, 10.25, 9.80 [18:14:12] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/cb1a16f05894...d1d1e65f823a [18:14:13] [02miraheze/ImportDump] 07Universal-Omega 03d1d1e65 - Update importdump_requests.sql [18:14:15] [02ImportDump] 07Universal-Omega synchronize pull request 03#1: Create extension - 13https://github.com/miraheze/ImportDump/pull/1 [18:14:31] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.45, 11.67, 11.78 [18:14:34] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.11, 4.42, 3.70 [18:14:57] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.19, 3.48, 3.23 [18:15:46] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 9.80, 10.11, 9.80 [18:15:57] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.06, 3.51, 3.09 [18:16:57] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/d1d1e65f823a...7d6a8afa9eeb [18:16:59] [02miraheze/ImportDump] 07Universal-Omega 037d6a8af - Update importdump_requests.sql [18:17:00] [02ImportDump] 07Universal-Omega synchronize pull request 03#1: Create extension - 13https://github.com/miraheze/ImportDump/pull/1 [18:17:54] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.71, 3.34, 3.08 [18:18:46] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.25, 3.75, 3.38 [18:19:34] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.01, 10.68, 10.26 [18:20:21] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/7d6a8afa9eeb...b45fde1b5aef [18:20:23] [02miraheze/ImportDump] 07Universal-Omega 03b45fde1 - Update importdump_request_comments.sql [18:20:24] [02ImportDump] 07Universal-Omega synchronize pull request 03#1: Create extension - 13https://github.com/miraheze/ImportDump/pull/1 [18:20:40] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.01, 3.47, 3.32 [18:21:34] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.74, 10.82, 10.35 [18:22:31] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.33, 11.71, 11.75 [18:22:35] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.89, 3.67, 3.65 [18:23:33] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.98, 11.72, 10.75 [18:24:24] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.23, 10.59, 9.99 [18:24:29] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.62, 3.00, 3.16 [18:26:19] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.92, 9.65, 9.72 [18:26:34] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.21, 3.78, 3.65 [18:27:32] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.56, 10.91, 10.67 [18:29:04] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.39, 10.40, 10.02 [18:30:11] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.36, 10.84, 10.17 [18:30:14] miraheze/ImportDump - Universal-Omega the build passed. [18:30:58] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.60, 9.58, 9.75 [18:32:34] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.74, 3.82, 3.73 [18:34:01] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.84, 10.69, 10.33 [18:34:35] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.73, 4.54, 4.03 [18:35:01] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.72, 4.29, 3.65 [18:35:27] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.67, 3.57, 3.27 [18:35:56] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.48, 9.94, 10.09 [18:37:24] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.40, 4.07, 3.50 [18:37:29] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.65, 8.75, 9.85 [18:39:20] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.57, 3.90, 3.51 [18:44:33] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.89, 3.68, 3.71 [18:46:27] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.82, 3.83, 3.75 [18:46:31] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.51, 11.51, 11.92 [18:49:02] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.42, 3.71, 3.53 [18:50:16] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.75, 3.84, 3.78 [18:50:25] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 13.26, 11.58, 10.63 [18:52:24] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.88, 10.37, 10.29 [18:54:05] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.64, 4.19, 3.91 [18:54:24] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.10, 9.55, 10.00 [18:54:52] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.61, 3.62, 3.59 [18:56:01] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.29, 3.48, 3.68 [18:56:31] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.18, 11.27, 11.49 [18:56:51] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.27, 3.10, 3.40 [18:58:01] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.22, 3.83, 3.79 [18:58:22] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.84, 10.43, 10.30 [19:00:22] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.82, 9.91, 10.13 [19:00:31] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.08, 11.77, 11.69 [19:00:51] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.29, 3.36, 3.45 [19:02:01] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.37, 3.73, 3.77 [19:04:01] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.98, 3.82, 3.78 [19:06:01] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.84, 3.38, 3.62 [19:06:31] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.31, 11.75, 11.62 [19:06:51] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.31, 2.92, 3.25 [19:08:31] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.63, 11.13, 11.43 [19:12:00] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.39, 3.45, 3.53 [19:13:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [19:14:00] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.57, 3.35, 3.48 [19:14:51] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.07, 4.54, 3.86 [19:16:01] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.23, 3.52, 3.52 [19:18:01] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.18, 3.62, 3.57 [19:18:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [19:18:51] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.59, 3.76, 3.72 [19:20:31] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.50, 10.64, 10.61 [19:22:00] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.23, 3.00, 3.34 [19:24:31] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.30, 11.83, 11.17 [19:24:51] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.50, 3.86, 3.67 [19:26:01] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.66, 3.59, 3.50 [19:28:01] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.14, 3.93, 3.64 [19:28:31] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.47, 12.15, 11.43 [19:28:51] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.41, 3.48, 3.58 [19:30:01] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.23, 3.27, 3.43 [19:30:31] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.52, 11.52, 11.29 [19:31:40] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 13.32, 11.48, 10.21 [19:32:01] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.16, 3.77, 3.60 [19:32:51] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.41, 2.87, 3.30 [19:33:35] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.64, 11.58, 10.40 [19:33:58] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.70, 9.93, 9.55 [19:34:01] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.77, 3.45, 3.51 [19:34:31] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.72, 11.92, 11.50 [19:35:57] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.35, 9.68, 9.50 [19:36:31] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.91, 11.35, 11.36 [19:36:34] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.29, 3.33, 4.00 [19:37:31] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.66, 9.77, 9.91 [19:38:01] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.61, 4.21, 3.77 [19:38:34] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.66, 3.77, 4.06 [19:40:01] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.44, 3.96, 3.74 [19:40:34] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.42, 3.39, 3.90 [19:43:29] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.31, 10.33, 9.14 [19:44:35] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.06, 4.25, 4.16 [19:44:51] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.78, 3.46, 3.38 [19:45:23] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.38, 9.63, 9.02 [19:45:37] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - dnd.bellinrattin.it All nameservers failed to answer the query. [19:46:01] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.69, 3.82, 3.69 [19:46:51] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.19, 3.09, 3.25 [19:48:01] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.73, 3.28, 3.50 [19:48:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [19:49:52] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.57, 9.67, 9.18 [19:50:01] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.96, 3.09, 3.39 [19:51:51] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.72, 9.64, 9.23 [19:52:34] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.94, 3.37, 3.84 [19:57:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [19:58:31] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.56, 11.64, 11.19 [19:58:34] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.07, 3.20, 3.55 [19:59:04] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.15, 9.78, 9.22 [20:00:31] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.15, 10.92, 10.98 [20:00:37] PROBLEM - test101 MediaWiki Rendering on test101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:00:58] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.73, 10.62, 10.02 [20:01:03] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.24, 9.31, 9.13 [20:02:40] RECOVERY - test101 MediaWiki Rendering on test101 is OK: HTTP OK: HTTP/1.1 200 OK - 51878 bytes in 3.750 second response time [20:02:48] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.07, 1.59, 1.35 [20:02:53] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.49, 10.57, 10.07 [20:04:43] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.04, 1.37, 1.29 [20:04:46] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.16, 11.03, 9.98 [20:04:48] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.43, 9.32, 9.67 [20:06:45] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.01, 11.11, 10.15 [20:06:51] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.70, 3.61, 3.36 [20:07:07] PROBLEM - test101 MediaWiki Rendering on test101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:07:36] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.95, 3.75, 3.46 [20:08:45] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.62, 11.58, 10.43 [20:09:01] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/d39953b8cee9...8fc6b6c01078 [20:09:03] [02miraheze/mediawiki] 07Universal-Omega 038fc6b6c - Fix weird conflict in MaintenanceTest whereas testOptionGetters was duplicated [20:09:08] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.04, 10.57, 9.63 [20:09:30] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.42, 3.38, 3.37 [20:10:02] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_38 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/7e0b5f9a1a2a...ed024835451a [20:10:03] [02miraheze/mediawiki] 07Universal-Omega 03ed02483 - Fix weird conflict in MaintenanceTest whereas testOptionGetters was duplicated [20:10:35] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.32, 10.42, 9.98 [20:10:42] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/8bc14c77fba6...4f941848a9f0 [20:10:44] [02miraheze/mediawiki] 07Universal-Omega 034f94184 - Fix weird conflict in MaintenanceTest whereas testOptionGetters was duplicated [20:10:47] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.17, 10.57, 10.19 [20:10:51] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.01, 3.79, 3.50 [20:11:02] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.31, 10.78, 9.81 [20:12:30] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 9.23, 10.08, 9.92 [20:12:41] [02mediawiki] 07Universal-Omega closed pull request 03#6484: Bump skins/Monaco from `e1dec4e` to `7637d88` - 13https://github.com/miraheze/mediawiki/pull/6484 [20:12:43] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/skins/Monaco-7637d88 [20:12:44] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/8fc6b6c01078...28a19a1a3a08 [20:12:46] [02miraheze/mediawiki] 07dependabot[bot] 0328a19a1 - Bump skins/Monaco from `e1dec4e` to `7637d88` (#6484) [20:12:46] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.66, 10.11, 10.08 [20:12:47] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/skins/Monaco-7637d88 - 13https://github.com/miraheze/mediawiki [20:12:56] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.67, 10.05, 9.67 [20:12:57] [02mediawiki] 07Universal-Omega closed pull request 03#6477: Bump extensions/AutoCreatePage from `da7ffc2` to `edd712e` - 13https://github.com/miraheze/mediawiki/pull/6477 [20:12:58] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/28a19a1a3a08...2e40fe437497 [20:13:00] [02miraheze/mediawiki] 07dependabot[bot] 032e40fe4 - Bump extensions/AutoCreatePage from `da7ffc2` to `edd712e` (#6477) [20:13:01] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/AutoCreatePage-edd712e - 13https://github.com/miraheze/mediawiki [20:13:03] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/AutoCreatePage-edd712e [20:13:08] [02mediawiki] 07Universal-Omega closed pull request 03#6460: Bump extensions/MsUpload from `13083d7` to `246d563` - 13https://github.com/miraheze/mediawiki/pull/6460 [20:13:09] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/2e40fe437497...17b9498e38f1 [20:13:11] [02miraheze/mediawiki] 07dependabot[bot] 0317b9498 - Bump extensions/MsUpload from `13083d7` to `246d563` (#6460) [20:13:12] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/MsUpload-246d563 - 13https://github.com/miraheze/mediawiki [20:13:14] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/MsUpload-246d563 [20:13:17] [02mediawiki] 07Universal-Omega closed pull request 03#6457: Bump extensions/Quiz from `34d3dbc` to `8750d79` - 13https://github.com/miraheze/mediawiki/pull/6457 [20:13:19] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/17b9498e38f1...6c81e547c250 [20:13:19] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.96, 3.93, 3.57 [20:13:20] [02miraheze/mediawiki] 07dependabot[bot] 036c81e54 - Bump extensions/Quiz from `34d3dbc` to `8750d79` (#6457) [20:13:22] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/Quiz-8750d79 [20:13:23] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/Quiz-8750d79 - 13https://github.com/miraheze/mediawiki [20:13:27] [02mediawiki] 07Universal-Omega closed pull request 03#6455: Bump extensions/DisplayTitle from `f3a4bf4` to `8d37fa4` - 13https://github.com/miraheze/mediawiki/pull/6455 [20:13:29] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/6c81e547c250...5974c3ac7fb0 [20:13:30] [02miraheze/mediawiki] 07dependabot[bot] 035974c3a - Bump extensions/DisplayTitle from `f3a4bf4` to `8d37fa4` (#6455) [20:13:32] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/DisplayTitle-8d37fa4 [20:13:33] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/DisplayTitle-8d37fa4 - 13https://github.com/miraheze/mediawiki [20:13:45] [02mediawiki] 07Universal-Omega closed pull request 03#6453: Bump extensions/GoogleDocs4MW from `562f30b` to `40ac303` - 13https://github.com/miraheze/mediawiki/pull/6453 [20:13:46] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/GoogleDocs4MW-40ac303 - 13https://github.com/miraheze/mediawiki [20:13:48] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/5974c3ac7fb0...c0d1fb55d475 [20:13:49] [02miraheze/mediawiki] 07dependabot[bot] 03c0d1fb5 - Bump extensions/GoogleDocs4MW from `562f30b` to `40ac303` (#6453) [20:13:51] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/GoogleDocs4MW-40ac303 [20:13:56] [02mediawiki] 07Universal-Omega closed pull request 03#6449: Bump extensions/Arrays from `3a000f8` to `1ec7c30` - 13https://github.com/miraheze/mediawiki/pull/6449 [20:13:57] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/Arrays-1ec7c30 - 13https://github.com/miraheze/mediawiki [20:13:59] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/Arrays-1ec7c30 [20:14:00] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/c0d1fb55d475...68cb1fa861a7 [20:14:02] [02miraheze/mediawiki] 07dependabot[bot] 0368cb1fa - Bump extensions/Arrays from `3a000f8` to `1ec7c30` (#6449) [20:14:05] [02mediawiki] 07Universal-Omega closed pull request 03#6448: Bump extensions/RandomSelection from `343fc67` to `76288ca` - 13https://github.com/miraheze/mediawiki/pull/6448 [20:14:06] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/68cb1fa861a7...a8a06a23a653 [20:14:08] [02miraheze/mediawiki] 07dependabot[bot] 03a8a06a2 - Bump extensions/RandomSelection from `343fc67` to `76288ca` (#6448) [20:14:09] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/RandomSelection-76288ca [20:14:11] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/RandomSelection-76288ca - 13https://github.com/miraheze/mediawiki [20:14:12] [02mediawiki] 07Universal-Omega closed pull request 03#6446: Bump extensions/CreateRedirect from `eee1ac0` to `6567fee` - 13https://github.com/miraheze/mediawiki/pull/6446 [20:14:14] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp20.miraheze.org - CNAME OK [20:14:14] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/CreateRedirect-6567fee - 13https://github.com/miraheze/mediawiki [20:14:15] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/CreateRedirect-6567fee [20:14:17] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/a8a06a23a653...f8b79612194d [20:14:18] [02miraheze/mediawiki] 07dependabot[bot] 03f8b7961 - Bump extensions/CreateRedirect from `eee1ac0` to `6567fee` (#6446) [20:14:44] [02mediawiki] 07Universal-Omega closed pull request 03#6445: Bump extensions/SubPageList3 from `fae904e` to `5208a7b` - 13https://github.com/miraheze/mediawiki/pull/6445 [20:14:45] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/SubPageList3-5208a7b - 13https://github.com/miraheze/mediawiki [20:14:47] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/SubPageList3-5208a7b [20:14:49] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/f8b79612194d...06e4126d2575 [20:14:50] [02miraheze/mediawiki] 07dependabot[bot] 0306e4126 - Bump extensions/SubPageList3 from `fae904e` to `5208a7b` (#6445) [20:14:51] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.64, 3.85, 3.59 [20:15:14] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.56, 3.59, 3.50 [20:15:38] RECOVERY - test101 MediaWiki Rendering on test101 is OK: HTTP OK: HTTP/1.1 200 OK - 51878 bytes in 5.288 second response time [20:16:51] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.18, 3.98, 3.66 [20:17:08] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.01, 3.93, 3.64 [20:18:51] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.10, 3.54, 3.53 [20:19:03] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.64, 3.38, 3.47 [20:20:57] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.52, 3.16, 3.38 [20:21:00] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.71, 10.27, 9.38 [20:22:34] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.12, 3.68, 4.00 [20:23:00] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.29, 9.60, 9.25 [20:23:05] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.95, 10.35, 10.03 [20:24:51] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.91, 3.06, 3.33 [20:25:00] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.21, 8.91, 9.54 [20:26:34] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.57, 3.82, 3.95 [20:28:35] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.14, 3.54, 3.46 [20:28:51] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.48, 4.00, 3.70 [20:30:29] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.01, 3.61, 3.48 [20:30:51] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.29, 4.28, 3.82 [20:32:08] PROBLEM - test101 MediaWiki Rendering on test101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:32:24] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.66, 3.67, 3.52 [20:32:31] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.02, 11.28, 10.87 [20:32:42] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.31, 10.31, 9.84 [20:34:18] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.34, 3.21, 3.37 [20:34:31] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.32, 11.14, 10.87 [20:34:37] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 9.52, 9.93, 9.75 [20:34:51] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.59, 3.64, 3.69 [20:35:38] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.99, 10.58, 9.88 [20:36:31] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.66, 12.12, 11.26 [20:38:07] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.57, 3.86, 3.59 [20:38:31] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.54, 11.68, 11.20 [20:38:51] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.60, 4.17, 3.88 [20:40:31] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.06, 11.78, 11.29 [20:42:31] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.66, 11.37, 11.23 [20:42:34] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.04, 1.71, 1.47 [20:42:45] RECOVERY - test101 MediaWiki Rendering on test101 is OK: HTTP OK: HTTP/1.1 200 OK - 51878 bytes in 1.580 second response time [20:44:51] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.66, 3.69, 3.81 [20:45:13] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.42, 10.01, 9.75 [20:45:34] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.44, 9.77, 9.99 [20:46:25] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.87, 1.46, 1.44 [20:47:12] PROBLEM - test101 MediaWiki Rendering on test101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:47:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [20:48:01] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.84, 3.73, 3.62 [20:50:31] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.52, 11.04, 10.99 [20:52:25] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.05, 1.63, 1.49 [20:52:51] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.54, 3.62, 3.69 [20:52:53] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.35, 9.79, 9.86 [20:54:01] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 6.69, 4.39, 3.81 [20:54:25] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.69, 1.73, 1.55 [20:54:31] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.39, 11.22, 11.12 [20:54:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [20:54:51] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.77, 3.82, 3.77 [20:55:45] RECOVERY - test101 MediaWiki Rendering on test101 is OK: HTTP OK: HTTP/1.1 200 OK - 51878 bytes in 6.935 second response time [20:56:01] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.52, 3.93, 3.71 [20:56:25] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.47, 1.60, 1.52 [20:56:31] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.09, 11.47, 11.21 [20:58:01] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.11, 4.03, 3.78 [20:58:31] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.37, 11.19, 11.15 [20:58:51] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.55, 3.82, 3.74 [21:00:51] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.60, 3.68, 3.73 [21:02:31] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.45, 11.13, 11.07 [21:04:00] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.08, 3.53, 3.77 [21:04:31] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.27, 11.26, 11.14 [21:04:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:05:57] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.76, 1.55, 1.12 [21:06:34] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.02, 3.25, 3.96 [21:07:57] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.47, 1.49, 1.14 [21:08:34] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.03, 3.47, 3.94 [21:08:51] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.80, 3.74, 3.64 [21:09:05] PROBLEM - mw111 Puppet on mw111 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/apt/trusted.gpg.d/puppetlabs.gpg] [21:10:01] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 6.61, 4.73, 4.10 [21:12:25] PROBLEM - test101 MediaWiki Rendering on test101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:14:01] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:14:51] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.93, 3.86, 3.81 [21:16:31] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.73, 11.33, 10.88 [21:18:31] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.31, 10.48, 10.63 [21:20:01] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.30, 3.79, 3.95 [21:20:58] RECOVERY - test101 MediaWiki Rendering on test101 is OK: HTTP OK: HTTP/1.1 200 OK - 53472 bytes in 7.097 second response time [21:24:51] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.09, 3.47, 3.57 [21:27:25] PROBLEM - test101 MediaWiki Rendering on test101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:28:01] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 6.13, 4.38, 4.06 [21:30:51] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.18, 3.90, 3.81 [21:33:08] [02mediawiki] 07Universal-Omega commented on pull request 03#6287: Bump skins/Cosmos from `8fe49c1` to `4d59e00` - 13https://github.com/miraheze/mediawiki/pull/6287#issuecomment-1112673820 [21:33:10] [02mediawiki] 07dependabot[bot] edited pull request 03#6287: Bump skins/Cosmos from `8fe49c1` to `4d59e00` - 13https://github.com/miraheze/mediawiki/pull/6287 [21:34:01] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:34:14] [02mediawiki] 07dependabot[bot] edited pull request 03#6287: Bump skins/Cosmos from `8fe49c1` to `4d59e00` - 13https://github.com/miraheze/mediawiki/pull/6287 [21:34:22] hello [21:34:22] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/skins/Cosmos-1a69692 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/a3b070b1624c [21:34:23] [02miraheze/mediawiki] 07dependabot[bot] 03a3b070b - Bump skins/Cosmos from `8fe49c1` to `1a69692` [21:34:25] [02mediawiki] 07dependabot[bot] opened pull request 03#6589: Bump skins/Cosmos from `8fe49c1` to `1a69692` - 13https://github.com/miraheze/mediawiki/pull/6589 [21:34:26] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/skins/Cosmos-1a69692 - 13https://github.com/miraheze/mediawiki [21:34:47] [02mediawiki] 07github-actions[bot] labeled pull request 03#6589: Bump skins/Cosmos from `8fe49c1` to `1a69692` - 13https://github.com/miraheze/mediawiki/pull/6589 [21:34:49] [02mediawiki] 07github-actions[bot] labeled pull request 03#6589: Bump skins/Cosmos from `8fe49c1` to `1a69692` - 13https://github.com/miraheze/mediawiki/pull/6589 [21:34:49] who is here?. [21:34:59] [02mediawiki] 07Universal-Omega closed pull request 03#6589: Bump skins/Cosmos from `8fe49c1` to `1a69692` - 13https://github.com/miraheze/mediawiki/pull/6589 [21:35:01] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/06e4126d2575...1075d4b2fc4f [21:35:02] [02miraheze/mediawiki] 07dependabot[bot] 031075d4b - Bump skins/Cosmos from `8fe49c1` to `1a69692` (#6589) [21:35:04] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/skins/Cosmos-1a69692 - 13https://github.com/miraheze/mediawiki [21:35:05] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/skins/Cosmos-1a69692 [21:35:15] [02mediawiki] 07Universal-Omega closed pull request 03#6287: Bump skins/Cosmos from `8fe49c1` to `4d59e00` - 13https://github.com/miraheze/mediawiki/pull/6287 [21:35:17] [02mediawiki] 07dependabot[bot] commented on pull request 03#6287: Bump skins/Cosmos from `8fe49c1` to `4d59e00` - 13https://github.com/miraheze/mediawiki/pull/6287#issuecomment-1112675291 [21:35:18] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/skins/Cosmos-4d59e00 [21:35:20] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/skins/Cosmos-4d59e00 - 13https://github.com/miraheze/mediawiki [21:35:22] RECOVERY - mw111 Puppet on mw111 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [21:35:36] :( [21:35:49] Hi Hispano76 [21:35:53] RECOVERY - test101 MediaWiki Rendering on test101 is OK: HTTP OK: HTTP/1.1 200 OK - 51269 bytes in 2.829 second response time [21:40:31] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.48, 9.88, 10.18 [21:44:31] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.05, 10.15, 10.29 [21:45:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:46:01] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.43, 3.58, 3.97 [21:48:01] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.53, 4.05, 4.08 [21:50:01] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.44, 3.80, 3.99 [21:50:24] PROBLEM - test101 MediaWiki Rendering on test101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:50:31] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.94, 11.94, 10.96 [21:52:31] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.09, 11.06, 10.78 [21:54:01] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.29, 3.86, 3.96 [21:56:01] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.99, 3.43, 3.78 [21:56:51] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.40, 3.48, 3.53 [21:58:36] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.44, 10.56, 9.57 [21:59:07] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 3.60, 2.62, 1.68 [22:00:36] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.17, 9.57, 9.32 [22:00:51] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.55, 3.66, 3.59 [22:03:02] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.62, 10.53, 9.52 [22:03:10] RECOVERY - test101 MediaWiki Rendering on test101 is OK: HTTP OK: HTTP/1.1 200 OK - 51269 bytes in 2.356 second response time [22:04:51] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.04, 2.87, 3.28 [22:04:57] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.75, 9.90, 9.42 [22:06:47] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.22, 1.71, 1.62 [22:08:43] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.70, 1.41, 1.52 [22:08:48] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.92, 10.49, 9.78 [22:09:07] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.86, 10.26, 9.56 [22:10:31] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.81, 9.57, 10.13 [22:10:43] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 9.17, 10.18, 9.77 [22:12:01] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.39, 2.84, 3.34 [22:13:06] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.15, 9.45, 9.48 [22:16:58] PROBLEM - test101 MediaWiki Rendering on test101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:18:01] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.07, 3.79, 3.54 [22:18:51] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.03, 3.89, 3.43 [22:20:01] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.60, 3.94, 3.64 [22:20:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [22:22:01] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.34, 4.35, 3.82 [22:23:17] RECOVERY - test101 MediaWiki Rendering on test101 is OK: HTTP OK: HTTP/1.1 200 OK - 51269 bytes in 2.330 second response time [22:24:51] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.69, 3.72, 3.55 [22:26:25] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.79, 2.02, 1.72 [22:26:51] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.93, 3.14, 3.36 [22:27:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [22:28:43] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.28, 9.23, 8.76 [22:30:01] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.00, 3.96, 3.88 [22:30:20] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.32, 2.03, 1.22 [22:30:34] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.28, 3.22, 3.84 [22:30:37] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.79, 8.33, 8.49 [22:32:01] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.05, 3.90, 3.86 [22:32:20] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.64, 1.51, 1.13 [22:32:51] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.64, 10.12, 9.58 [22:34:01] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.30, 3.66, 3.78 [22:36:01] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.81, 3.75, 3.77 [22:36:25] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.50, 1.86, 1.98 [22:36:41] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 9.33, 9.82, 9.59 [22:37:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [22:38:01] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.34, 3.13, 3.54 [22:39:52] PROBLEM - test101 MediaWiki Rendering on test101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:40:00] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.41, 2.78, 3.35 [22:42:25] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.16, 1.27, 1.66 [22:42:35] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.68, 2.85, 3.35 [22:44:01] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [22:50:28] RECOVERY - test101 MediaWiki Rendering on test101 is OK: HTTP OK: HTTP/1.1 200 OK - 51269 bytes in 1.420 second response time [22:50:35] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.73, 3.47, 3.45 [22:52:01] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.65, 3.68, 3.30 [22:52:31] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.76, 10.67, 9.77 [22:52:34] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.26, 4.04, 3.66 [22:54:25] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.05, 1.68, 1.57 [22:54:51] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.44, 4.22, 3.44 [22:56:25] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.57, 1.64, 1.56 [23:00:02] !log [@mwtask111] starting deploy of {'l10nupdate': True} to all [23:00:07] !log [@test101] starting deploy of {'l10nupdate': True} to all [23:00:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:00:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:01:46] [02miraheze/ssl] 07MacFan4000 pushed 031 commit to 03MacFan4000-patch-1 [+0/-1/±0] 13https://github.com/miraheze/ssl/commit/20bbc115a4ab [23:01:47] [02miraheze/ssl] 07MacFan4000 0320bbc11 - remove munwiki.info [23:01:49] [02ssl] 07MacFan4000 created branch 03MacFan4000-patch-1 - 13https://github.com/miraheze/ssl [23:01:52] [02ssl] 07MacFan4000 opened pull request 03#509: remove munwiki.info - 13https://github.com/miraheze/ssl/pull/509 [23:02:50] [02miraheze/ssl] 07MacFan4000 pushed 031 commit to 03MacFan4000-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/20bbc115a4ab...df859d18afd1 [23:02:51] [02miraheze/ssl] 07MacFan4000 03df859d1 - Update certs.yaml [23:02:54] [02ssl] 07MacFan4000 synchronize pull request 03#509: remove munwiki.info - 13https://github.com/miraheze/ssl/pull/509 [23:03:04] PROBLEM - test101 MediaWiki Rendering on test101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:04:21] [02ssl] 07MacFan4000 closed pull request 03#509: remove munwiki.info - 13https://github.com/miraheze/ssl/pull/509 [23:04:23] [02miraheze/ssl] 07MacFan4000 pushed 031 commit to 03master [+0/-1/±1] 13https://github.com/miraheze/ssl/compare/8c4ff82a34c8...605fb70553f1 [23:04:24] [02miraheze/ssl] 07MacFan4000 03605fb70 - remove munwiki.info (#509) [23:04:26] [02ssl] 07MacFan4000 deleted branch 03MacFan4000-patch-1 - 13https://github.com/miraheze/ssl [23:04:27] [02miraheze/ssl] 07MacFan4000 deleted branch 03MacFan4000-patch-1 [23:04:34] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.35, 10.71, 10.10 [23:04:53] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/03f22b46ebd0 [23:04:54] [02miraheze/mw-config] 07Universal-Omega 0303f22b4 - Only load the used instance of wgObjectCaches [23:04:56] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-2 - 13https://github.com/miraheze/mw-config [23:04:57] [02mw-config] 07Universal-Omega opened pull request 03#4652: Only load the used instance of wgObjectCaches - 13https://github.com/miraheze/mw-config/pull/4652 [23:06:03] miraheze/mw-config - Universal-Omega the build passed. [23:06:29] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.92, 10.06, 9.93 [23:08:32] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.85, 9.82, 10.09 [23:15:00] PROBLEM - cp21 Stunnel HTTP for test101 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:15:08] PROBLEM - cp30 Stunnel HTTP for test101 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:15:33] PROBLEM - cp20 Stunnel HTTP for test101 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:15:53] PROBLEM - cp31 Stunnel HTTP for test101 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:15:59] PROBLEM - test101 PowerDNS Recursor on test101 is CRITICAL: CRITICAL - Plugin timed out while executing system call [23:18:17] PROBLEM - test101 SSH on test101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:21:36] PROBLEM - test101 Puppet on test101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [23:21:44] PROBLEM - test101 NTP time on test101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [23:21:46] PROBLEM - test101 conntrack_table_size on test101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [23:22:44] PROBLEM - test101 HTTPS on test101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:23:32] paladox: ^^ I cannot ssh into test101 nor can i access betawiki so seems like a reboot is needed [23:23:43] I am doing something. [23:23:45] PROBLEM - test101 Check Gluster Clients on test101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [23:23:46] PROBLEM - test101 Redis Process on test101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [23:23:47] PROBLEM - test101 JobChron Service on test101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [23:23:48] PROBLEM - test101 JobRunner Service on test101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [23:23:57] I caused it on accident. I do have SSH access in though. [23:24:12] PROBLEM - test101 ferm_active on test101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [23:24:28] !log [@mwtask111] finished deploy of {'l10nupdate': True} to all - SUCCESS in 1465s [23:24:31] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.61, 10.64, 10.07 [23:24:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:24:41] I made a mistake doing something while l10nupdate was also running... [23:24:48] PROBLEM - test101 Disk Space on test101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [23:25:28] PROBLEM - test101 php-fpm on test101 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [23:30:40] RECOVERY - test101 HTTPS on test101 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 549 bytes in 6.171 second response time [23:30:45] RECOVERY - test101 conntrack_table_size on test101 is OK: OK: nf_conntrack is 0 % full [23:30:46] RECOVERY - cp21 Stunnel HTTP for test101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14642 bytes in 8.456 second response time [23:30:50] MacFan4000: ^ [23:30:58] RECOVERY - cp30 Stunnel HTTP for test101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14642 bytes in 0.653 second response time [23:31:00] ack [23:31:04] RECOVERY - test101 JobRunner Service on test101 is OK: PROCS OK: 1 process with args 'redisJobRunnerService' [23:31:07] RECOVERY - test101 JobChron Service on test101 is OK: PROCS OK: 1 process with args 'redisJobChronService' [23:31:07] RECOVERY - test101 Redis Process on test101 is OK: PROCS OK: 1 process with args 'redis-server' [23:31:07] RECOVERY - test101 SSH on test101 is OK: SSH OK - OpenSSH_8.4p1 Debian-5 (protocol 2.0) [23:31:08] RECOVERY - test101 Disk Space on test101 is OK: DISK OK - free space: / 4469 MB (25% inode=62%); [23:31:16] RECOVERY - cp20 Stunnel HTTP for test101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14642 bytes in 8.805 second response time [23:31:17] RECOVERY - test101 NTP time on test101 is OK: NTP OK: Offset -0.02987059951 secs [23:31:17] RECOVERY - test101 PowerDNS Recursor on test101 is OK: DNS OK: 1.866 second response time. miraheze.org returns 198.244.148.90,2001:41d0:801:2000::1b80,2001:41d0:801:2000::4c25,51.195.220.68 [23:31:18] RECOVERY - test101 ferm_active on test101 is OK: OK ferm input default policy is set [23:31:23] RECOVERY - test101 Check Gluster Clients on test101 is OK: PROCS OK: 1 process with args '/usr/sbin/glusterfs' [23:31:26] RECOVERY - test101 php-fpm on test101 is OK: PROCS OK: 13 processes with command name 'php-fpm7.4' [23:31:26] RECOVERY - test101 Puppet on test101 is OK: OK: Puppet is currently enabled, last run 35 minutes ago with 0 failures [23:32:44] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.88, 11.16, 10.15 [23:33:11] RECOVERY - cp31 Stunnel HTTP for test101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14642 bytes in 0.319 second response time [23:33:27] RECOVERY - test101 MediaWiki Rendering on test101 is OK: HTTP OK: HTTP/1.1 200 OK - 51269 bytes in 1.258 second response time [23:34:42] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.63, 10.26, 9.95 [23:34:59] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.21, 11.59, 10.65 [23:36:39] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.72, 9.63, 9.74 [23:36:55] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.24, 11.21, 10.64 [23:40:33] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.31, 10.39, 10.09 [23:40:48] PROBLEM - test101 MediaWiki Rendering on test101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:41:28] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.56, 7.19, 6.24 [23:42:31] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.30, 9.79, 9.91 [23:43:23] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.03, 7.26, 6.39 [23:46:30] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.47, 1.78, 1.51 [23:47:14] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 4.79, 6.26, 6.20 [23:48:12] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/a7279f64d01c [23:48:14] [02miraheze/mw-config] 07Universal-Omega 03a7279f6 - set $wgScribuntoEngineConf fully [23:48:15] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-3 - 13https://github.com/miraheze/mw-config [23:48:17] [02mw-config] 07Universal-Omega opened pull request 03#4653: set $wgScribuntoEngineConf fully - 13https://github.com/miraheze/mw-config/pull/4653 [23:48:27] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.41, 1.60, 1.47 [23:48:30] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.55, 3.53, 3.94 [23:48:53] [02mw-config] 07Universal-Omega closed pull request 03#4652: Only load the used instance of wgObjectCaches - 13https://github.com/miraheze/mw-config/pull/4652 [23:48:54] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/daf5c016e970...6df6cf971b08 [23:48:56] [02miraheze/mw-config] 07Universal-Omega 036df6cf9 - Only load the used instance of wgObjectCaches (#4652) [23:48:58] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 - 13https://github.com/miraheze/mw-config [23:48:59] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 [23:49:01] [02miraheze/mw-config] 07github-actions[bot] pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/a7279f64d01c...c91fe7b9118c [23:49:02] [02miraheze/mw-config] 07github-actions 03c91fe7b - CI: lint code to MediaWiki standards [23:49:03] [02mw-config] 07github-actions[bot] synchronize pull request 03#4653: set $wgScribuntoEngineConf fully - 13https://github.com/miraheze/mw-config/pull/4653 [23:49:22] !log [universalomega@test101] starting deploy of {'pull': 'config', 'config': True} to all [23:49:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:49:36] miraheze/mw-config - Universal-Omega the build passed. [23:49:37] !log [universalomega@test101] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 14s [23:49:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:49:59] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True} to all [23:50:01] miraheze/mw-config - Universal-Omega the build passed. [23:50:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:50:22] !log [universalomega@mwtask111] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 22s [23:50:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:50:34] [02mw-config] 07Universal-Omega closed pull request 03#4653: set $wgScribuntoEngineConf fully - 13https://github.com/miraheze/mw-config/pull/4653 [23:50:36] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/6df6cf971b08...5e393c2a5277 [23:50:37] [02miraheze/mw-config] 07Universal-Omega 035e393c2 - set $wgScribuntoEngineConf fully (#4653) [23:50:39] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-3 - 13https://github.com/miraheze/mw-config [23:50:40] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-3 [23:50:44] !log [universalomega@test101] starting deploy of {'pull': 'config', 'config': True} to all [23:50:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:50:58] !log [universalomega@test101] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 14s [23:51:01] RECOVERY - test101 MediaWiki Rendering on test101 is OK: HTTP OK: HTTP/1.1 200 OK - 51269 bytes in 4.009 second response time [23:51:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:51:19] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.34, 10.47, 10.15 [23:51:20] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True} to all [23:51:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:51:36] !log [universalomega@mwtask111] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 16s [23:51:39] miraheze/mw-config - Universal-Omega the build passed. [23:51:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:53:17] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 10.19, 10.19, 10.08 [23:54:17] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/5e393c2a5277...8dcbc7e2b395 [23:54:19] [02miraheze/mw-config] 07Universal-Omega 038dcbc7e - Remove wgMaxShellMemory from LocalSettings.php [23:54:25] !log [universalomega@test101] starting deploy of {'pull': 'config', 'config': True} to all [23:54:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:54:38] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True} to all [23:54:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:54:49] !log [universalomega@test101] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 23s [23:54:54] !log [universalomega@mwtask111] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 16s [23:55:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:55:17] miraheze/mw-config - Universal-Omega the build passed. [23:55:19] PROBLEM - test101 MediaWiki Rendering on test101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:55:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:56:23] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.56, 3.92, 3.82 [23:57:20] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/8dcbc7e2b395...57115bb01678 [23:57:21] [02miraheze/mw-config] 07Universal-Omega 0357115bb - wgDeleteRevisionsLimit should be integer [23:57:27] !log [universalomega@test101] starting deploy of {'pull': 'config', 'config': True} to all [23:57:28] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True} to all [23:57:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:57:41] !log [universalomega@mwtask111] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 12s [23:57:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:57:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:58:14] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.79, 10.59, 9.94 [23:58:14] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.19, 9.80, 10.15 [23:58:25] miraheze/mw-config - Universal-Omega the build passed. [23:58:32] !log [universalomega@test101] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 64s [23:58:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log