[00:02:05] [02CreateWiki] 07Universal-Omega edited pull request 03#233: Cleanup Echo notifications code and fix Echo email notifications - 13https://git.io/JBSa3 [00:02:45] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.70, 7.24, 6.88 [00:03:49] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 12.93, 8.26, 7.20 [00:03:57] PROBLEM - graylog2 Puppet on graylog2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:03:59] RECOVERY - cp12 Disk Space on cp12 is OK: DISK OK - free space: / 7823 MB (20% inode=96%); [00:05:27] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 27.44, 22.62, 19.46 [00:06:59] PROBLEM - wiki.mikrodev.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.mikrodev.com could not be found [00:09:49] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.21, 7.84, 7.53 [00:12:43] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 3.97, 6.54, 7.31 [00:13:47] RECOVERY - wiki.mikrodev.com - reverse DNS on sslhost is OK: SSL OK - wiki.mikrodev.com reverse DNS resolves to cp13.miraheze.org - CNAME OK [00:15:10] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 15.35, 21.04, 20.81 [00:17:07] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 14.52, 19.26, 20.21 [00:17:49] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.45, 7.68, 7.37 [00:20:40] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.95, 8.03, 7.59 [00:21:04] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 18.04, 20.45, 20.58 [00:22:38] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.84, 7.63, 7.51 [00:23:48] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.84, 7.93, 7.66 [00:24:58] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 17.21, 19.39, 20.15 [00:31:54] RECOVERY - graylog2 Puppet on graylog2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:33:48] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 10.47, 8.53, 7.86 [00:34:35] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 13.33, 8.76, 7.62 [00:38:34] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.23, 7.46, 7.41 [00:39:48] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.56, 7.50, 7.77 [00:42:32] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.16, 8.16, 7.66 [00:44:32] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.50, 7.78, 7.57 [00:49:36] PROBLEM - db13 Current Load on db13 is CRITICAL: CRITICAL - load average: 23.60, 11.82, 5.53 [00:49:48] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.60, 7.60, 7.47 [00:52:28] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.01, 8.10, 7.62 [00:53:34] RECOVERY - db13 Current Load on db13 is OK: OK - load average: 1.67, 6.03, 4.58 [00:54:25] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.80, 1.60, 1.31 [00:56:24] PROBLEM - cp12 Current Load on cp12 is CRITICAL: CRITICAL - load average: 2.10, 1.68, 1.36 [00:58:27] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.49, 7.72, 7.66 [01:00:20] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 1.43, 1.68, 1.44 [01:01:48] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.44, 7.30, 7.72 [01:04:24] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 10.08, 7.55, 7.41 [01:06:22] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.37, 7.10, 7.27 [01:10:14] PROBLEM - cp12 Current Load on cp12 is CRITICAL: CRITICAL - load average: 1.42, 2.01, 1.70 [01:12:13] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.57, 1.81, 1.66 [01:12:19] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.43, 6.02, 6.70 [01:14:14] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 0.62, 1.42, 1.54 [01:16:17] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.32, 6.76, 6.86 [01:17:49] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.91, 6.20, 6.75 [01:18:17] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 6.24, 6.44, 6.72 [01:33:34] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.74, 3.18, 2.91 [01:35:33] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.08, 3.23, 2.97 [01:36:16] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.05, 6.94, 6.57 [01:36:23] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.59, 7.20, 6.61 [01:38:10] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.58, 6.09, 6.30 [01:38:23] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.42, 6.40, 6.38 [02:05:14] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 11.65, 8.42, 6.95 [02:05:21] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.12, 7.93, 6.56 [02:07:20] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.33, 7.34, 6.51 [02:09:05] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.62, 7.39, 6.90 [02:09:08] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.53, 3.54, 3.19 [02:11:04] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.43, 3.06, 3.05 [02:11:20] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 10.50, 8.68, 7.23 [02:15:19] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.54, 7.70, 7.19 [02:16:47] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.88, 8.18, 7.40 [02:18:41] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.48, 7.14, 7.10 [02:20:35] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.57, 7.71, 7.32 [02:22:30] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.05, 7.08, 7.13 [02:23:15] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.45, 6.38, 6.77 [02:26:18] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.19, 6.07, 6.71 [02:26:59] [02miraheze/MirahezeMagic] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/JRmKI [02:27:00] [02miraheze/MirahezeMagic] 07Universal-Omega 03e1e8472 - Fix error handling [02:27:02] [02MirahezeMagic] 07Universal-Omega created branch 03Universal-Omega-patch-2 - 13https://git.io/fQRGX [02:27:03] [02MirahezeMagic] 07Universal-Omega opened pull request 03#279: Fix error handling - 13https://git.io/JRmKL [02:28:03] miraheze/MirahezeMagic - Universal-Omega the build passed. [02:31:35] [02miraheze/MirahezeMagic] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/JRmKh [02:31:37] [02miraheze/MirahezeMagic] 07Universal-Omega 0356b1520 - Update en.json [02:31:38] [02MirahezeMagic] 07Universal-Omega synchronize pull request 03#279: Fix error handling - 13https://git.io/JRmKL [02:32:46] miraheze/MirahezeMagic - Universal-Omega the build passed. [02:51:05] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.06, 6.81, 6.45 [02:54:51] [02miraheze/MirahezeMagic] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/JRmD2 [02:54:52] [02miraheze/MirahezeMagic] 07Universal-Omega 03b839e86 - Update ReCaptchaNoCaptchaHooks.php [02:54:54] [02MirahezeMagic] 07Universal-Omega synchronize pull request 03#279: Fix error handling - 13https://git.io/JRmKL [02:54:54] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.76, 6.71, 6.51 [02:55:41] miraheze/MirahezeMagic - Universal-Omega the build passed. [03:00:40] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.06, 7.22, 6.72 [03:02:36] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.02, 6.67, 6.58 [03:05:58] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.44, 7.15, 6.38 [03:06:27] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.90, 7.04, 6.83 [03:07:57] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.56, 6.17, 6.11 [03:08:21] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.54, 6.38, 6.61 [03:33:34] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.44, 3.20, 2.91 [03:35:35] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.21, 3.31, 2.99 [04:53:31] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.30, 6.16, 5.19 [04:53:48] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.19, 6.40, 5.76 [04:55:32] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 6.06, 5.92, 5.21 [04:55:48] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.41, 5.97, 5.68 [04:59:11] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.09, 6.96, 6.12 [05:01:11] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.58, 7.33, 6.35 [05:05:48] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.52, 7.99, 6.80 [05:07:48] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.26, 7.22, 6.66 [05:09:12] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.89, 7.37, 7.02 [05:09:49] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.91, 6.76, 6.56 [05:11:13] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.95, 7.93, 7.25 [05:15:49] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.07, 6.86, 6.70 [05:17:11] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.88, 7.56, 7.35 [05:17:49] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.82, 6.43, 6.55 [05:23:50] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.73, 6.81, 6.70 [05:25:48] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.42, 6.13, 6.47 [05:33:06] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.01, 6.86, 6.98 [05:35:04] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.37, 7.26, 7.14 [05:37:22] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JRYW1 [05:37:24] [02miraheze/mw-config] 07Reception123 033a65a2d - fix default for egChameleonLayoutFile [05:38:17] miraheze/mw-config - Reception123 the build passed. [05:39:03] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.06, 6.29, 6.78 [05:41:54] [02puppet] 07Universal-Omega opened pull request 03#1852: Revert "T7718: Remove stunnel check for test3" - 13https://git.io/JRYlw [05:43:07] [02puppet] 07Reception123 commented on pull request 03#1852: Revert "T7718: Remove stunnel check for test3" - 13https://git.io/JRYlA [05:43:11] [02puppet] 07Reception123 closed pull request 03#1852: Revert "T7718: Remove stunnel check for test3" - 13https://git.io/JRYlw [05:43:12] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±3] 13https://git.io/JRYlp [05:43:14] [02miraheze/puppet] 07Universal-Omega 038a877bc - Revert "T7718: Remove stunnel check for test3 (#1834)" (#1852) [05:44:20] !log [@test3] starting deploy of {'config': True} to skip [05:44:21] !log [@test3] finished deploy of {'config': True} to skip - FAIL: [256] [05:44:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:44:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:45:58] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.19, 7.25, 6.96 [05:47:13] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.66, 7.39, 6.90 [05:47:57] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.73, 7.10, 6.95 [05:49:07] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.72, 7.67, 7.04 [05:49:56] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 13.21, 8.94, 7.61 [05:52:57] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.31, 7.41, 7.13 [05:53:53] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.22, 7.85, 7.46 [05:56:47] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.05, 7.69, 7.30 [06:00:35] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.53, 7.27, 7.24 [06:03:49] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 14.26, 8.83, 7.77 [06:04:27] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 11.77, 8.59, 7.67 [06:04:31] [02puppet] 07Universal-Omega opened pull request 03#1853: Remove stunnel http monitoring for test3 - 13https://git.io/JRYue [06:04:54] [02puppet] 07Reception123 closed pull request 03#1853: Remove stunnel http monitoring for test3 - 13https://git.io/JRYue [06:04:55] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JRYum [06:04:57] [02miraheze/puppet] 07Universal-Omega 036ebabe4 - Remove stunnel http monitoring for test3 (#1853) [06:05:01] !log securitywiki to securipediawiki [06:05:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:05:13] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JRYuc [06:05:15] [02miraheze/services] 07MirahezeSSLBot 036f4cd68 - BOT: Updating services config for wikis [06:06:33] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JRYuy [06:06:35] [02miraheze/mw-config] 07Reception123 030fb7d51 - blacklist security in CreateWiki [06:07:38] miraheze/mw-config - Reception123 the build passed. [06:09:44] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://git.io/JRYzA [06:09:46] [02miraheze/mw-config] 07Universal-Omega 03d80da6f - securitywiki -> securipediawiki [06:09:47] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-3 - 13https://git.io/vbvb3 [06:09:49] [02mw-config] 07Universal-Omega opened pull request 03#4036: securitywiki -> securipediawiki - 13https://git.io/JRYzh [06:10:53] miraheze/mw-config - Universal-Omega the build passed. [06:13:05] [02mw-config] 07Reception123 closed pull request 03#4036: securitywiki -> securipediawiki - 13https://git.io/JRYzh [06:13:06] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JRYgx [06:13:08] [02miraheze/mw-config] 07Universal-Omega 0385bb9b8 - securitywiki -> securipediawiki (#4036) [06:13:09] [02mw-config] 07Reception123 deleted branch 03Universal-Omega-patch-3 - 13https://git.io/vbvb3 [06:13:11] [02miraheze/mw-config] 07Reception123 deleted branch 03Universal-Omega-patch-3 [06:13:57] miraheze/mw-config - Reception123 the build passed. [06:31:37] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.95, 6.81, 7.75 [06:33:04] !log [@test3] starting deploy of {'config': True} to skip [06:33:05] !log [@test3] finished deploy of {'config': True} to skip - FAIL: [256] [06:33:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:33:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:33:36] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 12.52, 8.58, 8.25 [06:47:26] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.77, 7.48, 7.93 [06:49:25] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 10.54, 8.84, 8.38 [06:56:53] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.34, 7.24, 6.08 [06:56:53] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.15, 8.68, 6.78 [06:58:49] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.30, 6.51, 5.95 [06:58:52] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.35, 7.94, 6.75 [07:02:50] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.10, 7.46, 6.78 [07:04:50] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.68, 7.17, 6.75 [07:06:49] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.15, 6.33, 6.49 [07:13:20] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.77, 6.92, 7.85 [07:19:17] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.92, 8.50, 8.21 [07:25:14] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.91, 7.62, 7.97 [07:26:07] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.36, 7.02, 7.90 [07:30:01] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 10.50, 7.70, 7.92 [07:31:11] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.43, 8.18, 8.01 [07:32:01] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.00, 7.28, 7.74 [07:34:01] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 11.17, 8.72, 8.21 [07:37:10] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.44, 7.70, 7.87 [07:40:01] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.47, 7.39, 7.87 [07:49:06] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.59, 7.17, 7.16 [07:50:03] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.13, 7.68, 7.54 [07:51:05] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.29, 7.34, 7.24 [07:54:01] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.22, 7.15, 7.44 [07:55:04] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.17, 6.18, 6.76 [08:02:01] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.87, 5.67, 6.68 [08:03:03] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.90, 6.87, 6.70 [08:06:01] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.12, 7.44, 7.23 [08:08:01] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.73, 6.83, 7.04 [08:09:00] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.48, 7.80, 7.21 [08:13:00] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.72, 7.26, 7.13 [08:20:54] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 10.40, 8.29, 7.55 [08:24:51] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.75, 7.91, 7.62 [08:26:03] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.03, 6.00, 6.66 [08:30:48] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.22, 7.02, 7.22 [08:34:53] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 10.25, 8.40, 7.30 [08:36:47] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.84, 7.97, 7.26 [08:38:42] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.60, 7.72, 7.60 [08:44:28] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.75, 6.06, 6.63 [08:48:40] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.57, 5.78, 6.63 [09:34:32] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 8.00, 6.97, 6.25 [09:36:28] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.34, 6.81, 6.18 [09:38:27] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.85, 6.43, 6.12 [09:40:16] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.33, 6.17, 6.22 [10:35:03] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.83, 7.33, 6.23 [10:36:01] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.26, 7.44, 6.28 [10:37:03] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.11, 7.02, 6.27 [10:39:03] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.74, 6.62, 6.22 [10:40:01] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.53, 6.26, 6.09 [11:33:36] !log [reception@jobrunner3] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=millerpediawiki /home/reception/millerpedia_pages_full.xml --username-prefix wikia:millerpedia --report 1 (END - exit=2) [11:33:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:39:28] Reception123: ^ [11:40:10] RhinosF1: I saw, I exited because for some reason it didn't log before thay [11:40:18] It ran without logging so that was strange [12:10:48] Ok [12:17:56] Reception123: why do you have so many issues [12:20:22] * Reception123 has no clue [12:20:56] Reception123: it's just shelling out what you type [12:21:11] Also did you merge config PRs? [12:35:10] which ones? [12:40:06] Reception123: any [12:40:18] I see failed auto deploys on SAL [12:40:20] This morning [12:41:22] if you mean mw-config I don't see anything I would feel comfortable merging immediately [12:41:57] You know the 2 PRs you merged? [12:42:06] Well not PRs simple commits [12:42:24] https://github.com/miraheze/mw-config/commit/3a65a2d487970ee6b5463020ffb31bf4881cabed & https://github.com/miraheze/mw-config/commit/0fb7d51d4e1f41726a68a21ff254334f902d8469 [12:42:25] [url] fix default for egChameleonLayoutFile · miraheze/mw-config@3a65a2d · GitHub | github.com [12:42:27] [url] blacklist security in CreateWiki · miraheze/mw-config@0fb7d51 · GitHub | github.com [13:08:13] PROBLEM - thestarsareright.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'thestarsareright.org' expires in 15 day(s) (Sun 22 Aug 2021 13:05:29 GMT +0000). [13:12:46] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JR348 [13:12:47] [02miraheze/ssl] 07MirahezeSSLBot 03686182c - Bot: Update SSL cert for thestarsareright.org [13:29:00] RECOVERY - thestarsareright.org - LetsEncrypt on sslhost is OK: OK - Certificate 'thestarsareright.org' will expire on Thu 04 Nov 2021 12:12:37 GMT +0000. [13:29:11] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb [13:29:21] PROBLEM - cp3 Current Load on cp3 is CRITICAL: CRITICAL - load average: 4.09, 2.07, 1.01 [13:29:27] PROBLEM - cp12 Current Load on cp12 is CRITICAL: CRITICAL - load average: 3.63, 2.11, 1.27 [13:29:38] Huh [13:29:42] * RhinosF1 here [13:29:57] RhinosF1: oh you mean those [13:30:14] I didn't realise since they were my own commits I thought you meant someone else's [13:30:20] Reception123: cp3 is down [13:30:22] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 51.38.69.175/cpweb, 2001:41d0:801:2000::58af/cpweb, 51.222.25.132/cpweb, 2607:5300:205:200::1c30/cpweb [13:31:01] Oh shit [13:31:05] I'll check if there's anything in logs but I can SSH fine for now [13:31:19] huh in the meantime it seems everything is down [13:31:21] PROBLEM - phab-storage.bots.miraheze.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:31:33] [02mw-config] 07R4356th opened pull request 03#4037: r4356thwiki -> redminwiki - 13https://git.io/JR3EA [13:31:47] 12 too [13:32:08] seems back now [13:32:11] Reception123: logs [13:32:15] PROBLEM - cp3 HTTPS on cp3 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:32:25] [02mw-config] 07Reception123 closed pull request 03#4037: r4356thwiki -> redminwiki - 13https://git.io/JR3EA [13:32:27] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JR3uq [13:32:28] [02miraheze/mw-config] 07R4356th 03e32892b - r4356thwiki -> redminwiki (#4037) [13:32:30] miraheze/mw-config - R4356th the build passed. [13:32:35] Reception123: no [13:32:42] Don't deploy mid incident [13:33:08] We're still seeing high error rates [13:33:21] !log [@test3] starting deploy of {'config': True} to skip [13:33:22] miraheze/mw-config - Reception123 the build passed. [13:33:22] !log [@test3] finished deploy of {'config': True} to skip - FAIL: [256] [13:33:22] I don't think a one word PR will change anything to a cp issue [13:33:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:33:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:33:36] and definitely no OOM in the kern logs as far as I can see [13:34:03] I think it's one of those incidents that occasionally happen where it says all datacentres down only to fix itself less than 1 minute after [13:34:15] afaik we've never managed to figure out why they happen / find any logs to explain it [13:34:32] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 0.75, 1.90, 1.50 [13:34:44] PROBLEM - cp3 Varnish Backends on cp3 is WARNING: No backends detected. If this is an error, see readme.txt [13:34:50] PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is CRITICAL: CRITICAL - NGINX Error Rate is 99% [13:35:11] though cp3 is clearly problematic, but that doesn't explain why the others went down temporarily [13:35:15] Reception123: load is stupidly high [13:35:49] Deploys shouldn't happen because it means a) you're not focused on the incident and b) it can have silly side effects we don't expect [13:36:01] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.48, 6.10, 5.10 [13:36:09] (Doing that can purge cache which means load goes higher) [13:36:28] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 0.93, 1.59, 1.43 [13:36:34] RECOVERY - cp3 Current Load on cp3 is OK: OK - load average: 0.16, 1.65, 1.40 [13:36:41] Reception123: can we manually depool & reboot cp3 [13:37:00] I was just about to suggest that myself, as I think that's the only thing we can do right now [13:37:18] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.92, 19.35, 16.34 [13:37:26] [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JR3zi [13:37:27] [02miraheze/dns] 07Reception123 03ff3b9ca - depool cp13 [13:37:57] [02dns] 07RhinosF1 opened pull request 03#211: Depool cp3; stuck in a very upset state - 13https://git.io/JR3z9 [13:38:02] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 6.24, 5.90, 5.14 [13:38:02] PROBLEM - phab-storage.bots.miraheze.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate '*.bots.miraheze.wiki' expires in 10 day(s) (Tue 17 Aug 2021 04:56:02 GMT +0000). [13:38:10] Reception123: why 13 [13:38:16] typo [13:38:20] [02dns] 07RhinosF1 closed pull request 03#211: Depool cp3; stuck in a very upset state - 13https://git.io/JR3z9 [13:38:33] !log cp3 not cp13 depooled [13:38:35] running puppet now [13:38:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:39:58] I see spikes everywhere but load was much worse on cp3 [13:40:03] .op [13:40:03] Attempting to OP... [13:40:14] well the other spikes are very likely because cp3 isn't working anymore so traffic is getting redirected [13:40:45] Possibly [13:41:16] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 20.00, 19.95, 17.26 [13:41:37] Reception123: let's leave it depooled until icinga is completely happy with it [13:41:46] Might be worth checking logs for a hint too [13:41:51] !log rebooted cp3 [13:41:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:42:02] so far I couldn't find anything useful in logs [13:42:02] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.32, 6.99, 5.85 [13:42:36] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 7 backends are healthy [13:42:38] Reception123: you'd be best looking in nginx logs [13:43:01] Either that or kernel [13:43:09] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [13:43:19] kernel I already checked and nothing for OOM [13:43:23] Error rate is high so let's wait [13:43:26] It wasn't OOM [13:43:30] More OOCPU [13:43:34] (It won't say that) [13:43:59] Next state check is in 30 seconds [13:44:01] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 11.09, 7.72, 6.20 [13:44:15] RECOVERY - cp3 HTTPS on cp3 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 2254 bytes in 1.201 second response time [13:44:16] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [13:44:41] RECOVERY - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is OK: OK - NGINX Error Rate is 4% [13:44:52] That's better [13:45:10] Icinga looks happy [13:45:16] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.03, 21.32, 18.48 [13:45:23] Beyond cpu usage completely maxed out, not sure what to say [13:46:01] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.94, 7.46, 6.30 [13:46:10] let's repool [13:46:16] [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JR327 [13:46:17] [02miraheze/dns] 07Reception123 0336a95f7 - repool cp3 [13:46:33] Reception123: keep an eye out though [13:46:45] It might be worth pinging void to follow up [13:46:47] yeah, will look [13:48:05] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 11.38, 8.55, 6.82 [13:50:07] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 10.76, 8.77, 6.90 [13:51:20] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 20.00, 20.32, 18.86 [13:52:06] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.58, 7.95, 6.83 [13:54:02] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 4.97, 7.95, 7.42 [13:56:02] Reception123: looks ok [13:56:10] seems like it yeah [13:58:05] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.77, 6.23, 6.42 [13:58:57] Reception123: the only thing I can see is disk space for the varnish mount dropping [13:59:08] So varnish logs might give a hint [13:59:25] that's interesting [14:02:04] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.20, 5.57, 6.49 [14:06:01] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.14, 7.09, 7.00 [14:06:02] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.61, 19.73, 19.32 [14:08:04] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 18.33, 19.39, 19.26 [14:10:03] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 6.58, 6.46, 6.74 [14:10:15] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.63, 5.13, 4.12 [14:14:17] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 2.87, 4.35, 4.07 [15:01:44] .deop [15:01:44] Attempting to OP... [15:35:24] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.01, 6.44, 5.73 [15:37:22] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.05, 5.84, 5.59 [16:05:07] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.91, 7.01, 6.23 [16:05:24] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.21, 7.35, 6.44 [16:07:21] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 6.02, 6.63, 6.28 [16:09:07] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.60, 6.24, 6.14 [16:17:02] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.43, 7.08, 6.54 [16:19:02] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.41, 7.49, 6.61 [16:20:51] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.25, 7.35, 6.76 [16:21:03] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.48, 7.72, 6.82 [16:23:02] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.57, 7.94, 7.00 [16:24:40] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 6.54, 6.74, 6.63 [16:25:02] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.38, 7.66, 7.01 [16:27:16] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.89, 6.69, 5.94 [16:29:00] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.73, 6.58, 6.72 [16:29:16] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.39, 6.13, 5.82 [16:32:55] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.47, 7.15, 6.89 [16:35:15] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.38, 6.75, 6.13 [16:36:14] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.20, 7.84, 7.04 [16:36:54] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.92, 7.70, 7.28 [16:38:09] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.40, 7.24, 6.91 [16:38:52] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.64, 7.79, 7.35 [16:39:14] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.50, 6.50, 6.21 [16:42:53] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.69, 7.92, 7.56 [16:44:01] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.89, 6.51, 6.79 [16:50:50] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 7.52, 8.07, 7.72 [16:52:01] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.43, 7.00, 6.84 [16:52:49] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.45, 7.98, 7.73 [16:54:01] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.70, 6.64, 6.74 [16:56:09] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.19, 6.26, 5.78 [16:58:05] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.76, 5.95, 5.72 [17:02:47] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.05, 7.25, 7.42 [17:06:01] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.31, 7.33, 7.01 [17:08:42] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.91, 7.54, 7.69 [17:14:38] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.28, 7.65, 7.63 [17:18:02] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.29, 7.59, 7.23 [17:20:03] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.14, 6.97, 7.04 [17:20:35] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.84, 7.62, 7.66 [17:30:02] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.40, 7.69, 7.35 [17:32:01] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.67, 7.34, 7.26 [17:33:40] [02puppet] 07Universal-Omega opened pull request 03#1854: Remove $wgReCaptchaSiteKey from PrivateSettings - 13https://git.io/JRs4u [17:34:01] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 11.31, 8.31, 7.59 [17:34:30] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 10.88, 8.44, 7.74 [17:34:51] [02puppet] 07Universal-Omega synchronize pull request 03#1854: Remove $wgReCaptchaSiteKey from PrivateSettings - 13https://git.io/JRs4u [17:35:44] [02puppet] 07Universal-Omega synchronize pull request 03#1854: Remove $wgReCaptchaSiteKey from PrivateSettings - 13https://git.io/JRs4u [17:38:02] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.04, 7.66, 7.51 [17:39:07] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.73, 6.54, 5.88 [17:40:01] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.67, 7.96, 7.64 [17:40:28] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.97, 7.28, 7.47 [17:41:04] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 6.07, 6.24, 5.84 [17:41:26] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://git.io/JRsBV [17:41:28] [02miraheze/mw-config] 07Universal-Omega 03099bdac - Update LocalSettings.php [17:41:29] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-3 - 13https://git.io/vbvb3 [17:41:59] [02mw-config] 07Universal-Omega opened pull request 03#4038: Add wgReCaptchaSiteKey - 13https://git.io/JRsBK [17:42:01] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.99, 7.57, 7.53 [17:43:00] miraheze/mw-config - Universal-Omega the build passed. [17:48:27] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.61, 7.13, 7.25 [17:50:01] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.57, 8.02, 7.65 [17:54:01] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.13, 7.57, 7.56 [17:54:29]