[00:04:00] PROBLEM - graylog2 Puppet on graylog2 is CRITICAL: CRITICAL: Puppet has 24 failures. Last run 3 minutes ago with 24 failures. Failed resources (up to 3 shown) [00:05:07] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 32.39, 21.92, 15.92 [00:09:07] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.90, 23.17, 17.85 [00:10:00] RECOVERY - graylog2 Puppet on graylog2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:13:07] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.99, 23.22, 18.97 [00:14:00] [02mw-config] 07Njsmooth reviewed pull request 03#3994 commit - 13https://git.io/Jchz7 [00:19:07] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 18.99, 23.48, 20.71 [00:21:56] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.81, 6.86, 5.69 [00:23:07] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.67, 22.78, 20.92 [00:23:52] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.50, 6.58, 5.72 [00:25:07] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.45, 23.18, 21.36 [00:29:07] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 12.49, 18.74, 20.08 [00:53:44] PROBLEM - cp12 Current Load on cp12 is CRITICAL: CRITICAL - load average: 2.12, 1.62, 1.11 [00:55:42] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 0.92, 1.32, 1.06 [01:14:59] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: SSL OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org - CNAME OK [01:44:41] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [01:50:25] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.66, 1.86, 1.40 [01:54:22] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 1.42, 1.61, 1.40 [05:53:54] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: SSL OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org - CNAME OK [06:02:45] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [06:08:52] PROBLEM - cp12 Current Load on cp12 is CRITICAL: CRITICAL - load average: 2.14, 1.54, 1.14 [06:10:52] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 1.36, 1.41, 1.13 [06:16:36] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: SSL OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org - CNAME OK [06:25:28] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [07:34:40] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: SSL OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org - CNAME OK [07:50:28] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [08:04:19] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: SSL OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org - CNAME OK [08:13:10] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [08:51:29] Universal_Omega: minor but https://gerrit.wikimedia.org/r/c/mediawiki/extensions/NativeSvgHandler/+/703887 [09:12:03] PROBLEM - wiki.bikinibottomdiscord.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.bikinibottomdiscord.org' expires in 15 day(s) (Tue 27 Jul 2021 09:07:46 GMT +0000). [09:14:36] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JCeCX [09:14:37] [02miraheze/ssl] 07MirahezeSSLBot 03585802e - Bot: Update SSL cert for wiki.bikinibottomdiscord.org [09:20:21] PROBLEM - puardian.xyz - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'puardian.xyz' expires in 15 day(s) (Tue 27 Jul 2021 09:14:30 GMT +0000). [09:22:47] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JCelA [09:22:49] [02miraheze/ssl] 07MirahezeSSLBot 030cc7cc0 - Bot: Update SSL cert for puardian.xyz [09:32:28] RECOVERY - wiki.bikinibottomdiscord.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.bikinibottomdiscord.org' will expire on Sat 09 Oct 2021 08:14:29 GMT +0000. [10:00:40] RECOVERY - puardian.xyz - LetsEncrypt on sslhost is OK: OK - Certificate 'puardian.xyz' will expire on Sat 09 Oct 2021 08:22:40 GMT +0000. [10:01:57] [02MirahezeMagic] 07Universal-Omega closed pull request 03#271: Replace Freenode with Libera.Chat - 13https://git.io/JcpNM [10:01:58] [02miraheze/MirahezeMagic] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JCegz [10:02:00] [02miraheze/MirahezeMagic] 07R4356th 03ee6a2de - Replace Freenode with Libera.Chat (#271) [10:02:50] miraheze/MirahezeMagic - Universal-Omega the build passed. [10:02:59] RhinosF1:seen thanks [10:28:28] thanks to all admins, engineers and volunteers in general for the good work [11:47:49] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: SSL OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org - CNAME OK [12:32:01] PROBLEM - wiki.beergeeks.co.il - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.beergeeks.co.il' expires in 15 day(s) (Tue 27 Jul 2021 12:23:18 GMT +0000). [12:38:57] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JCex6 [12:38:59] [02miraheze/ssl] 07MirahezeSSLBot 039eb9183 - Bot: Update SSL cert for wiki.beergeeks.co.il [12:45:41] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [12:59:35] RECOVERY - wiki.beergeeks.co.il - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.beergeeks.co.il' will expire on Sat 09 Oct 2021 11:38:50 GMT +0000. [15:07:30] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.41, 7.02, 5.82 [15:09:05] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.07, 7.69, 6.41 [15:10:45] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 12.18, 9.60, 7.21 [15:11:27] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.44, 7.86, 6.55 [15:13:03] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.93, 7.75, 6.79 [15:14:45] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.08, 7.50, 6.88 [15:15:24] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.39, 6.23, 6.18 [15:17:03] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.10, 6.47, 6.50 [15:18:45] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.76, 6.55, 6.64 [15:32:48] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 10.86, 7.53, 6.64 [15:33:05] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.26, 7.15, 6.28 [15:34:57] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.51, 6.94, 5.99 [15:35:22] RECOVERY - www.burnout.wiki - reverse DNS on sslhost is OK: SSL OK - www.burnout.wiki reverse DNS resolves to cp13.miraheze.org - CNAME OK [15:35:44] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.08, 8.10, 6.74 [15:36:57] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.27, 6.19, 5.83 [15:37:41] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.33, 7.47, 6.67 [15:39:37] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.00, 6.74, 6.51 [15:46:46] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.35, 7.40, 7.48 [15:47:04] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.45, 7.35, 7.71 [15:48:47] dmehus: hi [15:50:47] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.76, 5.60, 6.69 [15:52:06] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.87, 1.47, 1.16 [15:54:05] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 0.99, 1.24, 1.10 [15:57:04] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.61, 6.05, 6.73 [16:05:38] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.75, 7.16, 6.73 [16:05:55] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.82, 6.93, 6.03 [16:07:37] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 6.05, 6.78, 6.65 [16:07:51] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.16, 6.61, 6.01 [16:11:29] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.25, 7.41, 6.91 [16:13:28] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.75, 7.51, 7.03 [16:16:50] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.13, 7.58, 7.09 [16:17:27] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.37, 8.33, 7.46 [16:18:47] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.77, 6.93, 6.91 [16:19:28] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.37, 7.83, 7.37 [16:20:44] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.86, 7.67, 7.16 [16:22:40] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.72, 7.20, 7.05 [16:26:33] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.55, 6.43, 6.77 [16:29:21] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.02, 7.54, 7.31 [16:30:29] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.97, 7.50, 7.15 [16:31:21] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.74, 6.93, 7.12 [16:32:26] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.28, 8.34, 7.50 [16:33:19] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.68, 7.19, 7.17 [16:34:57] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.67, 6.76, 6.18 [16:35:18] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.93, 7.40, 7.25 [16:36:17] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 10.57, 7.65, 6.48 [16:36:57] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.96, 6.20, 6.05 [16:37:04] !log sudo chown -R www-data:www-data minecraftenchantmentswiki [16:37:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:38:13] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.70, 6.65, 6.24 [16:40:12] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.73, 7.59, 7.58 [16:41:15] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.15, 5.92, 6.63 [16:42:53] Thanks, Reception. [16:46:03] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.13, 7.91, 7.68 [16:48:00] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.06, 7.72, 7.65 [16:48:57] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.81, 6.26, 6.00 [16:51:54] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.58, 8.46, 7.97 [16:51:57] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.38, 6.82, 6.26 [16:53:53] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.51, 6.67, 6.28 [16:54:04] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.10, 7.73, 7.10 [16:54:57] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 6.37, 6.68, 6.34 [16:58:57] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.67, 7.30, 6.66 [16:59:47] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.07, 6.88, 6.50 [17:03:41] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 11.48, 7.93, 6.93 [17:04:25] no problem Bukkit :)( [17:07:25] I replied to you on WMF phab too [17:07:34] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.63, 7.16, 6.88 [17:10:57] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.19, 6.28, 6.64 [17:11:24] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 4.43, 6.39, 7.59 [17:11:26] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.01, 6.77, 6.79 [17:11:38] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.49, 7.48, 7.76 [17:13:24] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.34, 7.25, 7.73 [17:13:35] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.60, 7.85, 7.85 [17:15:21] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.54, 6.59, 7.44 [17:15:32] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.76, 7.88, 7.87 [17:17:29] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.50, 8.16, 7.97 [17:19:27] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.37, 7.85, 7.88 [17:21:23] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.31, 7.26, 7.00 [17:25:05] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.14, 7.77, 7.52 [17:25:22] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.33, 6.45, 6.76 [17:25:23] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 10.55, 8.58, 8.11 [17:27:04] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.04, 7.50, 7.45 [17:29:18] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 4.28, 6.80, 7.56 [17:33:13] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.13, 7.35, 7.54 [17:35:10] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.66, 7.18, 7.47 [17:39:06] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.44, 5.79, 6.57 [17:41:05] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.96, 5.79, 6.74 [17:45:34] @Universal_Omega: what actions https://phabricator.miraheze.org/T7619#153061 [17:45:35] [url] ⚓ T7619 A suggestion for a default global extension for the Miraheze farm | phabricator.miraheze.org [17:50:49] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.30, 6.28, 6.41 [17:52:47] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 6.62, 6.21, 6.36 [18:04:45] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.87, 6.85, 6.42 [18:06:45] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 6.19, 6.44, 6.31 [18:07:18] PROBLEM - graylog2 Disk Space on graylog2 is CRITICAL: DISK CRITICAL - free space: / 42697 MB (5% inode=99%); [18:09:18] PROBLEM - graylog2 Disk Space on graylog2 is WARNING: DISK WARNING - free space: / 43452 MB (6% inode=99%); [18:27:28] PROBLEM - www.wiki.mirahezebots.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'wiki.mirahezebots.org' expires in 7 day(s) (Mon 19 Jul 2021 18:23:57 GMT +0000). [18:27:48] Reception123: ^ [18:30:36] PROBLEM - wiki.mirahezebots.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'wiki.mirahezebots.org' expires in 7 day(s) (Mon 19 Jul 2021 18:23:57 GMT +0000). [18:31:03] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.06, 6.99, 6.57 [18:33:03] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.99, 6.55, 6.46 [18:33:18] PROBLEM - graylog2 Disk Space on graylog2 is CRITICAL: DISK CRITICAL - free space: / 42867 MB (5% inode=99%); [18:35:17] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.54, 6.43, 5.61 [18:37:14] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 6.01, 6.17, 5.62 [18:40:36] Will get to that tomorrow, if not please do remind me. [18:40:47] .in 10h Renew mirahezebots.org [18:40:47] Reception123: Okay, will remind at 2021-07-12 - 06:40:46CEST [18:40:51] that should be enough [18:45:10] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.92, 6.63, 6.13 [18:49:07] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.80, 6.36, 6.13 [18:53:24] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.93, 1.54, 1.24 [18:55:25] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 1.40, 1.47, 1.25 [19:05:23] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.25, 6.25, 5.44 [19:07:22] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.44, 5.80, 5.37 [19:29:43] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: SSL OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org - CNAME OK [19:38:40] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [19:48:50] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.33, 7.59, 6.43 [19:49:43] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.20, 6.95, 6.10 [19:50:48] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.21, 7.08, 6.39 [19:51:24] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.58, 7.15, 6.11 [19:51:41] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.95, 6.40, 5.99 [19:52:17] PROBLEM - cloud5 Current Load on cloud5 is CRITICAL: CRITICAL - load average: 35.27, 27.93, 19.77 [19:52:45] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.89, 7.97, 6.80 [19:53:25] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.98, 7.19, 6.24 [19:54:45] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.82, 7.75, 6.86 [19:54:57] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.44, 7.27, 6.44 [19:55:44] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.24, 7.25, 6.49 [19:57:23] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.44, 6.41, 6.14 [20:01:00] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.11, 7.43, 6.94 [20:02:22] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 12.44, 8.90, 7.35 [20:03:26] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.63, 7.14, 6.65 [20:04:30] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 3.41, 4.08, 3.10 [20:05:58] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.46, 7.37, 6.66 [20:06:31] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 1.82, 3.46, 3.00 [20:07:14] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.91, 7.70, 7.28 [20:07:55] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.04, 7.54, 6.78 [20:08:29] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.03, 3.15, 2.95 [20:09:23] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.49, 7.81, 7.26 [20:09:51] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.47, 7.24, 6.77 [20:13:08] PROBLEM - jobrunner4 Disk Space on jobrunner4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [20:13:32] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: SSL OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org - CNAME OK [20:13:48] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.24, 6.48, 6.59 [20:14:13] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.14, 7.63, 7.75 [20:15:11] RECOVERY - jobrunner4 Disk Space on jobrunner4 is OK: DISK OK - free space: / 28098 MB (64% inode=89%); [20:16:09] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 7.87, 8.02, 7.90 [20:18:06] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.53, 7.54, 7.75 [20:20:20] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.33, 7.54, 7.70 [20:21:30] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.46, 7.04, 6.96 [20:21:37] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 6.69, 6.55, 6.80 [20:22:17] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.23, 7.47, 7.66 [20:22:40] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.68, 3.97, 3.34 [20:22:44] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for lcn.zfc.id.lv could not be found [20:23:27] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.00, 6.92, 6.95 [20:24:38] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 1.99, 3.22, 3.14 [20:25:55] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.03, 6.99, 6.82 [20:27:26] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.99, 7.56, 7.17 [20:30:57] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.02, 3.48, 3.28 [20:31:35] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.14, 7.68, 7.32 [20:32:01] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.31, 7.50, 7.09 [20:33:10] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.18, 3.84, 3.42 [20:33:42] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.02, 7.76, 7.39 [20:34:25] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.84, 8.27, 7.79 [20:35:08] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.74, 3.49, 3.35 [20:35:39] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.52, 7.51, 7.33 [20:36:07] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.33, 7.72, 7.35 [20:37:08] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 1.75, 2.87, 3.14 [20:38:23] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.18, 7.70, 7.67 [20:42:21] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.64, 8.07, 7.76 [20:44:19] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.94, 7.19, 7.47 [20:47:39] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 6.29, 6.16, 6.70 [20:50:26] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.87, 7.41, 7.41 [20:50:52] PROBLEM - de.gyaanipedia.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for de.gyaanipedia.com could not be found [20:50:55] PROBLEM - www.bluepageswiki.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.bluepageswiki.org could not be found [20:50:57] PROBLEM - wiki.thehall.xyz - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.thehall.xyz could not be found [20:51:39] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 6.89, 4.31, 3.42 [20:53:37] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.94, 3.55, 3.25 [20:53:39] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.89, 6.98, 6.93 [20:54:30] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.58, 7.37, 7.42 [20:55:35] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.05, 3.27, 3.20 [20:55:38] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.23, 6.40, 6.72 [20:55:59] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.30, 6.23, 6.75 [20:57:37] RECOVERY - de.gyaanipedia.com - reverse DNS on sslhost is OK: SSL OK - de.gyaanipedia.com reverse DNS resolves to cp13.miraheze.org - CNAME OK [20:57:39] RECOVERY - www.bluepageswiki.org - reverse DNS on sslhost is OK: SSL OK - www.bluepageswiki.org reverse DNS resolves to cp13.miraheze.org - CNAME OK [20:57:41] RECOVERY - wiki.thehall.xyz - reverse DNS on sslhost is OK: SSL OK - wiki.thehall.xyz reverse DNS resolves to cp13.miraheze.org - CNAME OK [21:02:23] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 6.29, 6.16, 6.74 [21:03:36] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.29, 7.11, 6.45 [21:06:59] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.65, 6.89, 6.85 [21:07:49] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.45, 7.02, 6.66 [21:09:46] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.62, 6.30, 6.45 [21:11:03] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.49, 6.26, 6.61 [21:16:08] !log deleted oldest graylog index, space critical [21:16:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:17:22] PROBLEM - graylog2 Disk Space on graylog2 is WARNING: DISK WARNING - free space: / 64738 MB (9% inode=99%); [21:17:52] PROBLEM - graylog2 Current Load on graylog2 is WARNING: WARNING - load average: 3.80, 2.97, 2.30 [21:18:40] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: SSL OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org - CNAME OK [21:22:02] RECOVERY - graylog2 Current Load on graylog2 is OK: OK - load average: 2.81, 2.96, 2.45 [21:22:52] PROBLEM - oecumene.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for oecumene.org could not be found [21:22:53] PROBLEM - wiki.triplescripts.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.triplescripts.org could not be found [21:22:54] PROBLEM - munwiki.info - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for munwiki.info could not be found [21:22:55] PROBLEM - vitriol.spoon.army - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for vitriol.spoon.army could not be found [21:23:08] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.91, 6.86, 6.58 [21:26:05] PROBLEM - graylog2 Current Load on graylog2 is CRITICAL: CRITICAL - load average: 5.64, 4.05, 2.98 [21:28:04] PROBLEM - graylog2 Current Load on graylog2 is WARNING: WARNING - load average: 3.41, 3.79, 3.01 [21:29:01] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.82, 6.25, 6.46 [21:29:46] RECOVERY - oecumene.org - reverse DNS on sslhost is OK: SSL OK - oecumene.org reverse DNS resolves to cp13.miraheze.org - NS RECORDS OK [21:29:52] RECOVERY - wiki.triplescripts.org - reverse DNS on sslhost is OK: SSL OK - wiki.triplescripts.org reverse DNS resolves to cp13.miraheze.org - CNAME OK [21:29:53] RECOVERY - munwiki.info - reverse DNS on sslhost is OK: SSL OK - munwiki.info reverse DNS resolves to cp13.miraheze.org - NS RECORDS OK [21:29:54] RECOVERY - vitriol.spoon.army - reverse DNS on sslhost is OK: SSL OK - vitriol.spoon.army reverse DNS resolves to cp13.miraheze.org - CNAME OK [21:32:57] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.44, 6.95, 6.68 [21:34:20] RECOVERY - graylog2 Current Load on graylog2 is OK: OK - load average: 3.04, 3.35, 3.09 [21:34:40] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 148, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 129, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 66, in check_records nameserversans = dns_resolver.query(root_domain, 'NS') File "/usr/lib/python3/ [21:34:41] packages/dns/resolver.py", line 1002, in query raise NXDOMAIN(qnames=qnames_to_try, responses=nxdomain_responses)dns.resolver.NXDOMAIN: None of DNS query names exist: zfc.id.lv., zfc.id.lv. [21:35:07] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.93, 8.18, 7.20 [21:35:37] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.29, 21.90, 19.72 [21:36:38] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.78, 6.80, 6.30 [21:37:08] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.72, 7.22, 6.97 [21:37:34] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 16.70, 20.21, 19.38 [21:38:25] PROBLEM - cp12 Current Load on cp12 is CRITICAL: CRITICAL - load average: 2.33, 1.73, 1.42 [21:38:36] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.93, 6.28, 6.16 [21:40:24] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 1.55, 1.50, 1.36 [21:41:04] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.47, 6.47, 6.73 [21:45:24] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.46, 22.20, 20.26 [21:46:02] PROBLEM - www.winenjoy.net - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.winenjoy.net could not be found [21:46:13] PROBLEM - storytime.jdstroy.cf - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for storytime.jdstroy.cf could not be found [21:46:14] PROBLEM - www.sdiy.info - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.sdiy.info could not be found [21:46:14] PROBLEM - wiki.mussa.id - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.mussa.id could not be found [21:46:20] PROBLEM - piwik.miraheze.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for piwik.miraheze.org could not be found [21:49:25] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 13.84, 18.48, 19.27 [21:51:39] redis seems to be occasionally killed by oom, which definitely isn't a good thing [21:52:09] Voidwalker: is that the worlds biggest understatement [21:52:23] I mean it's not broke too much yet [21:52:30] well it's an instant high priority task, but yes [21:52:52] RECOVERY - www.winenjoy.net - reverse DNS on sslhost is OK: SSL OK - www.winenjoy.net reverse DNS resolves to cp13.miraheze.org - CNAME OK [21:52:58] I guess that's the CW issues [21:53:03] RECOVERY - piwik.miraheze.org - reverse DNS on sslhost is OK: SSL OK - piwik.miraheze.org reverse DNS resolves to cp13.miraheze.org - NS RECORDS OK [21:53:05] Redis socket error [21:53:06] RECOVERY - storytime.jdstroy.cf - reverse DNS on sslhost is OK: SSL OK - storytime.jdstroy.cf reverse DNS resolves to cp13.miraheze.org - CNAME OK [21:53:09] Oom? [21:53:09] 100%, that's how I found it [21:53:10] dmehus: ^ [21:53:12] RECOVERY - www.sdiy.info - reverse DNS on sslhost is OK: SSL OK - www.sdiy.info reverse DNS resolves to cp13.miraheze.org - NS RECORDS OK [21:53:12] RECOVERY - wiki.mussa.id - reverse DNS on sslhost is OK: SSL OK - wiki.mussa.id reverse DNS resolves to cp13.miraheze.org - CNAME OK [21:53:16] Out of memory [21:53:39] Ah. Wishes I could understand this more than what I could say [21:53:53] Something bad [21:54:49] How bad is it? Is it really severe? [21:55:00] PROBLEM - jobrunner4 Current Load on jobrunner4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:55:20] It's forcibly restarting a process that should be running 24/7 with no downtime [21:55:57] Must be a big pain the ass having to get it fixed, right? [21:57:34] In terms of having low memory, wouldn't that cause an outage? [21:57:45] darkmatterman450: it could [21:57:55] RECOVERY - jobrunner4 Current Load on jobrunner4 is OK: OK - load average: 4.27, 4.54, 4.42 [21:58:10] We've seen it with varnish before where things haven't restart cleanly [21:58:31] I've heard this happened to other platforming services, and that's a royal pain to go through the painstakingly trouble to get it back up and running. [21:58:48] Voidwalker: is there an easy solution beyond more memory? [21:58:50] In theory, it's causing 2-5 minutes of downtime to the jobrunner service [21:59:15] we can possibly configure the kernel to not kill redis-server processes as it is (by default) going to be the largest running process on the server [21:59:20] Voidwalker: won't redis impact sessions too? [21:59:21] Voidwalker I wonder what the cause of that might be? [21:59:25] PROBLEM - ping6 on jobrunner4 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds PROBLEM - ping4 on test3 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [21:59:37] darkmatterman450: we don't have enough resources [21:59:52] Most things auto restart fairly well [21:59:57] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 8.32, 6.59, 4.31 [22:00:15] What a shame. And to think you guys were doing so well with other volunteers who suddenly left to pursue other opportunities. [22:00:54] Voidwalker: why jbr4 alerting and is that safe or would it just break something else [22:00:56] PROBLEM - ping6 on ldap2 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [22:00:56] PROBLEM - ping6 on cp12 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [22:01:29] RECOVERY - ping6 on jobrunner4 is OK: PING OK - Packet loss = 0%, RTA = 3.37 ms [22:01:30] RECOVERY - ping4 on test3 is OK: PING OK - Packet loss = 0%, RTA = 1.06 ms [22:01:33] Something ain't right with icinga-miraheze [22:01:45] uhh, I'm not sure what's up with these other alerts, I'd suspect possible network issues in the DC [22:02:23] And it's definitely not the DC I'd be thinking, like maybe Washington DC, or anything. [22:02:23] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.40, 1.88, 1.66 [22:02:34] DC = data center [22:02:35] darkmatterman450: data centre [22:02:45] Voidwalker: might want to keep an eye [22:02:58] That makes a lot more sense. [22:03:01] RECOVERY - ping6 on ldap2 is OK: PING OK - Packet loss = 0%, RTA = 4.42 ms [22:03:01] RECOVERY - ping6 on cp12 is OK: PING OK - Packet loss = 0%, RTA = 78.00 ms [22:04:18] They're all OVH [22:04:24] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 0.71, 1.46, 1.53 [22:04:57] I have got to start knowing all these abbreviations like OVH, which I'm still learning. [22:05:32] OVH is our host [22:06:19] But what does OVH even stand for? Evening Vow Hosting? I'm sure that what I said wouldn't make an inch of sense at all. [22:08:02] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 1.49, 3.25, 3.71 [22:09:53] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: SSL OK - lcn.zfc.id.lv reverse DNS resolves to cp13.miraheze.org - CNAME OK [22:10:04] darkmatterman450: https://en.wikipedia.org/wiki/OVH [22:10:04] [wikipedia] OVH | "OVH, legally OVH Groupe SAS, is a French cloud computing company which offers VPS, dedicated servers and other web services. As of 2016 OVH owned the world's largest data center in surface area. As of 2019, it was the largest hosting provider in Europe, and the third largest in the world based on physical servers. The company was founded in 1999 by the Klaba family and is headquartered in Roubaix, France. OVH is […]" [22:14:06] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.13, 2.89, 3.37 [22:19:39] Voidwalker: I'm seeing alerts again [22:19:45] You want me to pop on? [22:20:36] which alerts? [22:20:46] Voidwalker: any [22:21:02] It looks like NRPE keeps timing out [22:21:16] Network probably needs testing from mon2 to anything [22:21:38] If you want to take a look, go ahead, I'm going to hang around debugging jobrunner3 for a bit [22:22:07] Ack [22:22:17] I'll try and ping mon2 from inside cluster [22:27:55] PROBLEM - wiki.seamly.net - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.seamly.net could not be found [22:28:03] Voidwalker: my ssh is laggy [22:28:22] i think newtork bugged [22:28:26] network [22:29:45] Voidwalker: yeah ping is well slow [22:29:53] rtt min/avg/max/mdev = 0.143/1.394/8.426/2.666 ms [22:30:11] PROBLEM - jobrunner4 Current Load on jobrunner4 is WARNING: WARNING - load average: 4.74, 5.23, 4.83 [22:30:53] where are you connecting to? [22:31:22] Voidwalker: from jobrunner4 -> mon2 was ping [22:31:36] I'm in North England [22:32:01] oh yeah, that connection was slow [22:33:24] Voidwalker: if I can, i'll get an mtr in a minute [22:34:29] RECOVERY - jobrunner4 Current Load on jobrunner4 is OK: OK - load average: 4.92, 5.02, 4.82 [22:34:36] !log generating mtr from jobrunner4->mon2 [22:34:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:34:45] RECOVERY - wiki.seamly.net - reverse DNS on sslhost is OK: SSL OK - wiki.seamly.net reverse DNS resolves to cp13.miraheze.org - CNAME OK [22:34:48] I think this is caused by load [22:35:27] CPU on jobrunner4 got really high for a bit, and memory usage is at 85% on php, which probably isn't good [22:35:43] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.26, 3.98, 3.44 [22:36:29] PROBLEM - jobrunner4 MediaWiki Rendering on jobrunner4 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:37:27] yea importDump.php running at 85% memory usage, definitely not good [22:37:41] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.03, 3.22, 3.22 [22:37:55] 11647 [22:37:57] almost looks like a memory leak tbh [22:38:13] Voidwalker: shall I kill iit? [22:38:26] RECOVERY - jobrunner4 MediaWiki Rendering on jobrunner4 is OK: HTTP OK: HTTP/1.1 200 OK - 19998 bytes in 1.023 second response time [22:38:50] PROBLEM - jobrunner4 Current Load on jobrunner4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [22:39:40] If you want to restart the import and try and figure out a way to load it without eating all the memory, then yeah. Otherwise we can just wait and see what happens [22:40:49] PROBLEM - jobrunner4 Current Load on jobrunner4 is WARNING: WARNING - load average: 5.02, 5.71, 5.29 [22:42:47] RECOVERY - jobrunner4 Current Load on jobrunner4 is OK: OK - load average: 2.80, 4.63, 4.94 [22:43:05] polcompballanarchywiki [22:44:15] !log killed import with PID 11647 [22:44:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:44:32] Voidwalker: look at that wiki's jobs [22:46:26] PROBLEM - lcn.zfc.id.lv - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 148, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 129, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 66, in check_records nameserversans = dns_resolver.query(root_domain, 'NS') File "/usr/lib/python3/ [22:46:26] packages/dns/resolver.py", line 1002, in query raise NXDOMAIN(qnames=qnames_to_try, responses=nxdomain_responses)dns.resolver.NXDOMAIN: None of DNS query names exist: zfc.id.lv., zfc.id.lv. [22:46:44] Voidwalker: I think it's that wiki [22:47:08] Oh, that wiki. Yeah, I approved that wiki request weeks ago. [22:47:29] 418 in header [22:47:35] 4 on showJobs [22:48:26] !log manually unmessing jobrunner [22:48:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:50:50] !log [rhinos@jobrunner4] sudo -u www-data php /srv/mediawiki/w/maintenance/runJobs.php --wiki=polcompballanarchywiki (END - exit=2) [22:50:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:57:17] PROBLEM - jobrunner4 Current Load on jobrunner4 is WARNING: WARNING - load average: 5.86, 4.89, 4.72 [22:59:17] RECOVERY - jobrunner4 Current Load on jobrunner4 is OK: OK - load average: 3.58, 4.52, 4.61 [23:03:36]