[00:02:32] PROBLEM - graylog2 Puppet on graylog2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:07:48] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.98, 3.69, 3.51 [00:11:39] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.31, 3.31, 3.38 [00:19:23] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.08, 3.74, 3.46 [00:21:59] PROBLEM - ping6 on jobchron1 is CRITICAL: PING CRITICAL - Packet loss = 0%, RTA = 1936.19 ms [00:23:13] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.86, 3.58, 3.48 [00:23:58] PROBLEM - ping6 on jobchron1 is WARNING: PING WARNING - Packet loss = 0%, RTA = 153.93 ms [00:25:13] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 8 datacenters are down: 51.38.69.175/cpweb, 54.38.211.199/cpweb, 2001:41d0:801:2000::58af/cpweb, 2001:41d0:800:170b::5/cpweb, 51.222.25.132/cpweb, 167.114.2.161/cpweb, 2607:5300:205:200::1c30/cpweb, 2607:5300:201:3100::1d3/cpweb [00:25:30] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 7 datacenters are down: 51.38.69.175/cpweb, 54.38.211.199/cpweb, 2001:41d0:801:2000::58af/cpweb, 2001:41d0:800:170b::5/cpweb, 51.222.25.132/cpweb, 2607:5300:205:200::1c30/cpweb, 2607:5300:201:3100::1d3/cpweb [00:25:57] RECOVERY - ping6 on jobchron1 is OK: PING OK - Packet loss = 0%, RTA = 7.72 ms [00:26:06] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:27:04] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.84, 3.30, 3.40 [00:27:13] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [00:27:29] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [00:31:06] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:32:33] RECOVERY - graylog2 Puppet on graylog2 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [00:39:14] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/J1jwv [00:39:16] [02miraheze/mw-config] 07Universal-Omega 034724a2c - Set `beta` cluster to c3 / db12, where `test3wiki` is located [00:39:17] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://git.io/vbvb3 [00:39:19] [02mw-config] 07Universal-Omega opened pull request 03#4232: Set `beta` cluster to c3 / db12, where `test3wiki` is located - 13https://git.io/J1jwJ [00:39:21] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 4 datacenters are down: 51.38.69.175/cpweb, 51.222.25.132/cpweb, 2607:5300:205:200::1c30/cpweb, 2607:5300:201:3100::1d3/cpweb [00:40:23] miraheze/mw-config - Universal-Omega the build passed. [00:40:36] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:41:20] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [00:42:46] [02mw-config] 07Universal-Omega edited pull request 03#4232: Set `beta` cluster to c3 / db12, where `test3wiki` is located - 13https://git.io/J1jwJ [00:45:36] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:02:15] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.91, 3.50, 3.25 [01:04:11] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.09, 3.33, 3.22 [01:08:03] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.48, 3.53, 3.33 [01:09:59] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.79, 3.39, 3.32 [01:16:14] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 17.26, 17.97, 23.19 [01:20:14] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 25.42, 20.55, 22.88 [01:22:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.21, 3.69, 3.35 [01:24:14] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.29, 22.34, 23.17 [01:24:27] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.79, 3.51, 3.32 [01:28:19] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.22, 3.33, 3.28 [01:31:24] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 2.77, 4.82, 5.85 [01:32:11] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.51, 3.51, 3.38 [01:32:14] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 12.71, 16.37, 20.23 [01:34:06] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.09, 4.06, 3.59 [01:36:02] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.81, 3.84, 3.57 [01:37:24] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 9.34, 6.18, 5.94 [01:39:24] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 3.97, 5.30, 5.65 [01:47:24] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.44, 4.28, 5.06 [01:47:35] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.11, 3.62, 3.55 [02:01:03] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.86, 3.78, 3.92 [02:03:00] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.31, 3.90, 3.94 [02:04:56] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.43, 3.75, 3.88 [02:08:56] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.88, 3.64, 3.77 [02:09:59] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.20, 5.21, 5.01 [02:11:53] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.26, 5.81, 5.25 [02:13:46] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.42, 5.78, 5.31 [02:17:34] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 2.52, 4.51, 4.95 [02:22:56] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.81, 3.87, 4.00 [02:26:56] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 3.98, 4.05, 4.04 [02:28:56] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.35, 3.79, 3.95 [02:34:56] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.55, 3.91, 3.92 [02:36:56] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.23, 3.63, 3.82 [02:40:56] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.21, 3.69, 3.78 [02:42:56] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.50, 3.59, 3.73 [02:44:25] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.59, 4.79, 4.47 [02:46:18] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 4.97, 4.88, 4.54 [02:46:53] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.79, 19.00, 16.99 [02:50:06] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.44, 5.88, 5.07 [02:50:44] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 30.91, 23.25, 19.03 [02:53:54] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.40, 6.12, 5.34 [02:54:56] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.65, 3.48, 3.55 [02:56:30] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 18.14, 22.10, 20.05 [02:56:56] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.32, 3.46, 3.54 [02:59:35] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.16, 5.41, 5.34 [03:00:22] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 14.53, 19.37, 19.55 [03:05:24] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.43, 4.70, 5.06 [03:06:56] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.92, 3.21, 3.39 [03:09:24] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.55, 5.02, 5.12 [03:13:24] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 4.68, 4.93, 5.04 [03:16:56] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.18, 3.39, 3.42 [03:18:56] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.76, 3.18, 3.34 [03:21:24] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 7.71, 6.21, 5.45 [03:22:14] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 25.41, 21.31, 19.06 [03:23:24] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.39, 5.72, 5.35 [03:24:14] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.09, 21.39, 19.35 [03:26:14] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 14.99, 19.47, 18.94 [03:27:24] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 4.31, 4.86, 5.08 [03:37:59] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.13, 5.40, 5.17 [03:39:53] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.58, 4.70, 4.94 [03:49:24] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.34, 5.21, 5.05 [03:49:48] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.89, 4.02, 3.47 [03:51:24] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.39, 5.11, 5.02 [03:51:44] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.47, 3.86, 3.48 [03:52:14] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.45, 19.24, 18.08 [03:53:24] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.60, 4.65, 4.87 [03:54:14] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 19.18, 18.64, 17.97 [03:55:35] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.54, 3.81, 3.50 [03:57:30] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.83, 3.86, 3.56 [03:59:24] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.91, 5.80, 5.26 [04:01:21] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.06, 3.73, 3.56 [04:01:24] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.37, 5.31, 5.15 [04:03:17] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.16, 3.67, 3.57 [04:03:24] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.83, 4.82, 4.99 [04:07:07] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.45, 3.15, 3.39 [04:10:11] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 11.96, 7.37, 5.89 [04:12:44] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 26.21, 25.12, 21.16 [04:14:40] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 15.07, 21.16, 20.17 [04:16:36] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 13.84, 18.43, 19.27 [04:22:24] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.72, 22.43, 20.62 [04:24:20] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.73, 21.07, 20.29 [04:25:24] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.00, 5.48, 5.93 [04:26:15] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 17.87, 19.35, 19.73 [04:31:24] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 8.39, 5.58, 5.67 [04:32:14] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.52, 20.33, 19.90 [04:33:24] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.29, 4.94, 5.43 [04:34:14] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 16.30, 18.38, 19.21 [04:37:24] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 2.61, 3.90, 4.91 [04:53:30] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.82, 3.11, 2.80 [04:55:26] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.08, 2.71, 2.69 [05:12:45] PROBLEM - wikipedia.yogfront.ooo - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 148, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 129, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 79, in check_records cname = str(dns_resolver.query(hostname, 'CNAME')[0]) File "/usr/lib/ [05:12:45] n3/dist-packages/dns/resolver.py", line 898, in query raise NoNameservers(request=request, errors=errors)dns.resolver.NoNameservers: All nameservers failed to answer the query wikipedia.yogfront.ooo. IN CNAME: Server 1.1.1.1 UDP port 53 answered SERVFAIL [05:19:30] RECOVERY - wikipedia.yogfront.ooo - reverse DNS on sslhost is OK: SSL OK - wikipedia.yogfront.ooo reverse DNS resolves to cp14.miraheze.org - CNAME OK [05:21:24] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.81, 4.74, 4.12 [05:23:24] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 4.87, 4.67, 4.17 [05:34:56] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.40, 3.61, 3.29 [05:38:56] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.01, 3.95, 3.50 [05:40:56] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.93, 3.64, 3.44 [05:42:56] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 1.88, 3.08, 3.27 [05:45:03] PROBLEM - www.winenjoy.net - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'www.winenjoy.net' expires in 7 day(s) (Thu 02 Dec 2021 05:38:37 GMT +0000). [05:51:24] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.92, 5.10, 4.42 [05:53:24] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 4.54, 4.82, 4.40 [06:05:12] PROBLEM - db13 Current Load on db13 is WARNING: WARNING - load average: 7.76, 5.22, 2.81 [06:07:09] PROBLEM - db11 Current Load on db11 is CRITICAL: CRITICAL - load average: 8.52, 6.62, 4.15 [06:07:12] RECOVERY - db13 Current Load on db13 is OK: OK - load average: 6.68, 5.78, 3.32 [06:09:09] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 7.01, 6.83, 4.53 [06:11:09] PROBLEM - db11 Current Load on db11 is CRITICAL: CRITICAL - load average: 8.58, 7.32, 4.97 [06:15:09] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 6.96, 7.42, 5.57 [06:18:25] RECOVERY - db12 Disk Space on db12 is OK: DISK OK - free space: / 50042 MB (11% inode=98%); [06:26:25] PROBLEM - db12 Disk Space on db12 is WARNING: DISK WARNING - free space: / 47961 MB (10% inode=98%); [06:27:09] PROBLEM - db11 Current Load on db11 is CRITICAL: CRITICAL - load average: 8.64, 7.39, 6.38 [06:29:09] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 6.99, 7.38, 6.51 [06:37:09] RECOVERY - db11 Current Load on db11 is OK: OK - load average: 6.05, 6.74, 6.60 [06:39:03] PROBLEM - wiki.cyberfurs.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:39:28] PROBLEM - wiki.homiedb.party - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:45:09] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 6.81, 6.68, 6.60 [06:45:42] PROBLEM - wikipedia.yogfront.ooo - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 148, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 129, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 79, in check_records cname = str(dns_resolver.query(hostname, 'CNAME')[0]) File "/usr/lib/ [06:45:43] n3/dist-packages/dns/resolver.py", line 898, in query raise NoNameservers(request=request, errors=errors)dns.resolver.NoNameservers: All nameservers failed to answer the query wikipedia.yogfront.ooo. IN CNAME: Server 1.1.1.1 UDP port 53 answered SERVFAIL [06:47:09] PROBLEM - db11 Current Load on db11 is CRITICAL: CRITICAL - load average: 8.12, 6.91, 6.67 [06:49:09] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 6.95, 6.75, 6.64 [06:51:09] RECOVERY - db11 Current Load on db11 is OK: OK - load average: 5.60, 6.26, 6.47 [06:52:24] RECOVERY - wikipedia.yogfront.ooo - reverse DNS on sslhost is OK: SSL OK - wikipedia.yogfront.ooo reverse DNS resolves to cp14.miraheze.org - CNAME OK [07:00:44] RECOVERY - wiki.homiedb.party - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.homiedb.party' will expire on Fri 07 Jan 2022 04:28:35 GMT +0000. [07:13:05] RECOVERY - wiki.cyberfurs.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.cyberfurs.org' will expire on Fri 14 Jan 2022 06:38:38 GMT +0000. [07:28:56] PROBLEM - wiki.cyberfurs.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:35:45] RECOVERY - wiki.cyberfurs.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.cyberfurs.org' will expire on Fri 14 Jan 2022 06:38:38 GMT +0000. [07:36:22] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 6.95, 6.91, 6.63 [07:38:19] RECOVERY - db11 Current Load on db11 is OK: OK - load average: 5.52, 6.41, 6.48 [07:42:57] PROBLEM - wiki.homiedb.party - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:44:09] [02mw-config] 07Reception123 closed pull request 03#4232: Set `beta` cluster to c3 / db12, where `test3wiki` is located - 13https://git.io/J1jwJ [07:44:10] [02mw-config] 07Reception123 deleted branch 03Universal-Omega-patch-1 - 13https://git.io/vbvb3 [07:44:12] [02miraheze/mw-config] 07Reception123 deleted branch 03Universal-Omega-patch-1 [07:44:13] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JMe5z [07:44:15] [02miraheze/mw-config] 07Universal-Omega 033efd415 - Set `beta` cluster to c3 / db12, where `test3wiki` is located (#4232) [07:45:19] miraheze/mw-config - Reception123 the build passed. [07:46:24] !log [@mw11] starting deploy of {'config': True} to all [07:46:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:46:52] !log [@mw11] finished deploy of {'config': True} to all - SUCCESS in 27s [07:46:55] !log [@test3] starting deploy of {'config': True} to skip [07:46:56] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [07:46:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:47:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:47:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:47:35] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.05, 5.30, 3.95 [07:48:07] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.33, 5.52, 3.89 [07:49:34] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.27, 4.84, 3.95 [07:49:57] PROBLEM - db11 Current Load on db11 is CRITICAL: CRITICAL - load average: 8.20, 6.96, 6.57 [07:49:57] RECOVERY - wiki.homiedb.party - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.homiedb.party' will expire on Fri 07 Jan 2022 04:28:35 GMT +0000. [07:50:07] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.47, 5.36, 4.03 [07:51:54] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 6.99, 6.77, 6.54 [07:55:48] RECOVERY - db11 Current Load on db11 is OK: OK - load average: 5.48, 6.31, 6.43 [08:04:00] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03revert-4232-Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JMeNA [08:04:01] [02miraheze/mw-config] 07Reception123 035ef363b - Revert "Set `beta` cluster to c3 / db12, where `test3wiki` is located (#4232)" [08:04:03] [02mw-config] 07Reception123 created branch 03revert-4232-Universal-Omega-patch-1 - 13https://git.io/vbvb3 [08:04:04] [02mw-config] 07Reception123 opened pull request 03#4233: Revert "Set `beta` cluster to c3 / db12, where `test3wiki` is located" - 13https://git.io/JMeNx [08:04:08] [02mw-config] 07Reception123 closed pull request 03#4233: Revert "Set `beta` cluster to c3 / db12, where `test3wiki` is located" - 13https://git.io/JMeNx [08:04:10] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JMeNp [08:04:11] [02miraheze/mw-config] 07Reception123 0355953d8 - Revert "Set `beta` cluster to c3 / db12, where `test3wiki` is located (#4232)" (#4233) [08:04:13] [02mw-config] 07Reception123 deleted branch 03revert-4232-Universal-Omega-patch-1 - 13https://git.io/vbvb3 [08:04:14] [02miraheze/mw-config] 07Reception123 deleted branch 03revert-4232-Universal-Omega-patch-1 [08:05:10] miraheze/mw-config - Reception123 the build passed. [08:05:11] !log [@test3] starting deploy of {'config': True} to skip [08:05:12] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [08:05:16] miraheze/mw-config - Reception123 the build passed. [08:05:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:05:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:06:32] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03revert-4233-revert-4232-Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JMeAu [08:06:34] [02miraheze/mw-config] 07Universal-Omega 035e11d96 - Revert "Revert "Set `beta` cluster to c3 / db12, where `test3wiki` is located (#4232)" (#4233)" [08:06:35] [02mw-config] 07Universal-Omega created branch 03revert-4233-revert-4232-Universal-Omega-patch-1 - 13https://git.io/vbvb3 [08:06:38] [02mw-config] 07Universal-Omega opened pull request 03#4234: Revert "Revert "Set `beta` cluster to c3 / db12, where `test3wiki` is located"" - 13https://git.io/JMeA2 [08:07:35] miraheze/mw-config - Universal-Omega the build passed. [08:08:58] [02miraheze/mw-config] 07Reception123 deleted branch 03revert-4233-revert-4232-Universal-Omega-patch-1 [08:09:00] [02mw-config] 07Reception123 closed pull request 03#4234: Revert "Revert "Set `beta` cluster to c3 / db12, where `test3wiki` is located"" - 13https://git.io/JMeA2 [08:09:01] [02mw-config] 07Reception123 deleted branch 03revert-4233-revert-4232-Universal-Omega-patch-1 - 13https://git.io/vbvb3 [08:09:03] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JMexU [08:09:04] [02miraheze/mw-config] 07Universal-Omega 03a57a78e - Revert "Revert "Set `beta` cluster to c3 / db12, where `test3wiki` is located (#4232)" (#4233)" (#4234) [08:10:00] miraheze/mw-config - Reception123 the build passed. [08:11:18] !log [@test3] starting deploy of {'config': True} to skip [08:11:19] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [08:11:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:11:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:11:42] !log sudo -u www-data php /srv/mediawiki/w/maintenance/sql.php /srv/mediawiki/w/extensions/Echo/db_patches/echo_unread_wikis.sql --wiki=test3wiki on test3 [08:11:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:14:25] RECOVERY - db12 Disk Space on db12 is OK: DISK OK - free space: / 52176 MB (11% inode=98%); [08:17:09] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 7.38, 6.53, 6.30 [08:22:55] [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+1/-0/±0] 13https://git.io/JMejw [08:22:57] [02miraheze/dns] 07Reception123 03ca697f8 - add wikipanel.org zone [08:27:10] [02miraheze/ssl] 07Reception123 pushed 033 commits to 03master [+3/-0/±3] 13https://git.io/JMve6 [08:27:11] [02miraheze/ssl] 07Reception123 03ccc0813 - add ngsswiki.com cert [08:27:13] [02miraheze/ssl] 07Reception123 035303185 - add wikipanel.org cert [08:27:14] [02miraheze/ssl] 07Reception123 039eda6f6 - add wiki.controlistes.fr cert [08:29:09] RECOVERY - db11 Current Load on db11 is OK: OK - load average: 5.22, 6.64, 6.67 [08:40:25] PROBLEM - db12 Disk Space on db12 is WARNING: DISK WARNING - free space: / 48502 MB (10% inode=98%); [09:44:01] PROBLEM - db12 APT on db12 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [09:45:58] RECOVERY - db12 APT on db12 is OK: APT OK: 9 packages available for upgrade (0 critical updates). [09:47:41] alerting : [FIRING:1] (!sre MediaWiki Exception Rate yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:52:41] ok : [RESOLVED] (!sre MediaWiki Exception Rate yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:29:22] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 12.42, 7.60, 4.97 [10:31:20] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.33, 6.61, 4.92 [10:52:09] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.65, 19.00, 16.26 [10:54:03] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 15.15, 17.55, 16.06 [12:16:02] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.14, 4.88, 4.11 [12:18:01] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.23, 4.24, 3.98 [12:47:05] PROBLEM - pt.graalmilitary.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'pt.graalmilitary.com' expires in 7 day(s) (Thu 02 Dec 2021 12:41:06 GMT +0000). [12:49:43] PROBLEM - egoish.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'egoish.org' expires in 7 day(s) (Thu 02 Dec 2021 12:44:14 GMT +0000). [13:26:54] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+0/-0/±11] 13https://git.io/JMfLg [13:26:56] [02miraheze/ssl] 07Reception123 035fdcb7c - renew SSL certs [13:29:25] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.77, 18.57, 16.24 [13:30:02] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.76, 4.96, 4.05 [13:30:21] PROBLEM - cp13 Puppet on cp13 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [13:31:15] PROBLEM - mw12 Puppet on mw12 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [13:31:25] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 19.84, 18.73, 16.56 [13:32:01] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.16, 5.42, 4.33 [13:34:01] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.75, 4.91, 4.28 [13:38:13] RECOVERY - www.winenjoy.net - LetsEncrypt on sslhost is OK: OK - Certificate 'www.winenjoy.net' will expire on Tue 22 Feb 2022 12:16:42 GMT +0000. [13:38:17] RECOVERY - popira.info - LetsEncrypt on sslhost is OK: OK - Certificate 'popira.info' will expire on Tue 22 Feb 2022 12:23:55 GMT +0000. [13:38:42] RECOVERY - pwsc.polandballwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'pwsc.polandballwiki.com' will expire on Tue 22 Feb 2022 12:22:18 GMT +0000. [13:38:43] RECOVERY - egoish.org - LetsEncrypt on sslhost is OK: OK - Certificate 'egoish.org' will expire on Tue 22 Feb 2022 12:25:32 GMT +0000. [13:39:10] RECOVERY - wiki.dunmoore.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.dunmoore.net' will expire on Tue 22 Feb 2022 12:20:38 GMT +0000. [13:39:34] RECOVERY - wiki.umbranet.mandelbrotgames.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.umbranet.mandelbrotgames.com' will expire on Tue 22 Feb 2022 12:18:37 GMT +0000. [13:43:05] RECOVERY - pt.graalmilitary.com - LetsEncrypt on sslhost is OK: OK - Certificate 'pt.graalmilitary.com' will expire on Tue 22 Feb 2022 12:22:58 GMT +0000. [13:43:15] RECOVERY - staff.polandballwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'staff.polandballwiki.com' will expire on Tue 22 Feb 2022 12:21:32 GMT +0000. [13:43:16] RECOVERY - wiki.ct777.cf - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.ct777.cf' will expire on Tue 22 Feb 2022 12:24:49 GMT +0000. [13:44:08] RECOVERY - wiki.simorgh.me - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.simorgh.me' will expire on Tue 22 Feb 2022 12:19:43 GMT +0000. [13:44:28] RECOVERY - www.recyclerie-uzege.org - LetsEncrypt on sslhost is OK: OK - Certificate 'www.recyclerie-uzege.org' will expire on Tue 22 Feb 2022 12:17:40 GMT +0000. [13:55:58] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 30.98, 21.68, 17.66 [13:57:53] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 18.45, 19.78, 17.42 [13:58:20] RECOVERY - cp13 Puppet on cp13 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [13:59:19] RECOVERY - mw12 Puppet on mw12 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [14:40:22] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.45, 3.01, 2.81 [14:42:50] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 8.29, 5.83, 4.46 [14:44:12] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.27, 3.14, 2.90 [14:44:45] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.87, 5.35, 4.44 [14:46:41] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.18, 4.76, 4.34 [15:28:41] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.22, 3.49, 3.07 [15:29:37] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.65, 19.62, 18.17 [15:30:36] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.22, 3.29, 3.05 [15:31:31] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 13.34, 17.87, 17.75 [15:35:47] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.11, 5.27, 4.70 [15:36:22] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.73, 3.59, 3.23 [15:38:17] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.00, 3.26, 3.14 [15:43:28] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 4.26, 5.02, 4.88 [15:49:15] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.23, 5.14, 4.94 [15:51:11] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.40, 4.60, 4.77 [15:51:44] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.87, 3.73, 3.39 [15:53:39] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.76, 3.31, 3.27 [15:59:31] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.58, 3.59, 3.39 [16:02:44] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.61, 5.03, 4.62 [16:04:39] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 5.01, 4.74, 4.54 [16:05:31] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.32, 3.11, 3.28 [16:24:02] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 9.00, 6.45, 4.98 [16:24:25] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 26.76, 22.82, 19.07 [16:28:01] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.14, 5.98, 5.17 [16:28:14] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.52, 21.70, 19.39 [16:30:08] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 17.25, 20.11, 19.09 [16:32:01] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.71, 5.74, 5.21 [16:33:58] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.52, 21.72, 19.93 [16:34:02] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.54, 5.54, 5.20 [16:35:53] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 15.74, 19.42, 19.30 [16:36:01] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.23, 4.82, 4.98 [16:40:36] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.62, 3.30, 3.08 [16:42:31] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.41, 3.64, 3.23 [16:44:26] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.29, 3.49, 3.22 [16:50:12] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.32, 3.74, 3.37 [16:54:02] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.31, 3.12, 3.21 [16:59:48] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.75, 3.49, 3.26 [17:01:43] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.38, 3.36, 3.24 [17:06:40] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.74, 5.02, 4.69 [17:08:27] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.83, 3.50, 3.30 [17:08:35] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 2.92, 4.21, 4.44 [17:10:22] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.17, 3.59, 3.35 [17:12:17] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.77, 3.48, 3.33 [17:12:27] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.24, 5.07, 4.71 [17:14:12] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.66, 4.00, 3.54 [17:14:23] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.70, 5.04, 4.73 [17:15:54] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.62, 5.86, 4.87 [17:16:07] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.60, 3.68, 3.47 [17:16:18] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 4.90, 4.95, 4.73 [17:17:51] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.08, 5.16, 4.74 [17:18:02] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.52, 3.37, 3.39 [17:21:54] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.40, 3.40, 3.43 [17:23:08] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JMJU7 [17:23:10] [02miraheze/ManageWiki] 07Universal-Omega 03b5bbb39 - ManageWikiCore: make database names case-insensitive (#317) [17:23:11] [02ManageWiki] 07Universal-Omega closed pull request 03#317: ManageWikiCore: make database names case-insensitive - 13https://git.io/J1hrB [17:23:13] [02ManageWiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://git.io/vpSns [17:23:14] [02miraheze/ManageWiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [17:23:49] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.74, 3.14, 3.33 [17:26:54] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.02, 5.76, 5.19 [17:30:11] miraheze/ManageWiki - Universal-Omega the build passed. [17:30:45] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.30, 5.77, 5.29 [17:32:40] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.33, 5.56, 5.26 [17:34:36] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.98, 4.90, 5.05 [17:38:28] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.45, 5.61, 5.36 [17:40:23] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 10.40, 6.89, 5.82 [17:41:31] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.44, 3.36, 3.36 [17:43:01] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 27.18, 24.62, 21.51 [17:43:28] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.44, 6.70, 5.38 [17:43:31] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.07, 2.85, 3.17 [17:45:27] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.23, 5.81, 5.21 [17:46:50] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.57, 23.79, 21.96 [17:52:02] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.35, 5.45, 5.73 [17:52:34] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.96, 23.25, 22.21 [17:54:29] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.36, 22.08, 21.90 [17:58:02] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 10.02, 7.02, 6.24 [17:58:18] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 27.73, 23.67, 22.48 [18:02:02] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 3.70, 5.49, 5.81 [18:02:07] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 19.45, 22.02, 22.16 [18:06:02] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 7.62, 5.88, 5.81 [18:08:48] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.71, 4.06, 3.42 [18:11:40] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 26.82, 23.01, 22.23 [18:12:02] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.74, 5.91, 5.87 [18:14:02] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.79, 6.32, 6.03 [18:14:33] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.12, 3.69, 3.51 [18:16:02] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.63, 5.94, 5.92 [18:17:25] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 19.40, 23.04, 22.65 [18:18:23] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.39, 3.20, 3.37 [18:19:25] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 28.73, 25.60, 23.62 [18:22:02] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 7.69, 6.41, 6.07 [18:24:09] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.45, 3.49, 3.39 [18:26:04] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.76, 3.49, 3.40 [18:29:54] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.12, 3.13, 3.30 [18:45:25] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 17.20, 21.12, 23.57 [18:49:25] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 28.75, 22.95, 23.53 [18:52:02] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.18, 5.48, 5.95 [18:53:25] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.62, 23.50, 23.70 [18:58:02] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.05, 5.52, 5.72 [19:00:02] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 3.30, 4.61, 5.36 [19:04:01] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.52, 4.24, 5.04 [19:07:25] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 17.53, 17.63, 19.77 [19:17:30] Can’t access wikis but Phab is working [19:18:04] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.66, 21.08, 20.22 [19:18:35] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 7.48, 5.74, 5.09 [19:19:34] Just slow bukkit [19:19:59] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 31.77, 24.94, 21.71 [19:20:24] Works now [19:20:30] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 14.00, 9.02, 6.34 [19:22:47] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.90, 7.44, 5.93 [19:24:47] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.21, 7.31, 6.07 [19:25:31] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.21, 3.77, 3.24 [19:26:17] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.33, 5.85, 5.57 [19:27:31] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.86, 3.91, 3.36 [19:28:12] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 9.55, 6.83, 5.92 [19:28:23] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.25, 7.66, 6.95 [19:28:46] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 10.25, 8.63, 6.83 [19:30:01] PROBLEM - mw12 APT on mw12 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:31:57] RECOVERY - mw12 APT on mw12 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [19:32:03] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 3.66, 5.64, 5.71 [19:33:25] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 15.92, 21.74, 22.66 [19:33:31] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.16, 3.81, 3.46 [19:38:15] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.24, 6.49, 6.78 [19:38:46] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 4.69, 7.03, 7.35 [19:40:02] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 2.19, 3.80, 4.87 [19:41:25] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 18.73, 17.69, 20.29 [19:41:31] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.74, 3.81, 3.67 [19:45:31] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.26, 3.79, 3.67 [19:46:46] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.32, 7.03, 7.10 [19:48:46] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.57, 6.72, 6.98 [19:49:31] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.11, 3.69, 3.68 [19:51:25] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 25.35, 20.88, 20.41 [19:52:02] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.87, 8.12, 7.03 [19:52:46] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 12.98, 9.16, 7.80 [19:54:00] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.05, 7.58, 6.96 [19:54:46] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 5.50, 7.39, 7.30 [19:55:25] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.47, 22.91, 21.44 [19:57:31] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.73, 3.07, 3.39 [19:57:57] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.09, 6.13, 6.51 [19:58:01] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.47, 5.59, 4.95 [19:58:46] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.77, 8.02, 7.49 [20:00:46] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.36, 7.65, 7.42 [20:01:26] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 28.25, 25.62, 23.08 [20:02:02] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 7.14, 6.09, 5.25 [20:03:25]