[00:01:49] PROBLEM - mw101 Current Load on mw101 is CRITICAL: connect to address 2a10:6740::6:107 port 5666: No route to hostconnect to host 2a10:6740::6:107 port 5666: No route to host [00:02:00] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.09, 3.96, 3.77 [00:03:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:03:43] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - dnd.bellinrattin.it All nameservers failed to answer the query. [00:03:48] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.11, 1.86, 0.65 [00:03:57] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.25, 3.64, 3.67 [00:05:54] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.40, 3.60, 3.62 [00:06:45] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.49, 1.53, 1.95 [00:07:37] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.44, 10.62, 9.78 [00:07:42] !log sudo service php7.4-fpm restart on mw* [00:07:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:09:37] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.78, 9.85, 9.59 [00:10:49] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.28, 3.60, 3.10 [00:11:44] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.29, 4.04, 3.39 [00:12:44] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.59, 3.67, 3.19 [00:13:42] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.25, 3.76, 3.37 [00:15:46] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.13, 9.53, 5.45 [00:17:36] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.43, 3.07, 3.17 [00:17:46] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.56, 9.35, 5.91 [00:18:37] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.83, 3.13, 3.16 [00:18:45] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.99, 2.06, 1.90 [00:19:30] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.43, 10.88, 9.40 [00:21:49] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.76, 7.10, 6.27 [00:23:30] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.30, 10.57, 9.66 [00:24:37] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 8.19, 5.15, 3.87 [00:24:45] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.84, 1.99, 1.95 [00:25:23] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.39, 4.12, 3.53 [00:27:38] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.53, 11.05, 8.34 [00:27:49] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 5.11, 6.20, 6.20 [00:28:02] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.05, 1.77, 1.48 [00:28:32] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.70, 9.55, 8.36 [00:29:18] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.80, 3.60, 3.46 [00:29:37] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.35, 11.35, 8.80 [00:30:00] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.30, 1.55, 1.43 [00:30:30] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.45, 9.10, 8.32 [00:30:37] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.11, 3.95, 3.75 [00:31:15] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.84, 2.92, 3.22 [00:31:35] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 13.04, 12.09, 9.39 [00:31:39] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.53, 3.19, 3.97 [00:32:57] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 86, in check_records cname = str(dns_resolver.resolve(hostname, 'CNAME')[0]) File "/usr/li [00:32:58] 3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query dnd.bellinrattin.it. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [00:33:30] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 9.28, 10.03, 9.92 [00:33:33] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.41, 11.57, 9.52 [00:36:37] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.27, 2.96, 3.37 [00:37:38] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.61, 10.79, 10.27 [00:39:39] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.66, 2.65, 3.37 [00:41:27] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.61, 8.91, 9.12 [00:43:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:43:37] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.33, 9.26, 9.83 [00:44:45] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.35, 1.55, 1.69 [00:47:39] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.10, 3.32, 3.41 [00:48:45] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.20, 1.93, 1.81 [00:49:39] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.18, 3.10, 3.31 [00:52:45] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.49, 1.86, 1.83 [00:54:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:59:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:02:11] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - dnd.bellinrattin.it All nameservers failed to answer the query. [01:02:30] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.27, 4.08, 3.54 [01:05:01] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:06:37] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.89, 3.29, 2.88 [01:08:37] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.70, 3.01, 2.83 [01:10:01] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:10:20] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.38, 3.69, 3.68 [01:12:45] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.50, 1.59, 1.70 [01:13:52] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.28, 10.16, 9.23 [01:16:45] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.48, 1.85, 1.78 [01:17:39] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.19, 9.41, 9.18 [01:18:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:20:05] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.31, 3.61, 3.54 [01:22:02] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.11, 3.28, 3.43 [01:22:25] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.12, 10.32, 9.61 [01:22:45] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.62, 1.86, 1.83 [01:24:20] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.82, 9.94, 9.57 [01:25:56] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.66, 3.24, 3.40 [01:27:49] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.60, 7.09, 5.79 [01:29:49] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 5.16, 6.35, 5.68 [01:31:25] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp21.miraheze.org - CNAME OK [01:33:41] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.36, 3.72, 3.51 [01:34:45] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.22, 1.52, 1.66 [01:35:39] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.10, 3.13, 3.32 [01:38:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:45:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:46:40] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.95, 3.50, 3.05 [01:47:39] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.64, 3.35, 3.23 [01:48:39] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.19, 3.08, 2.96 [01:49:39] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.56, 3.50, 3.28 [01:50:46] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.52, 2.03, 1.77 [01:52:45] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.78, 1.92, 1.77 [01:56:40] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.91, 3.82, 3.35 [01:56:45] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.27, 1.63, 1.68 [01:56:53] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.85, 3.57, 3.06 [01:57:10] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 1.78, 2.16, 1.60 [01:58:40] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 6.19, 4.68, 3.72 [01:58:47] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.90, 4.34, 3.40 [01:59:10] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.17, 1.83, 1.55 [02:00:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:01:10] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.65, 1.39, 1.42 [02:04:37] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.27, 3.58, 3.39 [02:04:40] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.45, 3.84, 3.68 [02:06:37] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.78, 4.30, 3.67 [02:06:40] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.16, 4.16, 3.80 [02:08:37] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.86, 4.00, 3.63 [02:10:39] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.38, 3.48, 3.62 [02:14:37] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.63, 3.11, 3.34 [02:16:39] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.77, 2.59, 3.20 [02:17:39] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 1.96, 3.24, 3.92 [02:20:19] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.47, 2.03, 1.75 [02:21:39] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.05, 4.00, 4.00 [02:22:16] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.81, 1.98, 1.78 [02:23:39] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.36, 3.27, 3.73 [02:27:39] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.49, 2.60, 3.38 [02:30:02] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.65, 1.67, 1.70 [02:36:39] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.89, 3.67, 3.04 [02:40:39] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.91, 3.29, 3.03 [02:42:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:45:37] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.75, 1.74, 1.69 [02:47:34] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.31, 1.92, 1.76 [02:52:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:53:24] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.59, 2.00, 1.89 [02:53:46] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.71, 9.82, 8.95 [02:55:46] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.05, 8.95, 8.73 [02:59:01] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:02:04] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - dnd.bellinrattin.it All nameservers failed to answer the query. [03:08:56] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.64, 1.59, 1.70 [03:09:01] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:14:30] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.20, 3.78, 3.23 [03:16:27] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.74, 3.54, 3.20 [03:18:24] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.57, 3.23, 3.13 [03:19:41] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.29, 1.90, 1.77 [03:20:31] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.72, 2.00, 1.50 [03:21:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:23:35] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.54, 1.86, 1.79 [03:23:37] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.09, 10.03, 9.07 [03:24:26] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.22, 1.87, 1.59 [03:25:37] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.43, 9.66, 9.04 [03:26:24] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.06, 1.59, 1.52 [03:26:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:29:25] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.43, 1.61, 1.70 [03:31:18] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 86, in check_records cname = str(dns_resolver.resolve(hostname, 'CNAME')[0]) File "/usr/li [03:31:18] 3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query dnd.bellinrattin.it. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [03:48:53] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.52, 2.05, 1.74 [03:49:39] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.57, 3.61, 3.14 [03:51:39] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.15, 3.20, 3.05 [03:53:47] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.09, 1.68, 1.31 [03:54:45] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.40, 1.80, 1.75 [03:55:44] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.71, 1.70, 1.36 [03:57:39] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.52, 3.70, 3.34 [03:57:42] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.44, 1.65, 1.38 [03:59:39] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.85, 3.09, 3.16 [04:00:32] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp21.miraheze.org - CNAME OK [04:02:45] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.46, 1.59, 1.68 [04:20:45] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.17, 1.94, 1.77 [04:22:45] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.46, 1.77, 1.72 [04:31:56] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.05, 2.48, 1.76 [04:33:54] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 0.69, 1.82, 1.60 [04:34:45] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.39, 1.63, 1.70 [04:35:51] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.88, 1.46, 1.49 [04:50:45] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.35, 2.00, 1.79 [04:52:45] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.58, 1.86, 1.77 [04:56:45] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.31, 1.64, 1.70 [05:00:39] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.24, 3.51, 2.74 [05:02:12] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.94, 3.46, 2.98 [05:02:39] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.66, 3.15, 2.70 [05:02:45] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.47, 2.04, 1.84 [05:06:06] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.81, 2.88, 2.89 [05:15:37] [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+1/-0/±0] 13https://github.com/miraheze/dns/compare/098ef4fff189...e90a8990a795 [05:15:38] [url] Comparing 098ef4fff189...e90a8990a795 · miraheze/dns · GitHub | github.com [05:15:39] [02miraheze/dns] 07Reception123 03e90a899 - add datawiki.online zone [05:20:30] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/9de772a17c1d...515e9a879c82 [05:20:31] [url] Comparing 9de772a17c1d...515e9a879c82 · miraheze/ssl · GitHub | github.com [05:20:32] [02miraheze/ssl] 07Reception123 03515e9a8 - add datawiki.online cert [05:23:49] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 9.02, 6.52, 5.48 [05:25:37] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.47, 10.47, 8.66 [05:25:39] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.57, 3.13, 2.97 [05:27:39] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.39, 2.73, 2.84 [05:29:01] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:29:37] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.48, 10.48, 9.14 [05:29:49] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 5.71, 6.65, 5.96 [05:31:37] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.41, 11.20, 9.56 [05:34:01] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:35:37] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.94, 11.59, 10.10 [05:36:30] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.64, 3.65, 3.17 [05:39:37] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.82, 10.14, 9.92 [05:40:37] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.61, 4.28, 3.06 [05:41:13] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.18, 4.01, 3.10 [05:42:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:43:11] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.35, 3.53, 3.05 [05:45:08] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.06, 3.89, 3.21 [05:46:37] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.01, 3.77, 3.24 [05:47:05] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.61, 3.61, 3.19 [05:47:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:52:37] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.51, 3.18, 3.18 [05:52:58] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 3.30, 3.26, 3.18 [05:56:00] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.38, 3.82, 3.98 [05:56:51] [02puppet] 07Reception123 closed pull request 03#2528: MediaWiki: include svg_* in the clean-tmp-files cron - 13https://github.com/miraheze/puppet/pull/2528 [05:56:52] [url] Page not found · GitHub · GitHub | github.com [05:56:53] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/7d537b8e7441...32460f84687c [05:56:53] [url] Comparing 7d537b8e7441...32460f84687c · miraheze/puppet · GitHub | github.com [05:56:54] [02miraheze/puppet] 07Universal-Omega 0332460f8 - MediaWiki: include svg_* in the clean-tmp-files cron (#2528) [06:01:04] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.30, 4.06, 4.02 [06:01:06] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.74, 3.92, 3.52 [06:01:10] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.18, 3.74, 3.43 [06:03:03] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.12, 3.50, 3.41 [06:03:08] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.62, 3.54, 3.41 [06:06:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:06:56] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.68, 3.29, 3.36 [06:07:01] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.82, 3.49, 3.83 [06:07:02] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.29, 3.19, 3.32 [06:10:51] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.01, 3.43, 3.41 [06:12:47] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.72, 3.24, 3.34 [06:14:49] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.45, 3.74, 3.56 [06:14:58] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.13, 4.18, 3.95 [06:18:43] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.44, 3.99, 3.66 [06:21:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:22:38] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.84, 3.84, 3.72 [06:24:53] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.61, 3.52, 3.81 [06:26:01] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:28:30] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 6.34, 4.20, 3.82 [06:28:52] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.09, 3.35, 3.63 [06:30:27] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.65, 3.57, 3.63 [06:30:51] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.49, 3.11, 3.52 [06:31:01] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:32:50] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.19, 2.90, 3.39 [06:34:22] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.85, 3.01, 3.37 [06:34:24] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 73, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') File "/usr/l [06:34:24] n3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query bellinrattin.it. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [06:44:13] PROBLEM - cloud10 Puppet on cloud10 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [06:45:42] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.03, 3.85, 3.60 [06:47:41] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.40, 3.51, 3.52 [06:53:41] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.31, 2.89, 3.29 [07:03:43] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - dnd.bellinrattin.it All nameservers failed to answer the query. [07:08:39] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 9.72, 7.13, 5.86 [07:09:25] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.20, 9.59, 8.36 [07:09:50] RECOVERY - cloud10 Puppet on cloud10 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:10:39] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.35, 7.27, 6.08 [07:10:57] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 13.02, 9.74, 7.66 [07:11:01] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:11:25] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.55, 10.81, 8.97 [07:14:39] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 9.78, 7.95, 6.59 [07:14:57] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.16, 10.07, 8.37 [07:17:25] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.09, 11.54, 9.96 [07:19:25] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.09, 11.67, 10.19 [07:19:25] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 16.07, 11.75, 9.23 [07:20:39] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 5.92, 7.55, 6.99 [07:21:21] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.22, 11.59, 9.48 [07:21:25] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.78, 11.58, 10.33 [07:26:39] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 4.86, 6.49, 6.75 [07:27:08] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.57, 9.67, 9.28 [07:27:25] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.06, 9.88, 10.08 [07:31:25] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.48, 11.06, 10.51 [07:31:50] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.85, 10.89, 9.35 [07:33:03] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp21.miraheze.org - CNAME OK [07:33:25] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.62, 10.40, 10.33 [07:33:44] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.48, 10.32, 9.33 [07:35:38] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.41, 9.43, 9.12 [07:36:01] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:37:22] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 3.30, 4.23, 2.23 [07:37:25] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.38, 9.43, 10.02 [07:43:22] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 0.84, 1.90, 1.80 [07:45:22] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.29, 1.38, 1.62 [08:18:44] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 6.15, 4.35, 3.06 [08:19:22] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.18, 2.07, 1.39 [08:20:05] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.90, 4.38, 3.03 [08:20:12] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.59, 5.19, 3.72 [08:21:22] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.26, 1.77, 1.36 [08:22:02] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.20, 3.79, 2.97 [08:22:38] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.55, 3.74, 3.12 [08:23:22] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.84, 1.44, 1.29 [08:25:55] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.50, 4.15, 3.27 [08:28:09] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.39, 3.97, 3.73 [08:28:31] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.62, 3.22, 3.11 [08:29:51] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.02, 3.50, 3.20 [08:33:51] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.61, 3.05, 3.07 [08:34:06] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.55, 3.02, 3.36 [08:41:19] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 5.43, 6.86, 6.43 [08:42:01] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.04, 4.02, 3.59 [08:44:03] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.39, 4.15, 3.48 [08:45:13] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 4.65, 6.23, 6.31 [08:46:00] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.08, 3.73, 3.60 [08:46:01] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.02, 3.67, 3.38 [08:47:58] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.76, 3.33, 3.28 [08:49:58] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.18, 4.14, 3.78 [08:51:57] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.45, 3.59, 3.62 [08:59:53] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.61, 2.68, 3.23 [09:05:25] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.91, 10.03, 9.07 [09:07:25] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.92, 9.27, 8.92 [09:07:36] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.69, 3.46, 3.40 [09:07:49] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.32, 3.53, 3.35 [09:07:51] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.94, 3.77, 3.16 [09:09:33] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.40, 3.12, 3.28 [09:09:48] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.47, 3.49, 3.35 [09:11:47] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.34, 3.29, 3.29 [09:11:51] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.02, 3.23, 3.11 [09:35:25] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.84, 10.10, 9.56 [09:37:25] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.02, 8.62, 9.09 [09:41:51] PROBLEM - cloud10 Puppet on cloud10 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [09:59:07] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.53, 2.87, 2.58 [10:01:04] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.15, 3.14, 2.71 [10:02:39] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 9.15, 6.74, 5.70 [10:03:00] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.57, 2.83, 2.65 [10:03:30] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 13.22, 9.79, 7.80 [10:03:34] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 73, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') File "/usr/l [10:03:35] n3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query bellinrattin.it. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [10:05:25] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.43, 10.13, 8.44 [10:06:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:06:39] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.60, 7.35, 6.19 [10:07:25] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.18, 10.30, 8.73 [10:09:25] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.49, 10.08, 8.85 [10:09:30] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.30, 11.04, 9.05 [10:09:50] RECOVERY - cloud10 Puppet on cloud10 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [10:11:41] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.63, 10.67, 8.69 [10:12:39] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 10.69, 8.62, 7.03 [10:13:38] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.38, 10.21, 8.76 [10:15:35] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 10.12, 10.19, 8.93 [10:15:44] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 13.16, 10.54, 8.71 [10:17:44] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.28, 9.81, 8.68 [10:19:30] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 9.51, 10.15, 9.59 [10:20:27] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.85, 4.04, 3.27 [10:21:18] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.04, 4.12, 3.38 [10:22:24] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.11, 3.30, 3.09 [10:22:39] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 5.07, 7.77, 7.58 [10:23:17] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.16, 3.75, 3.33 [10:24:14] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.06, 10.22, 9.71 [10:25:16] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.10, 3.36, 3.23 [10:26:09] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.59, 9.36, 9.46 [10:28:39] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 2.56, 4.71, 6.29 [10:31:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:32:54] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp21.miraheze.org - CNAME OK [11:07:55] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.16, 3.58, 3.17 [11:09:40] [02miraheze/MatomoAnalytics] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/MatomoAnalytics/compare/bd8e9efcaa2a...27a79a75d73d [11:09:41] [url] Comparing bd8e9efcaa2a...27a79a75d73d · miraheze/MatomoAnalytics · GitHub | github.com [11:09:42] [02miraheze/MatomoAnalytics] 07translatewiki 0327a79a7 - Localisation updates from https://translatewiki.net. [11:09:42] [url] Main page - translatewiki.net | translatewiki.net [11:09:43] [02miraheze/ManageWiki] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ManageWiki/compare/bb9d8478869f...0865c92bfe7d [11:09:44] [url] Comparing bb9d8478869f...0865c92bfe7d · miraheze/ManageWiki · GitHub | github.com [11:09:45] [02miraheze/ManageWiki] 07translatewiki 030865c92 - Localisation updates from https://translatewiki.net. [11:09:45] [url] Main page - translatewiki.net | translatewiki.net [11:09:54] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.16, 3.18, 3.08 [11:13:55] miraheze/MatomoAnalytics - translatewiki the build passed. [11:16:05] miraheze/ManageWiki - translatewiki the build passed. [11:33:30] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - dnd.bellinrattin.it All nameservers failed to answer the query. [11:44:35] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.56, 3.69, 3.24 [11:50:32] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.80, 3.91, 3.41 [11:52:32] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.06, 3.51, 3.32 [11:54:31] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.20, 3.71, 3.41 [11:56:30] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.32, 3.38, 3.32 [12:13:01] @Orduin Would you be able to move MirahezeBots SAL logging from #miraheze-bots to #fossbots? [12:32:09] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 73, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') File "/usr/l [12:32:09] n3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query bellinrattin.it. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [12:36:17] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.88, 3.57, 2.92 [12:37:09] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.65, 3.53, 2.97 [12:39:09] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.50, 3.38, 2.98 [12:40:11] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.47, 3.70, 3.09 [12:42:09] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.83, 3.67, 3.15 [12:45:06] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.90, 5.06, 3.83 [12:46:03] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.53, 3.17, 3.07 [12:49:04] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.47, 3.79, 3.60 [12:50:19] @Orduin now it quits every time i try to log somthing [12:50:30] (Void-bot) [12:57:00] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.94, 3.23, 3.40 [13:01:29] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp20.miraheze.org - CNAME OK [13:01:36] alerting : [FIRING:1] (!sre High Job Queue Backlog yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:06:36] ok : [RESOLVED] (!sre High Job Queue Backlog yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:07:25] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.51, 10.27, 9.51 [13:09:25] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.91, 9.11, 9.16 [13:10:53] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.39, 4.02, 3.56 [13:11:08] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.05, 3.81, 3.04 [13:12:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:12:52] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.87, 3.74, 3.52 [13:13:05] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.99, 3.45, 3.00 [13:13:57] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.09, 3.51, 3.20 [13:14:52] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.31, 4.20, 3.72 [13:15:01] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.99, 3.16, 2.94 [13:17:57] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.78, 3.38, 3.22 [13:18:50] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.25, 3.82, 3.75 [13:20:49] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.86, 4.11, 3.85 [13:22:48] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.76, 3.55, 3.67 [13:26:39] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.72, 6.77, 5.35 [13:26:46] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.25, 3.42, 3.54 [13:27:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:28:39] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 6.43, 6.47, 5.41 [13:29:57] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.69, 4.02, 3.30 [13:30:34] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.64, 3.55, 3.09 [13:31:57] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.70, 3.99, 3.39 [13:32:31] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.44, 3.17, 3.01 [13:32:43] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.74, 3.92, 3.76 [13:33:57] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.51, 3.80, 3.36 [13:35:57] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.96, 3.43, 3.28 [13:37:57] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.94, 3.15, 3.18 [13:42:02] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.15, 1.44, 1.02 [13:44:00] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.95, 1.27, 1.01 [13:51:55] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.42, 3.23, 3.03 [13:52:34] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.68, 3.10, 3.35 [13:53:51] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.92, 3.03, 2.97 [13:58:25] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.57, 10.29, 9.44 [14:00:30] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.21, 3.94, 3.55 [14:02:14] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.70, 9.76, 9.44 [14:06:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:08:26] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.85, 3.43, 3.50 [14:10:25] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.21, 3.07, 3.36 [14:10:49] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.30, 10.63, 9.94 [14:11:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:11:34] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.61, 3.13, 2.99 [14:12:44] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.66, 10.19, 9.86 [14:13:31] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.68, 2.81, 2.88 [14:16:22] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.86, 4.38, 3.77 [14:16:33] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.33, 10.29, 10.02 [14:17:01] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:20:20] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.30, 3.70, 3.64 [14:20:22] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.99, 10.10, 10.00 [14:24:11] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.86, 10.26, 10.10 [14:24:18] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.21, 2.88, 3.32 [14:32:01] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:34:44] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.13, 3.50, 3.18 [14:35:38] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.67, 11.51, 10.76 [14:36:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:36:51] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.13, 3.49, 3.12 [14:37:11] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 3.67, 4.10, 3.65 [14:37:33] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.81, 10.96, 10.66 [14:38:38] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.23, 3.50, 3.26 [14:38:48] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.24, 3.29, 3.08 [14:41:25] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.84, 10.21, 9.27 [14:41:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:42:38] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.87, 3.72, 3.31 [14:43:22] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.55, 9.98, 9.33 [14:44:34] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.07, 3.97, 3.46 [14:45:25] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.26, 9.22, 10.03 [14:46:31] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.02, 3.71, 3.43 [14:48:25] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.20, 3.95, 3.62 [14:48:28] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.64, 3.37, 3.35 [14:50:22] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.45, 3.34, 3.43 [14:53:04] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.87, 4.00, 3.97 [14:57:02] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.87, 3.85, 3.85 [14:57:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:59:02] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.34, 3.49, 3.71 [15:00:09] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.83, 2.80, 3.21 [15:01:31] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.25, 9.95, 8.98 [15:02:09] [02dns] 07MacFan4000 closed pull request 03#270: add datawiki.online zone - 13https://github.com/miraheze/dns/pull/270 [15:02:09] [url] Page not found · GitHub · GitHub | github.com [15:03:00] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.58, 3.78, 3.76 [15:03:30] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.72, 10.14, 9.17 [15:04:14] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.53, 10.58, 9.80 [15:04:59] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.87, 3.40, 3.62 [15:05:41] [02miraheze/ssl] 07MacFan4000 pushed 031 commit to 03MacFan4000-patch-1 [+1/-0/±0] 13https://github.com/miraheze/ssl/commit/7314113339a3 [15:05:43] [02miraheze/ssl] 07MacFan4000 037314113 - mhbots -> fossbots [15:05:44] [02ssl] 07MacFan4000 created branch 03MacFan4000-patch-1 - 13https://github.com/miraheze/ssl [15:05:45] [url] Page not found · GitHub · GitHub | github.com [15:05:46] [02ssl] 07MacFan4000 opened pull request 03#512: mhbots -> fossbots - 13https://github.com/miraheze/ssl/pull/512 [15:05:46] [url] Page not found · GitHub · GitHub | github.com [15:06:26] [02miraheze/ssl] 07MacFan4000 pushed 031 commit to 03MacFan4000-patch-1 [+0/-1/±0] 13https://github.com/miraheze/ssl/compare/7314113339a3...52c1a5638f70 [15:06:26] [url] Comparing 7314113339a3...52c1a5638f70 · miraheze/ssl · GitHub | github.com [15:06:27] [02miraheze/ssl] 07MacFan4000 0352c1a56 - Delete wiki.mirahezebots.org.crt [15:06:29] [02ssl] 07MacFan4000 synchronize pull request 03#512: mhbots -> fossbots - 13https://github.com/miraheze/ssl/pull/512 [15:06:29] [url] Page not found · GitHub · GitHub | github.com [15:07:10] [02miraheze/ssl] 07MacFan4000 pushed 031 commit to 03MacFan4000-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/52c1a5638f70...4616b05da88a [15:07:11] [url] Comparing 52c1a5638f70...4616b05da88a · miraheze/ssl · GitHub | github.com [15:07:11] [02miraheze/ssl] 07MacFan4000 034616b05 - Update certs.yaml [15:07:13] [02ssl] 07MacFan4000 synchronize pull request 03#512: mhbots -> fossbots - 13https://github.com/miraheze/ssl/pull/512 [15:07:13] [url] Page not found · GitHub · GitHub | github.com [15:07:30] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.06, 9.61, 9.23 [15:08:03] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.50, 11.13, 10.25 [15:08:34] [02ssl] 07MacFan4000 closed pull request 03#512: mhbots -> fossbots - 13https://github.com/miraheze/ssl/pull/512 [15:08:35] ... [15:08:36] [02miraheze/ssl] 07MacFan4000 pushed 031 commit to 03master [+1/-1/±1] 13https://github.com/miraheze/ssl/compare/515e9a879c82...1488a9978486 [15:08:36] [url] Comparing 515e9a879c82...1488a9978486 · miraheze/ssl · GitHub | github.com [15:08:37] [02miraheze/ssl] 07MacFan4000 031488a99 - mhbots -> fossbots (#512) [15:08:39] [02ssl] 07MacFan4000 deleted branch 03MacFan4000-patch-1 - 13https://github.com/miraheze/ssl [15:08:39] [url] Page not found · GitHub · GitHub | github.com [15:08:40] [02miraheze/ssl] 07MacFan4000 deleted branch 03MacFan4000-patch-1 [15:08:57] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.28, 3.04, 3.40 [15:09:08] [02miraheze/ssl] 07MacFan4000 pushed 031 commit to 03master [+1/-1/±0] 13https://github.com/miraheze/ssl/compare/1488a9978486...c9f85c9fde6e [15:09:09] [url] Comparing 1488a9978486...c9f85c9fde6e · miraheze/ssl · GitHub | github.com [15:09:10] [02miraheze/ssl] 07MacFan4000 03c9f85c9 - Rename wiki.fossbots.org to wiki.fossbots.org.crt [15:14:48] PROBLEM - mw121 Puppet on mw121 is CRITICAL: CRITICAL: Puppet has 3 failures. Last run 3 minutes ago with 3 failures. Failed resources (up to 3 shown): File[/etc/ssl/localcerts/wiki.fossbots.org.crt],File[/etc/ssl/localcerts/wiki.mirahezebots.org.crt],File[/etc/ssl/private/wiki.mirahezebots.org.key] [15:15:12] PROBLEM - mw102 Puppet on mw102 is CRITICAL: CRITICAL: Puppet has 3 failures. Last run 3 minutes ago with 3 failures. Failed resources (up to 3 shown): File[/etc/ssl/localcerts/wiki.fossbots.org.crt],File[/etc/ssl/localcerts/wiki.mirahezebots.org.crt],File[/etc/ssl/private/wiki.mirahezebots.org.key] [15:15:15] that'll resolve it self soon enough [15:22:55] PROBLEM - mw112 Puppet on mw112 is CRITICAL: CRITICAL: Puppet has 3 failures. Last run 3 minutes ago with 3 failures. Failed resources (up to 3 shown): File[/etc/ssl/localcerts/wiki.fossbots.org.crt],File[/etc/ssl/localcerts/wiki.mirahezebots.org.crt],File[/etc/ssl/private/wiki.mirahezebots.org.key] [15:23:25] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 10.11, 10.07, 10.18 [15:23:37] PROBLEM - cp31 Puppet on cp31 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 3 minutes ago with 2 failures. Failed resources (up to 3 shown): File[/etc/ssl/localcerts/wiki.mirahezebots.org.crt],File[/etc/ssl/private/wiki.mirahezebots.org.key] [15:25:26] PROBLEM - cp30 Puppet on cp30 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): File[/etc/ssl/localcerts/wiki.mirahezebots.org.crt],File[/etc/ssl/private/wiki.mirahezebots.org.key] [15:25:40] PROBLEM - mwtask111 Puppet on mwtask111 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): File[/etc/ssl/localcerts/wiki.mirahezebots.org.crt],File[/etc/ssl/private/wiki.mirahezebots.org.key] [15:27:25] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.72, 10.72, 10.41 [15:28:44] PROBLEM - test101 Puppet on test101 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 3 minutes ago with 2 failures. Failed resources (up to 3 shown): File[/etc/ssl/localcerts/wiki.mirahezebots.org.crt],File[/etc/ssl/private/wiki.mirahezebots.org.key] [15:28:45] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.14, 3.50, 3.39 [15:31:12] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.21, 2.96, 3.20 [15:31:35] PROBLEM - cp20 Puppet on cp20 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 5 minutes ago with 2 failures. Failed resources (up to 3 shown): File[/etc/ssl/localcerts/wiki.mirahezebots.org.crt],File[/etc/ssl/private/wiki.mirahezebots.org.key] [15:32:24] PROBLEM - cp21 Puppet on cp21 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 6 minutes ago with 2 failures. Failed resources (up to 3 shown): File[/etc/ssl/localcerts/wiki.mirahezebots.org.crt],File[/etc/ssl/private/wiki.mirahezebots.org.key] [15:33:18] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.69, 10.60, 9.76 [15:33:18] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.05, 6.66, 5.96 [15:33:34] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.57, 9.75, 10.10 [15:33:55] PROBLEM - mw101 Puppet on mw101 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): File[/etc/ssl/localcerts/wiki.mirahezebots.org.crt],File[/etc/ssl/private/wiki.mirahezebots.org.key] [15:35:15] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.68, 10.03, 9.66 [15:35:18] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 9.85, 7.90, 6.50 [15:37:02] PROBLEM - mw122 Puppet on mw122 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): File[/etc/ssl/localcerts/wiki.mirahezebots.org.crt],File[/etc/ssl/private/wiki.mirahezebots.org.key] [15:37:18] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.33, 7.86, 6.68 [15:37:29] PROBLEM - mw111 Puppet on mw111 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): File[/etc/ssl/localcerts/wiki.mirahezebots.org.crt],File[/etc/ssl/private/wiki.mirahezebots.org.key] [15:39:05] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 3.82, 4.19, 3.73 [15:40:20] MacFan4000: ^ [15:40:51] as i said it will resolve itself [15:40:55] next puppet run [15:41:18] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 5.66, 6.79, 6.52 [15:41:22] just down to timing between when private and when public keys are deployed [15:43:03] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.54, 3.85, 3.68 [15:43:19] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.18, 10.08, 9.95 [15:45:15] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.66, 9.38, 9.70 [15:50:04] MacFan4000: cool [15:52:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:53:36] PROBLEM - cloud11 Puppet on cloud11 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [15:54:55] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.65, 4.26, 3.95 [15:54:59] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.72, 10.81, 10.23 [15:56:54] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.86, 3.86, 3.84 [15:59:01] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [16:00:51] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.21, 4.28, 4.02 [16:02:44] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.31, 9.96, 10.07 [16:04:41] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.81, 3.55, 3.30 [16:04:46] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.54, 3.35, 3.16 [16:04:48] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.56, 3.74, 3.88 [16:06:40] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.60, 3.23, 3.21 [16:06:44] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.97, 3.10, 3.08 [16:06:44] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.48, 10.90, 10.43 [16:08:09] [02miraheze/ssl] 07MacFan4000 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/c9f85c9fde6e...b52b2d6c28c4 [16:08:10] [url] Comparing c9f85c9fde6e...b52b2d6c28c4 · miraheze/ssl · GitHub | github.com [16:08:11] [02miraheze/ssl] 07MacFan4000 03b52b2d6 - Update redirects.yaml [16:08:46] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.64, 3.93, 3.90 [16:10:35] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.86, 3.71, 3.42 [16:10:40] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.33, 3.65, 3.29 [16:10:44] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.06, 11.18, 10.63 [16:11:19] RECOVERY - mw121 Puppet on mw121 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [16:12:33] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.10, 4.13, 3.60 [16:12:38] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.80, 3.62, 3.32 [16:12:44] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.30, 10.32, 10.40 [16:13:50] RECOVERY - mw102 Puppet on mw102 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:14:31] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.30, 3.91, 3.59 [16:14:37] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.93, 3.07, 3.16 [16:16:44] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.61, 9.69, 10.14 [16:19:01] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [16:20:38] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.10, 3.43, 3.79 [16:21:18] RECOVERY - mw112 Puppet on mw112 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:21:22] RECOVERY - cloud11 Puppet on cloud11 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:21:37] RECOVERY - cp31 Puppet on cp31 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:22:28] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.73, 3.08, 3.34 [16:23:09] [02miraheze/MatomoAnalytics] 07dependabot[bot] pushed 031 commit to 03dependabot/npm_and_yarn/stylelint-config-wikimedia-0.13.0 [+0/-0/±2] 13https://github.com/miraheze/MatomoAnalytics/commit/b726251a545c [16:23:11] [02miraheze/MatomoAnalytics] 07dependabot[bot] 03b726251 - Bump stylelint-config-wikimedia from 0.12.2 to 0.13.0 [16:23:12] [02MatomoAnalytics] 07dependabot[bot] created branch 03dependabot/npm_and_yarn/stylelint-config-wikimedia-0.13.0 - 13https://github.com/miraheze/MatomoAnalytics [16:23:13] [url] Page not found · GitHub · GitHub | github.com [16:23:14] [02MatomoAnalytics] 07dependabot[bot] opened pull request 03#76: Bump stylelint-config-wikimedia from 0.12.2 to 0.13.0 - 13https://github.com/miraheze/MatomoAnalytics/pull/76 [16:23:14] [url] Page not found · GitHub · GitHub | github.com [16:23:15] [02MatomoAnalytics] 07dependabot[bot] labeled pull request 03#76: Bump stylelint-config-wikimedia from 0.12.2 to 0.13.0 - 13https://github.com/miraheze/MatomoAnalytics/pull/76 [16:23:16] [url] Page not found · GitHub · GitHub | github.com [16:23:17] [02MatomoAnalytics] 07dependabot[bot] labeled pull request 03#76: Bump stylelint-config-wikimedia from 0.12.2 to 0.13.0 - 13https://github.com/miraheze/MatomoAnalytics/pull/76 [16:23:17] [url] Page not found · GitHub · GitHub | github.com [16:24:20] RECOVERY - mwtask111 Puppet on mwtask111 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:24:32] RECOVERY - cp30 Puppet on cp30 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:24:36] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.96, 4.09, 3.89 [16:25:56] RECOVERY - test101 Puppet on test101 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:26:01] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [16:26:12] RECOVERY - cp20 Puppet on cp20 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:26:18] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 1.82, 2.02, 1.38 [16:26:29] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.16, 3.68, 3.52 [16:27:01] RECOVERY - cp21 Puppet on cp21 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [16:27:51] miraheze/MatomoAnalytics - dependabot[bot] the build passed. [16:28:18] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.52, 1.89, 1.42 [16:30:40] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.80, 1.41, 1.30 [16:30:43] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.82, 3.48, 3.72 [16:30:50] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.05, 3.53, 3.52 [16:31:06] RECOVERY - test101 APT on test101 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [16:31:25] RECOVERY - mw101 Puppet on mw101 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [16:31:40] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.68, 0.38, 0.12 [16:33:38] PROBLEM - test101 Check Gluster Clients on test101 is CRITICAL: PROCS CRITICAL: 0 processes with args '/usr/sbin/glusterfs' [16:34:07] CosmicAlpha: ^ [16:34:15] That's an issue I've not seen in a fair bit [16:34:19] Puppet might remount [16:34:23] was me. [16:34:34] RECOVERY - mw122 Puppet on mw122 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [16:34:41] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.42, 4.65, 3.97 [16:34:42] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.18, 4.50, 4.05 [16:34:43] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.04, 3.81, 3.31 [16:34:53] !log upgrade and reboot mwtask111 and test101 [16:35:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:35:29] !log mount /mnt/mediawiki-static on test101 [16:35:32] RhinosF1: ^ [16:35:33] RECOVERY - mwtask111 APT on mwtask111 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [16:35:34] RECOVERY - test101 Check Gluster Clients on test101 is OK: PROCS OK: 1 process with args '/usr/sbin/glusterfs' [16:35:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:36:24] CosmicAlpha: ah! [16:36:29] Makes sense [16:36:34] Gluster hasn't failed in a while [16:37:00] RECOVERY - mw111 Puppet on mw111 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:37:23] PROBLEM - mwtask111 Check Gluster Clients on mwtask111 is CRITICAL: PROCS CRITICAL: 0 processes with args '/usr/sbin/glusterfs' [16:37:46] !log mount /mnt/mediawiki-static on mwtask111 [16:37:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:38:41] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.07, 3.74, 3.41 [16:39:21] RECOVERY - mwtask111 Check Gluster Clients on mwtask111 is OK: PROCS OK: 1 process with args '/usr/sbin/glusterfs' [16:40:18] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.41, 10.10, 9.46 [16:40:40] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.01, 3.71, 3.44 [16:42:18] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.60, 11.19, 9.95 [16:42:18] RECOVERY - mw111 APT on mw111 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [16:42:21] RECOVERY - mw121 APT on mw121 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [16:42:43] RECOVERY - mw112 APT on mw112 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [16:42:49] RECOVERY - mw122 APT on mw122 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [16:43:03] RECOVERY - mw102 APT on mw102 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [16:43:15] !log upgrade mw* [16:43:21] !log reboot mw101 [16:43:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:43:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:44:18] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.33, 9.46, 9.45 [16:44:18] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.92, 3.71, 3.85 [16:44:39] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.95, 3.60, 3.45 [16:44:51] RECOVERY - mw101 APT on mw101 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [16:44:56] are we aware that there are 503s? [16:46:01] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [16:46:13] Ugochimobi: Could've been me for less then 10 seconds, if that. [16:46:38] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.66, 3.21, 3.32 [16:46:42] !log reboot mw102 [16:46:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:47:00] it's just coming and going :) [16:48:09] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.84, 3.80, 3.83 [16:49:29] !log reboot mw111 [16:49:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:51:00] !log reboot mw112 [16:51:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:52:01] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.99, 3.74, 3.82 [16:53:37] !log reboot mw121 [16:54:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [16:56:33] !log reboot mw121 [16:56:37] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.79, 3.29, 3.84 [16:56:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:56:57] !log reboot mw122 [16:57:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:58:36] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.75, 3.79, 3.93 [16:59:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [17:00:31] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.26, 3.48, 3.33 [17:02:30] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.95, 3.62, 3.40 [17:04:29] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.43, 3.21, 3.28 [17:05:29] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.99, 3.84, 3.74 [17:05:48] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/7f2f97a7c760 [17:05:50] [02miraheze/mw-config] 07Universal-Omega 037f2f97a - Update LocalSettings.php [17:05:51] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/mw-config [17:05:52] [url] Page not found · GitHub · GitHub | github.com [17:06:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [17:07:46] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/7f2f97a7c760...81486a502593 [17:07:47] [url] Comparing 7f2f97a7c760...81486a502593 · miraheze/mw-config · GitHub | github.com [17:07:48] [02miraheze/mw-config] 07Universal-Omega 0381486a5 - Update ManageWikiSettings.php [17:08:24] [02mw-config] 07Universal-Omega opened pull request 03#4663: T9181: Add `$wgNamespacePreloadDoExpansion` to ManageWikiSettings - 13https://github.com/miraheze/mw-config/pull/4663 [17:08:25] ... [17:08:27] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.08, 3.61, 3.47 [17:09:20] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.42, 3.45, 3.64 [17:09:47] miraheze/mw-config - Universal-Omega the build passed. [17:10:12] [02mw-config] 07Universal-Omega closed pull request 03#4663: T9181: Add `$wgNamespacePreloadDoExpansion` to ManageWikiSettings - 13https://github.com/miraheze/mw-config/pull/4663 [17:10:12] ... [17:10:13] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/mw-config/compare/4546c6400bbe...20c28ee3ea50 [17:10:14] [url] Comparing 4546c6400bbe...20c28ee3ea50 · miraheze/mw-config · GitHub | github.com [17:10:15] [02miraheze/mw-config] 07Universal-Omega 0320c28ee - T9181: Add `$wgNamespacePreloadDoExpansion` to ManageWikiSettings (#4663) [17:10:16] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [17:10:18] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/mw-config [17:10:18] [url] Page not found · GitHub · GitHub | github.com [17:10:26] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.75, 3.35, 3.39 [17:11:16] miraheze/mw-config - Universal-Omega the build passed. [17:11:56] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 6.84, 6.34, 5.64 [17:12:21] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True} to all [17:12:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:12:38] !log [universalomega@mwtask111] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 17s [17:12:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:13:55] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 5.38, 6.10, 5.64 [17:15:08] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.62, 2.72, 3.29 [17:16:32] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.63, 3.55, 3.98 [17:18:37] !log [@test101] starting deploy of {'config': True} to all [17:18:38] !log [@test101] finished deploy of {'config': True} to all - SUCCESS in 0s [17:18:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:18:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:22:30] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.17, 3.95, 3.96 [17:22:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.93, 10.20, 8.33 [17:23:05] [02miraheze/ManageWiki] 07dependabot[bot] pushed 031 commit to 03dependabot/npm_and_yarn/stylelint-config-wikimedia-0.13.0 [+0/-0/±2] 13https://github.com/miraheze/ManageWiki/commit/678543ccd8d9 [17:23:07] [02miraheze/ManageWiki] 07dependabot[bot] 03678543c - Bump stylelint-config-wikimedia from 0.12.2 to 0.13.0 [17:23:08] [02ManageWiki] 07dependabot[bot] opened pull request 03#357: Bump stylelint-config-wikimedia from 0.12.2 to 0.13.0 - 13https://github.com/miraheze/ManageWiki/pull/357 [17:23:09] [url] Page not found · GitHub · GitHub | github.com [17:23:10] [02ManageWiki] 07dependabot[bot] created branch 03dependabot/npm_and_yarn/stylelint-config-wikimedia-0.13.0 - 13https://github.com/miraheze/ManageWiki [17:23:10] [url] Page not found · GitHub · GitHub | github.com [17:23:11] [02ManageWiki] 07dependabot[bot] labeled pull request 03#357: Bump stylelint-config-wikimedia from 0.12.2 to 0.13.0 - 13https://github.com/miraheze/ManageWiki/pull/357 [17:23:12] [url] Page not found · GitHub · GitHub | github.com [17:23:13] [02ManageWiki] 07dependabot[bot] labeled pull request 03#357: Bump stylelint-config-wikimedia from 0.12.2 to 0.13.0 - 13https://github.com/miraheze/ManageWiki/pull/357 [17:23:13] [url] Page not found · GitHub · GitHub | github.com [17:25:56] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.82, 3.83, 3.49 [17:27:51] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.69, 3.63, 3.45 [17:27:57] miraheze/ManageWiki - dependabot[bot] the build passed. [17:28:29] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.34, 3.80, 3.90 [17:29:46] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.39, 3.26, 3.35 [17:30:28] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.36, 4.39, 4.08 [17:30:42] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.63, 9.96, 9.12 [17:32:14] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.60, 3.69, 3.43 [17:34:13] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.33, 3.36, 3.35 [17:35:59] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 73, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') File "/usr/l [17:35:59] n3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query bellinrattin.it. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [17:36:42] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.24, 11.33, 9.99 [17:38:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.73, 11.09, 10.07 [17:42:25] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.37, 3.74, 3.98 [17:42:42] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.20, 9.45, 9.67 [17:46:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [17:47:08] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.78, 1.51, 1.18 [17:49:08] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.62, 1.53, 1.23 [17:52:23] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.72, 3.45, 3.63 [17:54:22] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.70, 3.49, 3.62 [17:56:22] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.51, 3.70, 3.67 [17:57:41] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.23, 1.78, 1.31 [17:57:45] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.75, 3.53, 3.28 [17:58:21] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.42, 3.50, 3.59 [17:59:36] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.88, 1.46, 1.25 [17:59:40] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.78, 4.04, 3.48 [18:00:21] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.26, 4.09, 3.79 [18:01:36] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.69, 4.00, 3.55 [18:03:31] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 3.94, 4.05, 3.62 [18:05:27] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.44, 3.42, 3.43 [18:05:53] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - dnd.bellinrattin.it All nameservers failed to answer the query. [18:07:22] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.34, 3.20, 3.36 [18:10:19] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.11, 3.36, 3.74 [18:12:18] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.21, 3.54, 3.75 [18:12:57] @Orduin please let me know when you can deal with Void0bot SAL logging [18:13:08] Void-bot* [18:14:01] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [18:16:17] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.40, 3.37, 3.65 [18:16:43] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.25, 10.07, 9.63 [18:18:42] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.15, 10.61, 9.87 [18:18:55] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.58, 1.96, 1.27 [18:20:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.65, 10.80, 10.02 [18:22:19] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.12, 2.89, 3.38 [18:22:42] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.61, 10.12, 9.88 [18:24:52] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.50, 1.85, 1.48 [18:25:44] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.08, 1.51, 1.15 [18:26:17] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.94, 3.61, 3.59 [18:27:28] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.61, 10.62, 9.43 [18:27:43] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.96, 1.27, 1.10 [18:28:51] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.41, 1.61, 1.46 [18:30:07] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 13.37, 11.03, 9.73 [18:30:16] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.21, 3.20, 3.40 [18:30:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.59, 10.72, 10.17 [18:31:48] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.68, 11.41, 9.73 [18:32:54] PROBLEM - cp20 Stunnel HTTP for mw101 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:33:30] PROBLEM - cp30 Stunnel HTTP for mw101 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:33:45] PROBLEM - cp31 Stunnel HTTP for mw101 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:33:51] PROBLEM - cp21 Stunnel HTTP for mw101 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:33:56] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.25, 6.78, 5.70 [18:34:35] PROBLEM - mw101 MediaWiki Rendering on mw101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:34:42] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.42, 11.11, 10.43 [18:34:42] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:34:47] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:34:59] RECOVERY - cp20 Stunnel HTTP for mw101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14634 bytes in 9.193 second response time [18:35:35] RECOVERY - cp30 Stunnel HTTP for mw101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14640 bytes in 8.444 second response time [18:35:42] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 1595 bytes in 0.012 second response time [18:35:46] RECOVERY - cp31 Stunnel HTTP for mw101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14640 bytes in 5.379 second response time [18:35:46] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.64, 11.50, 10.15 [18:35:48] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 73, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') File "/usr/l [18:35:48] n3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query bellinrattin.it. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [18:35:53] RECOVERY - cp21 Stunnel HTTP for mw101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14634 bytes in 5.530 second response time [18:36:38] RECOVERY - mw101 MediaWiki Rendering on mw101 is OK: HTTP OK: HTTP/1.1 200 OK - 22334 bytes in 3.995 second response time [18:36:46] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:37:08] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:37:46] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.62, 11.91, 10.47 [18:38:51] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.30, 11.90, 10.95 [18:39:56] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 6.76, 6.80, 6.07 [18:40:45] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.77, 12.14, 11.14 [18:42:38] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.29, 11.58, 11.07 [18:43:04] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14634 bytes in 9.686 second response time [18:43:08] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14640 bytes in 9.642 second response time [18:43:31] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14634 bytes in 8.432 second response time [18:43:56] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 6.65, 6.93, 6.30 [18:44:05] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 22335 bytes in 8.183 second response time [18:44:32] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 13.06, 11.89, 11.22 [18:45:04] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14640 bytes in 8.641 second response time [18:45:46] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.14, 11.43, 10.96 [18:45:56] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 6.15, 6.63, 6.27 [18:48:41] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.07, 1.74, 1.61 [18:49:46] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 13.17, 11.86, 11.21 [18:51:46] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.08, 11.63, 11.21 [18:52:40] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.51, 1.75, 1.66 [18:54:07] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.41, 12.00, 11.88 [18:54:18] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.47, 11.89, 11.68 [18:54:39] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.26, 1.59, 1.61 [18:56:38] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.88, 1.51, 1.17 [18:58:37] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.27, 1.33, 1.14 [18:58:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.71, 11.32, 11.70 [19:01:29] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.82, 8.14, 7.10 [19:03:24] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 6.48, 7.39, 6.94 [19:03:46] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.73, 8.51, 9.90 [19:05:43] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp21.miraheze.org - CNAME OK [19:06:07] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.23, 8.85, 10.13 [19:06:18] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.80, 8.64, 10.17 [19:06:53] [02ManageWiki] 07Universal-Omega closed pull request 03#357: Bump stylelint-config-wikimedia from 0.12.2 to 0.13.0 - 13https://github.com/miraheze/ManageWiki/pull/357 [19:06:53] [url] Page not found · GitHub · GitHub | github.com [19:06:54] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/ManageWiki/compare/0865c92bfe7d...37ba236b61b9 [19:06:55] [url] Comparing 0865c92bfe7d...37ba236b61b9 · miraheze/ManageWiki · GitHub | github.com [19:06:56] [02miraheze/ManageWiki] 07dependabot[bot] 0337ba236 - Bump stylelint-config-wikimedia from 0.12.2 to 0.13.0 (#357) [19:06:57] [02ManageWiki] 07Universal-Omega deleted branch 03dependabot/npm_and_yarn/stylelint-config-wikimedia-0.13.0 - 13https://github.com/miraheze/ManageWiki [19:06:57] [url] Page not found · GitHub · GitHub | github.com [19:06:59] [02miraheze/ManageWiki] 07Universal-Omega deleted branch 03dependabot/npm_and_yarn/stylelint-config-wikimedia-0.13.0 [19:07:15] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 5.81, 6.60, 6.71 [19:07:20] [02ManageWiki] 07Universal-Omega commented on pull request 03#356: Combine addModuleStyles - 13https://github.com/miraheze/ManageWiki/pull/356#issuecomment-1118952787 [19:07:20] [url] Combine addModuleStyles by Universal-Omega · Pull Request #356 · miraheze/ManageWiki · GitHub | github.com [19:07:21] [02ManageWiki] 07Universal-Omega closed pull request 03#356: Combine addModuleStyles - 13https://github.com/miraheze/ManageWiki/pull/356 [19:07:22] [url] Page not found · GitHub · GitHub | github.com [19:07:23] [02miraheze/ManageWiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [19:07:25] [02ManageWiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/ManageWiki [19:07:25] [url] Page not found · GitHub · GitHub | github.com [19:07:47] [02MatomoAnalytics] 07Universal-Omega closed pull request 03#76: Bump stylelint-config-wikimedia from 0.12.2 to 0.13.0 - 13https://github.com/miraheze/MatomoAnalytics/pull/76 [19:07:47] [url] Page not found · GitHub · GitHub | github.com [19:07:48] [02miraheze/MatomoAnalytics] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/MatomoAnalytics/compare/27a79a75d73d...79374aa88e00 [19:07:49] [url] Comparing 27a79a75d73d...79374aa88e00 · miraheze/MatomoAnalytics · GitHub | github.com [19:07:50] [02miraheze/MatomoAnalytics] 07dependabot[bot] 0379374aa - Bump stylelint-config-wikimedia from 0.12.2 to 0.13.0 (#76) [19:07:51] [02miraheze/MatomoAnalytics] 07Universal-Omega deleted branch 03dependabot/npm_and_yarn/stylelint-config-wikimedia-0.13.0 [19:07:53] [02MatomoAnalytics] 07Universal-Omega deleted branch 03dependabot/npm_and_yarn/stylelint-config-wikimedia-0.13.0 - 13https://github.com/miraheze/MatomoAnalytics [19:07:53] ... [19:08:03] [02IncidentReporting] 07Universal-Omega closed pull request 03#41: Bump stylelint-config-wikimedia from 0.12.2 to 0.13.0 - 13https://github.com/miraheze/IncidentReporting/pull/41 [19:08:03] [url] Page not found · GitHub · GitHub | github.com [19:08:04] [02miraheze/IncidentReporting] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/IncidentReporting/compare/246c75f8f441...2a4ae16b7b80 [19:08:05] [url] Comparing 246c75f8f441...2a4ae16b7b80 · miraheze/IncidentReporting · GitHub | github.com [19:08:06] [02miraheze/IncidentReporting] 07dependabot[bot] 032a4ae16 - Bump stylelint-config-wikimedia from 0.12.2 to 0.13.0 (#41) [19:08:07] [02miraheze/IncidentReporting] 07Universal-Omega deleted branch 03dependabot/npm_and_yarn/stylelint-config-wikimedia-0.13.0 [19:08:09] [02IncidentReporting] 07Universal-Omega deleted branch 03dependabot/npm_and_yarn/stylelint-config-wikimedia-0.13.0 - 13https://github.com/miraheze/IncidentReporting [19:08:09] [url] Page not found · GitHub · GitHub | github.com [19:08:24] [02CreateWiki] 07Universal-Omega closed pull request 03#319: Bump stylelint-config-wikimedia from 0.12.2 to 0.13.0 - 13https://github.com/miraheze/CreateWiki/pull/319 [19:08:25] ... [19:08:26] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/CreateWiki/compare/1600a079bd87...173f1c179086 [19:08:26] [url] Comparing 1600a079bd87...173f1c179086 · miraheze/CreateWiki · GitHub | github.com [19:08:27] [02miraheze/CreateWiki] 07dependabot[bot] 03173f1c1 - Bump stylelint-config-wikimedia from 0.12.2 to 0.13.0 (#319) [19:08:29] [02CreateWiki] 07Universal-Omega deleted branch 03dependabot/npm_and_yarn/stylelint-config-wikimedia-0.13.0 - 13https://github.com/miraheze/CreateWiki [19:08:29] [url] Page not found · GitHub · GitHub | github.com [19:08:30] [02miraheze/CreateWiki] 07Universal-Omega deleted branch 03dependabot/npm_and_yarn/stylelint-config-wikimedia-0.13.0 [19:09:01] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [19:09:08] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.48, 3.44, 2.77 [19:11:15] miraheze/ManageWiki - Universal-Omega the build passed. [19:12:42] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.21, 9.44, 10.18 [19:13:08] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.38, 3.00, 2.75 [19:17:17] miraheze/MatomoAnalytics - Universal-Omega the build passed. [19:17:22] miraheze/IncidentReporting - Universal-Omega the build passed. [19:18:03] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.51, 3.50, 3.09 [19:20:03] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.46, 3.22, 3.04 [19:20:28] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.56, 1.86, 1.60 [19:22:27] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.76, 1.79, 1.60 [19:23:26] miraheze/CreateWiki - Universal-Omega the build passed. [19:26:01] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.51, 3.59, 3.20 [19:26:25] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.38, 1.59, 1.57 [19:28:00] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.32, 3.31, 3.14 [19:38:01] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [19:43:01] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [19:43:29] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.11, 10.73, 9.97 [19:45:22] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.82, 9.65, 9.67 [19:48:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [19:50:08] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/dfcb56bb33ec [19:50:09] [02miraheze/mw-config] 07Universal-Omega 03dfcb56b - Add GitHub link to the footer for datawiki [19:50:11] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/mw-config [19:50:11] [url] Page not found · GitHub · GitHub | github.com [19:50:15] [02mw-config] 07Universal-Omega opened pull request 03#4664: T9177: add GitHub link to the footer for datawiki - 13https://github.com/miraheze/mw-config/pull/4664 [19:50:16] [url] Page not found · GitHub · GitHub | github.com [19:51:06] [02mw-config] 07Universal-Omega synchronize pull request 03#4664: T9177: add GitHub link to the footer for datawiki - 13https://github.com/miraheze/mw-config/pull/4664 [19:51:06] ... [19:51:07] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/dfcb56bb33ec...6dbb9c81a839 [19:51:08] [url] Comparing dfcb56bb33ec...6dbb9c81a839 · miraheze/mw-config · GitHub | github.com [19:51:08] [02miraheze/mw-config] 07Universal-Omega 036dbb9c8 - Update LocalWiki.php [19:51:13] [02mw-config] 07Universal-Omega edited pull request 03#4664: T9177: add GitHub link to the footer for datawikiwiki - 13https://github.com/miraheze/mw-config/pull/4664 [19:51:13] [url] Page not found · GitHub · GitHub | github.com [19:51:33] miraheze/mw-config - Universal-Omega the build passed. [19:52:24] miraheze/mw-config - Universal-Omega the build passed. [19:53:48] [02mw-config] 07Universal-Omega closed pull request 03#4664: T9177: add GitHub link to the footer for datawikiwiki - 13https://github.com/miraheze/mw-config/pull/4664 [19:53:48] [url] Page not found · GitHub · GitHub | github.com [19:53:49] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/20c28ee3ea50...364a89cf3d6e [19:53:50] [url] Comparing 20c28ee3ea50...364a89cf3d6e · miraheze/mw-config · GitHub | github.com [19:53:51] [02miraheze/mw-config] 07Universal-Omega 03364a89c - T9177: add GitHub link to the footer for datawikiwiki (#4664) [19:53:52] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/mw-config [19:53:53] ... [19:53:54] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [19:54:52] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.82, 4.03, 3.43 [19:54:56] miraheze/mw-config - Universal-Omega the build passed. [19:57:08] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 7.50, 4.83, 3.49 [19:57:21] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.63, 3.94, 3.08 [19:58:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [19:59:20] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.65, 3.82, 3.14 [20:00:51] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.38, 3.69, 3.56 [20:01:08] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.03, 3.35, 3.20 [20:05:18] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.68, 3.72, 3.24 [20:07:17] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.40, 3.50, 3.22 [20:07:28] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - dnd.bellinrattin.it All nameservers failed to answer the query. [20:08:49] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 8.54, 5.23, 4.18 [20:09:16] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.76, 4.58, 3.65 [20:12:37] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.33, 1.77, 1.07 [20:12:48] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.84, 3.99, 3.90 [20:13:14] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.79, 3.65, 3.49 [20:14:37] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.14, 1.45, 1.03 [20:14:47] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 3.94, 4.03, 3.92 [20:15:14] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.69, 2.92, 3.24 [20:18:36] !log [@test101] starting deploy of {'config': True} to all [20:18:37] !log [@test101] finished deploy of {'config': True} to all - SUCCESS in 0s [20:18:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:18:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:19:01] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True} to all [20:19:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:19:11] !log [universalomega@mwtask111] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 10s [20:19:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:20:46] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.13, 3.77, 3.88 [20:26:01] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [20:28:24] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/364a89cf3d6e...713ca6d7d4b8 [20:28:24] [url] Comparing 364a89cf3d6e...713ca6d7d4b8 · miraheze/mw-config · GitHub | github.com [20:28:25] [02miraheze/mw-config] 07Universal-Omega 03713ca6d - Fix GitHub link on datawikiwiki to only appear in the places column [20:28:27] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.76, 3.48, 3.22 [20:28:39] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True} to all [20:28:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:28:49] !log [universalomega@mwtask111] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 9s [20:28:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:29:28] miraheze/mw-config - Universal-Omega the build passed. [20:30:23] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.86, 3.22, 3.15 [20:31:22] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/5f59ee42ee1e [20:31:24] [02miraheze/mw-config] 07Universal-Omega 035f59ee4 - Update LocalSettings.php [20:31:25] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/mw-config [20:31:26] [url] Page not found · GitHub · GitHub | github.com [20:34:10] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/5f59ee42ee1e...4bf0d25a86a1 [20:34:10] [url] Comparing 5f59ee42ee1e...4bf0d25a86a1 · miraheze/mw-config · GitHub | github.com [20:34:11] [02miraheze/mw-config] 07Universal-Omega 034bf0d25 - Update ManageWikiSettings.php [20:34:38] [02mw-config] 07Universal-Omega opened pull request 03#4665: Add `$wgPropertySuggesterMinProbability` to ManageWikiSettings - 13https://github.com/miraheze/mw-config/pull/4665 [20:34:39] [url] Page not found · GitHub · GitHub | github.com [20:35:30] [02mw-config] 07Universal-Omega closed pull request 03#4665: Add `$wgPropertySuggesterMinProbability` to ManageWikiSettings - 13https://github.com/miraheze/mw-config/pull/4665 [20:35:31] [url] Page not found · GitHub · GitHub | github.com [20:35:32] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/mw-config/compare/713ca6d7d4b8...e49592b72879 [20:35:32] [url] Comparing 713ca6d7d4b8...e49592b72879 · miraheze/mw-config · GitHub | github.com [20:35:33] [02miraheze/mw-config] 07Universal-Omega 03e49592b - Add `$wgPropertySuggesterMinProbability` to ManageWikiSettings (#4665) [20:35:35] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/mw-config [20:35:35] [url] Page not found · GitHub · GitHub | github.com [20:35:36] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [20:35:38] [02mw-config] 07Universal-Omega commented on pull request 03#4623: set wgPropertySuggesterMinProbability for gratisdatawiki - 13https://github.com/miraheze/mw-config/pull/4623#issuecomment-1119021917 [20:35:39] [url] set wgPropertySuggesterMinProbability for gratisdatawiki by ugochimobi · Pull Request #4623 · miraheze/mw-config · GitHub | github.com [20:35:39] [02mw-config] 07Universal-Omega closed pull request 03#4623: set wgPropertySuggesterMinProbability for gratisdatawiki - 13https://github.com/miraheze/mw-config/pull/4623 [20:35:40] [url] Page not found · GitHub · GitHub | github.com [20:35:52] miraheze/mw-config - Universal-Omega the build passed. [20:36:01] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [20:36:32] miraheze/mw-config - Universal-Omega the build passed. [20:36:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.48, 9.73, 8.99 [20:37:22] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp20.miraheze.org - CNAME OK [20:38:02] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.95, 3.64, 3.35 [20:38:08] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.21, 3.73, 3.39 [20:38:36] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/b96ce58c32ac [20:38:38] [02miraheze/mw-config] 07Universal-Omega 03b96ce58 - Update LocalSettings.php [20:38:39] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/mw-config [20:38:40] [url] Page not found · GitHub · GitHub | github.com [20:38:42] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 7.85, 4.70, 3.97 [20:38:42] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.85, 9.76, 9.10 [20:42:54] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/b96ce58c32ac...5e80c170ca42 [20:42:55] [url] Comparing b96ce58c32ac...5e80c170ca42 · miraheze/mw-config · GitHub | github.com [20:42:56] [02miraheze/mw-config] 07Universal-Omega 035e80c17 - Update ManageWikiSettings.php [20:43:23] [02mw-config] 07Universal-Omega opened pull request 03#4666: Add two WikibaseLexeme configuration variables to ManageWikiSettings - 13https://github.com/miraheze/mw-config/pull/4666 [20:43:24] ... [20:44:01] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [20:44:20] [02mw-config] 07Universal-Omega closed pull request 03#4666: Add two WikibaseLexeme configuration variables to ManageWikiSettings - 13https://github.com/miraheze/mw-config/pull/4666 [20:44:21] [url] Page not found · GitHub · GitHub | github.com [20:44:22] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [20:44:23] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/mw-config/compare/e49592b72879...c83798069653 [20:44:24] [url] Comparing e49592b72879...c83798069653 · miraheze/mw-config · GitHub | github.com [20:44:25] [02miraheze/mw-config] 07Universal-Omega 03c837980 - Add two WikibaseLexeme configuration variables to ManageWikiSettings (#4666) [20:44:26] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/mw-config [20:44:27] [url] Page not found · GitHub · GitHub | github.com [20:44:28] [02mw-config] 07Universal-Omega commented on pull request 03#4622: set wgLexemeLanguageCodePropertyId and wgLexemeEnableDataTransclusion for gratisdatawiki - 13https://github.com/miraheze/mw-config/pull/4622#issuecomment-1119028703 [20:44:29] [url] set wgLexemeLanguageCodePropertyId and wgLexemeEnableDataTransclusion for gratisdatawiki by ugochimobi · Pull Request #4622 · miraheze/mw-config · GitHub | github.com [20:44:29] [02mw-config] 07Universal-Omega closed pull request 03#4622: set wgLexemeLanguageCodePropertyId and wgLexemeEnableDataTransclusion for gratisdatawiki - 13https://github.com/miraheze/mw-config/pull/4622 [20:44:30] [url] Page not found · GitHub · GitHub | github.com [20:44:32] miraheze/mw-config - Universal-Omega the build passed. [20:45:23] miraheze/mw-config - Universal-Omega the build passed. [20:46:57] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True} to all [20:47:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:47:10] !log [universalomega@mwtask111] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 12s [20:47:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:48:41] !log [@test101] starting deploy of {'config': True} to all [20:48:42] !log [@test101] finished deploy of {'config': True} to all - SUCCESS in 0s [20:48:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:48:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:49:01] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [20:49:59] !log [universalomega@test101] sudo -u www-data php /srv/mediawiki/w/extensions/SemanticMediaWiki/maintenance/setupStore.php --wiki=smwwikibeta (END - exit=0) [20:50:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:53:32] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.53, 3.53, 3.73 [20:55:06] !log [universalomega@test101] starting deploy of {'world': True, 'l10n': True} to all [20:55:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:57:23] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.79, 3.78, 3.76 [20:57:54] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.88, 1.57, 1.00 [20:59:53] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.37, 1.85, 1.17 [21:01:14] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.98, 3.99, 3.90 [21:01:49] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/4257fb2c4e0b [21:01:51] [02miraheze/mw-config] 07Universal-Omega 034257fb2 - Re-enable Maps on 1.38 [21:01:52] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.00, 3.64, 3.92 [21:01:52] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/mw-config [21:01:52] [url] Page not found · GitHub · GitHub | github.com [21:01:54] [02mw-config] 07Universal-Omega opened pull request 03#4667: Re-enable Maps on 1.38 - 13https://github.com/miraheze/mw-config/pull/4667 [21:01:54] [url] Page not found · GitHub · GitHub | github.com [21:03:04] miraheze/mw-config - Universal-Omega the build passed. [21:05:02] !log [universalomega@test101] finished deploy of {'world': True, 'l10n': True} to all - SUCCESS in 596s [21:05:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:05:35] [02mw-config] 07Universal-Omega closed pull request 03#4667: Re-enable Maps on 1.38 - 13https://github.com/miraheze/mw-config/pull/4667 [21:05:36] [url] Page not found · GitHub · GitHub | github.com [21:05:37] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/c83798069653...cc0c22019167 [21:05:38] [url] Comparing c83798069653...cc0c22019167 · miraheze/mw-config · GitHub | github.com [21:05:39] [02miraheze/mw-config] 07Universal-Omega 03cc0c220 - Re-enable Maps on 1.38 (#4667) [21:05:40] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/mw-config [21:05:40] ... [21:05:42] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [21:06:39] miraheze/mw-config - Universal-Omega the build passed. [21:07:15] @Orduin ping [21:07:15] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 3.66, 2.27, 1.37 [21:07:58] [02mw-config] 07Universal-Omega commented on pull request 03#4612: override page HTML title with the wikibase label, if available, or else the id - 13https://github.com/miraheze/mw-config/pull/4612#issuecomment-1119047059 [21:08:00] [02mw-config] 07Universal-Omega closed pull request 03#4612: override page HTML title with the wikibase label, if available, or else the id - 13https://github.com/miraheze/mw-config/pull/4612 [21:08:00] [url] override page HTML title with the wikibase label, if available, or else the id by ugochimobi · Pull Request #4612 · miraheze/mw-config · GitHub | github.com [21:08:01] [url] Page not found · GitHub · GitHub | github.com [21:08:24] [02mw-config] 07Universal-Omega closed pull request 03#4655: Move SiteConfiguration setup to MirahezeFunctions - 13https://github.com/miraheze/mw-config/pull/4655 [21:08:24] ... [21:08:26] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 - 13https://github.com/miraheze/mw-config [21:08:27] [url] Page not found · GitHub · GitHub | github.com [21:08:28] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 [21:09:08] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.19, 3.86, 3.76 [21:09:10] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.76, 1.64, 1.25 [21:11:08] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.50, 3.58, 3.67 [21:11:49] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.23, 1.84, 1.79 [21:12:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:13:08] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.28, 3.93, 3.76 [21:15:46] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.46, 3.62, 3.62 [21:17:08] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.89, 3.78, 3.73 [21:17:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:17:45] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.48, 3.31, 3.51 [21:17:46] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.05, 1.78, 1.75 [21:19:08] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.03, 3.67, 3.68 [21:21:16] !log [@test101] starting deploy of {'config': True} to all [21:21:17] !log [@test101] finished deploy of {'config': True} to all - SUCCESS in 1s [21:21:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:21:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:22:23] !log [@mwtask111] starting deploy of {'config': True} to all [21:22:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:22:36] !log [@mwtask111] finished deploy of {'config': True} to all - SUCCESS in 13s [21:22:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:23:44] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.60, 1.91, 1.85 [21:25:08] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.50, 3.63, 3.73 [21:25:22] PROBLEM - test101 MediaWiki Rendering on test101 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 785 bytes in 0.632 second response time [21:29:41] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.18, 2.92, 3.33 [21:32:29] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.44, 3.17, 3.84 [21:33:40] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.38, 1.50, 1.67 [21:35:08] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.46, 3.54, 3.60 [21:36:28] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.13, 3.72, 3.90 [21:36:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:37:08] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.86, 3.25, 3.48 [21:37:30] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.99, 2.93, 1.85 [21:38:27] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.73, 3.75, 3.90 [21:39:08] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 3.08, 3.06, 3.38 [21:40:05] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - dnd.bellinrattin.it All nameservers failed to answer the query. [21:41:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:43:18] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.07, 2.00, 1.77 [21:45:08] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.45, 3.43, 3.45 [21:45:14] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.81, 1.59, 1.65 [21:46:26] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 7.56, 5.17, 4.37 [21:47:08] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.27, 3.74, 3.55 [21:49:33] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.57, 2.11, 1.79 [21:49:47] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03revert-4667-Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/d73a1df35176 [21:49:48] [02miraheze/mw-config] 07Universal-Omega 03d73a1df - Revert "Re-enable Maps on 1.38 (#4667)" [21:49:50] [02mw-config] 07Universal-Omega created branch 03revert-4667-Universal-Omega-patch-1 - 13https://github.com/miraheze/mw-config [21:49:50] [url] Page not found · GitHub · GitHub | github.com [21:49:51] [02mw-config] 07Universal-Omega opened pull request 03#4668: Revert "Re-enable Maps on 1.38" - 13https://github.com/miraheze/mw-config/pull/4668 [21:49:52] [url] Page not found · GitHub · GitHub | github.com [21:49:53] [02mw-config] 07Universal-Omega closed pull request 03#4668: Revert "Re-enable Maps on 1.38" - 13https://github.com/miraheze/mw-config/pull/4668 [21:49:54] [url] Page not found · GitHub · GitHub | github.com [21:49:55] [02mw-config] 07Universal-Omega deleted branch 03revert-4667-Universal-Omega-patch-1 - 13https://github.com/miraheze/mw-config [21:49:55] ... [21:49:56] [02miraheze/mw-config] 07Universal-Omega deleted branch 03revert-4667-Universal-Omega-patch-1 [21:49:58] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/cc0c22019167...5ee639bf6519 [21:49:58] [url] Comparing cc0c22019167...5ee639bf6519 · miraheze/mw-config · GitHub | github.com [21:49:59] [02miraheze/mw-config] 07Universal-Omega 035ee639b - Revert "Re-enable Maps on 1.38 (#4667)" (#4668) [21:50:07] !log [@test101] starting deploy of {'config': True} to all [21:50:08] !log [@test101] DEPLOY ABORTED: Canary check failed for beta.betaheze.org@localhost [21:50:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:50:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:50:55] miraheze/mw-config - Universal-Omega the build passed. [21:51:04] miraheze/mw-config - Universal-Omega the build passed. [21:51:22] RECOVERY - test101 MediaWiki Rendering on test101 is OK: HTTP OK: HTTP/1.1 200 OK - 52094 bytes in 1.101 second response time [21:52:20] !log [@mwtask111] starting deploy of {'config': True} to all [21:52:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:52:29] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.51, 3.58, 3.41 [21:52:32] !log [@mwtask111] finished deploy of {'config': True} to all - SUCCESS in 12s [21:52:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:54:28] PROBLEM - test101 Puppet on test101 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[MediaWiki Config Sync] [21:54:29] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.23, 3.24, 3.31 [21:57:08] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.25, 3.39, 3.61 [21:57:29] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.45, 1.85, 1.85 [21:58:23] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.23, 3.56, 3.98 [22:01:08] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.26, 2.89, 3.36 [22:04:22] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.04, 3.86, 3.87 [22:06:21] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.77, 3.69, 3.81 [22:09:24] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.65, 1.59, 1.69 [22:09:59] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp20.miraheze.org - CNAME OK [22:12:20] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.71, 3.76, 3.73 [22:14:19] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.85, 3.68, 3.70 [22:18:18] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.04, 4.28, 3.89 [22:19:22] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.28, 1.88, 1.73 [22:20:18] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.36, 3.63, 3.68 [22:22:29] RECOVERY - test101 Puppet on test101 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [22:23:20] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.56, 1.90, 1.80 [22:23:56] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.17, 7.24, 5.99 [22:24:19] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.18, 4.36, 3.97 [22:25:55] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 4.81, 6.50, 5.89 [22:27:19] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.21, 1.50, 1.65 [22:28:18] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.73, 3.91, 3.90 [22:31:03] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.15, 10.60, 9.00 [22:31:27] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.90, 9.58, 7.44 [22:31:27] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.47, 8.76, 7.23 [22:31:56] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 9.36, 8.11, 6.71 [22:32:07] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.47, 10.53, 8.69 [22:32:18] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.90, 9.60, 8.10 [22:32:57] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.76, 10.61, 9.19 [22:33:56] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.27, 7.68, 6.72 [22:34:18] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 13.52, 10.65, 8.65 [22:34:39] PROBLEM - mw101 MediaWiki Rendering on mw101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:34:56] PROBLEM - cp20 Stunnel HTTP for mw101 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:35:27] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.00, 10.71, 8.44 [22:35:31] PROBLEM - cp30 Stunnel HTTP for mw101 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:35:45] PROBLEM - cp31 Stunnel HTTP for mw101 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:35:52] PROBLEM - cp21 Stunnel HTTP for mw101 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:36:16] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.68, 2.52, 3.27 [22:36:18] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.05, 10.37, 8.80 [22:36:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [22:36:47] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:37:01] RECOVERY - cp20 Stunnel HTTP for mw101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14640 bytes in 8.636 second response time [22:37:27] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.53, 10.12, 8.49 [22:37:27] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 9.56, 9.56, 8.16 [22:37:35] RECOVERY - cp30 Stunnel HTTP for mw101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14634 bytes in 7.032 second response time [22:37:47] RECOVERY - cp31 Stunnel HTTP for mw101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14640 bytes in 6.813 second response time [22:37:55] RECOVERY - cp21 Stunnel HTTP for mw101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14634 bytes in 6.250 second response time [22:37:56] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.96, 7.97, 7.02 [22:38:42] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.04, 11.62, 10.09 [22:38:52] RECOVERY - mw101 MediaWiki Rendering on mw101 is OK: HTTP OK: HTTP/1.1 200 OK - 22335 bytes in 5.598 second response time [22:38:53] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 22334 bytes in 7.028 second response time [22:39:56] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.81, 7.65, 7.01 [22:40:18] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 13.34, 11.87, 9.76 [22:40:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.91, 11.76, 10.33 [22:41:56] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.39, 8.00, 7.22 [22:42:10] PROBLEM - cloud10 Puppet on cloud10 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [22:44:18] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.15, 11.47, 10.18 [22:45:56] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 6.27, 7.42, 7.19 [22:46:13] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.54, 3.53, 3.35 [22:49:09] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.69, 1.93, 1.61 [22:49:28] PROBLEM - cp20 Stunnel HTTP for mw101 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:50:37] PROBLEM - mw101 MediaWiki Rendering on mw101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:51:32] RECOVERY - cp20 Stunnel HTTP for mw101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14626 bytes in 8.030 second response time [22:51:56] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 5.90, 6.27, 6.72 [22:52:12] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.85, 3.54, 3.47 [22:52:33] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 0.93, 1.77, 1.36 [22:52:43] RECOVERY - mw101 MediaWiki Rendering on mw101 is OK: HTTP OK: HTTP/1.1 200 OK - 22334 bytes in 7.048 second response time [22:54:11] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.79, 4.44, 3.81 [22:54:28] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.49, 1.35, 1.25 [22:55:07] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.35, 1.83, 1.71 [22:58:07] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.20, 11.97, 11.86 [22:59:05] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.43, 1.63, 1.66 [22:59:27] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.31, 10.04, 9.19 [23:00:02] !log [@mwtask111] starting deploy of {'l10nupdate': True} to all [23:00:03] !log [@test101] starting deploy of {'l10nupdate': True} to all [23:00:07] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 13.20, 12.45, 12.05 [23:00:10] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.45, 3.81, 3.75 [23:00:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:00:42] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.35, 11.54, 11.04 [23:01:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:01:27] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.56, 9.05, 8.93 [23:06:18] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.68, 9.69, 10.13 [23:06:42] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.81, 11.76, 11.30 [23:08:08] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.42, 3.00, 3.38 [23:10:10] RECOVERY - cloud10 Puppet on cloud10 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:11:44] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 73, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') File "/usr/l [23:11:45] n3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query bellinrattin.it. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [23:12:07] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.53, 10.57, 11.54 [23:12:54] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.17, 3.95, 3.09 [23:14:06] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.81, 4.69, 3.95 [23:14:42] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.05, 8.88, 10.07 [23:16:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [23:16:52] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 1.76, 3.55, 3.20 [23:16:56] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 3.20, 2.23, 1.86 [23:18:05] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.64, 3.78, 3.76 [23:20:07] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.96, 7.94, 9.89 [23:20:51] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.71, 2.89, 3.03 [23:23:07] PROBLEM - cloud11 Puppet on cloud11 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [23:24:04] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.99, 3.03, 3.38 [23:25:15] !log [@mwtask111] finished deploy of {'l10nupdate': True} to all - SUCCESS in 1513s [23:25:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:30:26] !log [@test101] finished deploy of {'l10nupdate': True} to all - SUCCESS in 1823s [23:30:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:42:46] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.37, 1.59, 1.98 [23:46:44] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.25, 1.66, 1.91 [23:50:43] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.88, 1.89, 1.95 [23:51:06] RECOVERY - cloud11 Puppet on cloud11 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures