[06:47:52] PROBLEM - ns1 NTP time on ns1 is UNKNOWN: [06:49:34] PROBLEM - mon1 SSH on mon1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:49:39] PROBLEM - mon1 Current Load on mon1 is CRITICAL: LOAD CRITICAL - total load average: 33.50, 33.51, 27.03 PROBLEM - mon1 Current Load on mon1 is CRITICAL: LOAD CRITICAL - total load average: 34.95, 33.75, 27.61 [06:49:42] PROBLEM - mon1 grafana.inside.wf HTTPS on mon1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds PROBLEM - mon1 PowerDNS Recursor on mon1 is CRITICAL: CRITICAL - Plugin timed out while executing system call PROBLEM - ns2 Auth DNS on ns2 is CRITICAL: CRITICAL - Plugin timed out while executing system call PROBLEM - mon1 PowerDNS Recursor on mon1 is CRITICAL: CRITICAL - Plugin timed out while executing system call [06:49:46] PROBLEM - mon1 monitoring.inside.wf HTTPS on mon1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds PROBLEM - mon1 monitoring.inside.wf HTTPS on mon1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:53:00] PROBLEM - ns1 NTP time on ns1 is CRITICAL: CHECK_NRPE: (ssl_err != 5) Error - Could not complete SSL handshake with 158.69.62.222: 1 PROBLEM - mon1 Check correctness of the icinga configuration on mon1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:53:03] PROBLEM - ns1 Auth DNS on ns1 is CRITICAL: CRITICAL - Plugin timed out while executing system call [06:58:40] PROBLEM - mon1 Backups Grafana on mon1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:59:07] PROBLEM - mon1 Puppet on mon1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:33:59] PROBLEM - cp5 PowerDNS Recursor on cp5 is UNKNOWN: PROBLEM - cp3 Nginx Backend for matomo1 on cp3 is UNKNOWN: PROBLEM - cp5 Nginx Backend for phorge1 on cp5 is UNKNOWN: PROBLEM - cp2 Nginx Backend for mwdedi2 on cp2 is UNKNOWN: Terminated).> PROBLEM - cp3 Nginx Backend for mwdedi2 on cp3 is UNKNOWN: PROBLEM - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is UNKNOWN: [07:34:25] PROBLEM - mon1 APT on mon1 is UNKNOWN: PROBLEM - cloud1 Puppet on cloud1 is UNKNOWN: [07:39:57] PROBLEM - ns1 Current Load on ns1 is UNKNOWN: PROBLEM - cp6 PowerDNS Recursor on cp6 is UNKNOWN: [07:40:36] PROBLEM - cp6 Nginx Backend for mw1 on cp6 is UNKNOWN: [08:38:37] !log [reception@jobrunner1] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/eval.php --wiki=loginwiki (END - exit=65280) [08:38:42] !log [reception@jobrunner1] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/eval.php --wiki=metawiki (END - exit=65280) [08:38:56] !log [reception@jobrunner1] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/eval.php --wiki=hubwiki (END - exit=65280) [08:39:19] !log [reception@jobrunner1] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/eval.php --wiki=hubwiki (END - exit=65280) [08:47:38] [02WikiForge/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiForge/mw-config/compare/dab2e3878cce...e9298a6de6e9 [08:47:41] [02WikiForge/mw-config] 07Reception123 03e9298a6 - raise transaction limit to 60 due to persistent CW error [08:48:37] !log [reception@jobrunner1] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/eval.php --wiki=hubwiki (END - exit=2) [08:48:39] WikiForge/mw-config - Reception123 the build passed. [08:48:43] !log [reception@jobrunner1] starting deploy of {'pull': 'config', 'config': True, 'versions': ['1.40']} to [mw1, mw2, mwdedi1, mwdedi2, jobrunner1] [08:48:58] !log [reception@jobrunner1] finished deploy of {'pull': 'config', 'config': True, 'versions': ['1.40']} to [mw1, mw2, mwdedi1, mwdedi2, jobrunner1] - SUCCESS in 15s [09:45:42] PROBLEM - cp5 PowerDNS Recursor on cp5 is CRITICAL: CHECK_NRPE: (ssl_err != 5) Error - Could not complete SSL handshake with 15.235.192.162: 1 PROBLEM - cp2 Nginx Backend for mwdedi2 on cp2 is CRITICAL: CHECK_NRPE: (ssl_err != 5) Error - Could not complete SSL handshake with 158.69.62.132: 1 PROBLEM - cp3 Nginx Backend for matomo1 on cp3 is CRITICAL: CHECK_NRPE: (ssl_err != 5) Error - Could not complete SSL handshake with 51.68.214.24 [09:45:42] - cp5 Nginx Backend for phorge1 on cp5 is CRITICAL: CHECK_NRPE: (ssl_err != 5) Error - Could not complete SSL handshake with 15.235.192.162: 1 PROBLEM - cp3 Nginx Backend for mwdedi2 on cp3 is CRITICAL: CHECK_NRPE: (ssl_err != 5) Error - Could not complete SSL handshake with 51.68.214.246: 1 PROBLEM - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is CRITICAL: CHECK_NRPE: (ssl_err != 5) Error - Could not complete SSL handshake with 158.69.62.132 [09:45:43] - cloud1 Puppet on cloud1 is CRITICAL: CRITICAL: Puppet has 3 failures. Last run 7 minutes ago with 3 failures. Failed resources (up to 3 shown): Package[openssh-client],Package[openssh-server],File[/etc/resolv.conf] PROBLEM - mon1 APT on mon1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [09:46:07] PROBLEM - ns1 Current Load on ns1 is CRITICAL: CHECK_NRPE: (ssl_err != 5) Error - Could not complete SSL handshake with 158.69.62.222: 1 [09:47:02] PROBLEM - cp6 PowerDNS Recursor on cp6 is CRITICAL: CHECK_NRPE: (ssl_err != 5) Error - Could not complete SSL handshake with 51.161.131.113: 1 [19:43:12] RECOVERY - mon1 Puppet on mon1 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [19:43:13] PROBLEM - cp5 Nginx Backend for phorge1 on cp5 is CRITICAL: CHECK_NRPE: Error - Could not connect to 15.235.192.162: Connection reset by peer [19:43:14] RECOVERY - wiki.fishonmc.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.fishonmc.net' will expire on Sat 20 Apr 2024 12:49:56 PM GMT +0000. [19:43:15] PROBLEM - cp5 Nginx Backend for test1 on cp5 is CRITICAL: CHECK_NRPE: Error - Could not connect to 15.235.192.162: Connection reset by peer [19:43:15] PROBLEM - cp3 Disk Space on cp3 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.68.214.246: Connection reset by peer [19:43:16] PROBLEM - cp6 NTP time on cp6 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.161.131.113: Connection reset by peer [19:43:22] RECOVERY - dcmultiverse.wikiforge.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wikiforge.net' will expire on Wed 10 Apr 2024 07:15:27 PM GMT +0000. [19:43:22] PROBLEM - cp3 NTP time on cp3 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.68.214.246: Connection reset by peer [19:43:23] PROBLEM - cp3 Nginx Backend for puppet1 on cp3 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.68.214.246: Connection reset by peer [19:43:23] PROBLEM - cp5 Nginx Backend for mwdedi2 on cp5 is CRITICAL: CHECK_NRPE: Error - Could not connect to 15.235.192.162: Connection reset by peer [19:43:25] PROBLEM - cp5 Disk Space on cp5 is CRITICAL: CHECK_NRPE: Error - Could not connect to 15.235.192.162: Connection reset by peer [19:43:29] RECOVERY - beaconspace.unrestrictedlorefare.com - LetsEncrypt on sslhost is OK: OK - Certificate 'beaconspace.unrestrictedlorefare.com' will expire on Wed 10 Apr 2024 08:22:51 PM GMT +0000. [19:43:35] RECOVERY - mon1 Check correctness of the icinga configuration on mon1 is OK: Icinga configuration is correct [19:43:39] PROBLEM - cp2 Nginx Backend for test1 on cp2 is CRITICAL: CHECK_NRPE: Error - Could not connect to 158.69.62.132: Connection reset by peer [19:43:42] RECOVERY - www.rippaversewiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'rippaversewiki.com' will expire on Thu 11 Apr 2024 04:59:45 AM GMT +0000. [19:43:47] PROBLEM - cp2 Current Load on cp2 is CRITICAL: CHECK_NRPE: Error - Could not connect to 158.69.62.132: Connection reset by peer [19:43:48] RECOVERY - churchof0days.com - LetsEncrypt on sslhost is OK: OK - Certificate 'churchof0days.com' will expire on Sat 13 Apr 2024 06:35:48 PM GMT +0000. [19:43:51] RECOVERY - mon1 IRCEcho on mon1 is OK: PROCS OK: 1 process with args '/usr/local/bin/ircecho' [19:43:51] PROBLEM - cp6 Disk Space on cp6 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.161.131.113: Connection reset by peer [19:43:52] PROBLEM - cp2 Nginx Backend for mail1 on cp2 is CRITICAL: CHECK_NRPE: Error - Could not connect to 158.69.62.132: Connection reset by peer [19:43:52] RECOVERY - mon1 PowerDNS Recursor on mon1 is OK: DNS OK: 0.032 seconds response time. wikiforge.net returns 2001:41d0:801:2000::2200,51.68.214.246 [19:44:01] RECOVERY - avid.wikiforge.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wikiforge.net' will expire on Wed 10 Apr 2024 07:15:27 PM GMT +0000. [19:44:01] PROBLEM - cp3 Nginx Backend for phorge1 on cp3 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.68.214.246: Connection reset by peer [19:44:01] RECOVERY - www.theharrypotter.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'theharrypotter.wiki' will expire on Tue 26 Mar 2024 03:34:20 AM GMT +0000. [19:44:02] PROBLEM - cp3 Nginx Backend for mw1 on cp3 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.68.214.246: Connection reset by peer [19:44:05] PROBLEM - cp4 Nginx Backend for mwdedi2 on cp4 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.38.134.38: Connection reset by peer [20:18:31] PROBLEM - mon1 Current Load on mon1 is WARNING: LOAD WARNING - total load average: 0.07, 0.13, 5.30 [20:20:26] RECOVERY - mon1 Current Load on mon1 is OK: LOAD OK - total load average: 0.11, 0.13, 4.70