[00:05:59] [02WikiForge/ssl] 07WikiForgeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiForge/ssl/compare/489ee265a84b...cb08f2e640a3 [00:06:01] [02WikiForge/ssl] 07WikiForgeSSLBot 03cb08f2e - Bot: Update SSL cert for lophocmatngu.wiki [03:00:07] RECOVERY - mon1 Backups Grafana on mon1 is OK: FILE_AGE OK: /var/log/grafana-backup.log is 5 seconds old and 349 bytes [09:10:57] PROBLEM - cp6 SSH on cp6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:11:40] PROBLEM - Host cp6 is DOWN: PING CRITICAL - Packet loss = 100% [09:13:03] PROBLEM - cp5 SSH on cp5 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:13:51] PROBLEM - cp4 SSH on cp4 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:14:15] PROBLEM - Host cp4 is DOWN: PING CRITICAL - Packet loss = 100% [09:15:24] PROBLEM - Host cp5 is DOWN: PING CRITICAL - Packet loss = 100% [09:16:12] PROBLEM - ns2 SSH on ns2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:16:25] PROBLEM - ns2 Auth DNS on ns2 is CRITICAL: CRITICAL - Plugin timed out while executing system call [09:17:43] PROBLEM - Host ns2 is DOWN: PING CRITICAL - Packet loss = 100% [09:21:21] PROBLEM - www.wikiforge.net - LetsEncrypt on sslhost is CRITICAL: connect to address www.wikiforge.net and port 443: No route to hostHTTP CRITICAL - Unable to open TCP socket [09:21:31] PROBLEM - wc.wikiforge.net on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:22:31] PROBLEM - Host cp3 is DOWN: PING CRITICAL - Packet loss = 100% [09:23:40] PROBLEM - swiftproxy1 PowerDNS Recursor on swiftproxy1 is CRITICAL: Domain 'wikiforge.net' was not found by the server [09:24:28] PROBLEM - ns1 Auth DNS on ns1 is CRITICAL: CRITICAL - Plugin timed out while executing system call [09:24:56] PROBLEM - jobrunner1 MediaWiki Rendering on jobrunner1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:25:01] PROBLEM - mon1 PowerDNS Recursor on mon1 is CRITICAL: Domain 'wikiforge.net' was not found by the server [09:25:07] PROBLEM - ns1 SSH on ns1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:25:29] PROBLEM - Host ns1 is DOWN: PING CRITICAL - Packet loss = 100% [09:25:46] PROBLEM - mail1 PowerDNS Recursor on mail1 is CRITICAL: Domain 'wikiforge.net' was not found by the server [09:25:50] PROBLEM - swiftobject1 PowerDNS Recursor on swiftobject1 is CRITICAL: Domain 'wikiforge.net' was not found by the server [09:25:58] PROBLEM - jobrunner1 PowerDNS Recursor on jobrunner1 is CRITICAL: Domain 'wikiforge.net' was not found by the server [09:26:08] PROBLEM - cp2 SSH on cp2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:26:09] PROBLEM - Host cp2 is DOWN: PING CRITICAL - Packet loss = 100% [09:26:18] PROBLEM - phorge1 phorge-static.wikiforge.net HTTPS on phorge1 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/1.1 500 Internal Server Error [09:26:18] PROBLEM - bast1 PowerDNS Recursor on bast1 is CRITICAL: Domain 'wikiforge.net' was not found by the server [09:26:19] PROBLEM - phorge1 support.wikiforge.net HTTPS on phorge1 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 4253 bytes in 0.032 second response time [09:26:31] PROBLEM - swiftac1 PowerDNS Recursor on swiftac1 is CRITICAL: Domain 'wikiforge.net' was not found by the server [09:26:34] PROBLEM - mw1 PowerDNS Recursor on mw1 is CRITICAL: Domain 'wikiforge.net' was not found by the server [09:26:40] PROBLEM - puppet1 PowerDNS Recursor on puppet1 is CRITICAL: Domain 'wikiforge.net' was not found by the server [09:26:43] PROBLEM - beaconspace.unrestrictedlorefare.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:26:53] PROBLEM - mw1 MediaWiki Rendering on mw1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:26:55] PROBLEM - mw2 PowerDNS Recursor on mw2 is CRITICAL: Domain 'wikiforge.net' was not found by the server [09:27:20] PROBLEM - phorge1 PowerDNS Recursor on phorge1 is CRITICAL: Domain 'wikiforge.net' was not found by the server [09:27:36] PROBLEM - cloud1 PowerDNS Recursor on cloud1 is CRITICAL: Domain 'wikiforge.net' was not found by the server [09:27:54] PROBLEM - churchof0days.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:28:51] PROBLEM - avid.wikiforge.net - LetsEncrypt on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket [09:29:24] PROBLEM - wikiforge.net - LetsEncrypt on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket [09:30:01] PROBLEM - mw2 MediaWiki Rendering on mw2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:30:28] PROBLEM - harrypotter.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:30:45] PROBLEM - meta.wikiforge.net - LetsEncrypt on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket [09:32:20] PROBLEM - www.harrypotter.wiki - LetsEncrypt on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket [09:33:30] PROBLEM - dc.wikiforge.net - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:34:22] PROBLEM - theharrypotter.wiki - LetsEncrypt on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket [09:37:02] PROBLEM - avid.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:37:51] PROBLEM - dcmultiverse.wikiforge.net - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:38:46] PROBLEM - dcmultiversewiki.com - LetsEncrypt on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket [09:41:04] PROBLEM - ghostmachinewiki.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:42:46] PROBLEM - www.dcmultiversewiki.com - LetsEncrypt on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket [09:47:27] PROBLEM - rippaverse.wikiforge.net - LetsEncrypt on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket [09:49:52] PROBLEM - www.theharrypotter.wiki - LetsEncrypt on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket [09:51:00] PROBLEM - director.wikiforge.net - LetsEncrypt on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket [10:29:40] PROBLEM - bast1 Puppet on bast1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [10:34:23] PROBLEM - phorge1 Puppet on phorge1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [10:49:43] PROBLEM - mon1 Puppet on mon1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [10:56:53] RECOVERY - bast1 Puppet on bast1 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [11:02:46] RECOVERY - phorge1 Puppet on phorge1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [11:18:20] RECOVERY - mon1 Puppet on mon1 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [11:51:34] PROBLEM - mon1 Puppet on mon1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/apt/trusted.gpg.d/puppetlabs.gpg] [12:05:18] PROBLEM - phorge1 Puppet on phorge1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [12:20:06] RECOVERY - mon1 Puppet on mon1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [12:27:57] PROBLEM - mail1 Puppet on mail1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [12:33:09] RECOVERY - phorge1 Puppet on phorge1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [12:51:21] PROBLEM - mon1 Puppet on mon1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/apt/trusted.gpg.d/puppetlabs.gpg] [12:53:51] RECOVERY - mail1 Puppet on mail1 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [13:10:08] RECOVERY - Host ns2 is UP: PING OK - Packet loss = 0%, RTA = 95.38 ms [13:10:10] RECOVERY - cloud1 PowerDNS Recursor on cloud1 is OK: DNS OK: 0.229 seconds response time. wikiforge.net returns 208.110.84.235,2604:4300:a:37::116 [13:10:10] RECOVERY - Host cp3 is UP: PING OK - Packet loss = 0%, RTA = 99.16 ms [13:10:11] RECOVERY - Host cp4 is UP: PING OK - Packet loss = 0%, RTA = 138.72 ms [13:10:24] RECOVERY - mw1 MediaWiki Rendering on mw1 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.312 second response time [13:10:27] RECOVERY - mw1 PowerDNS Recursor on mw1 is OK: DNS OK: 0.228 seconds response time. wikiforge.net returns 208.110.84.235,2604:4300:a:37::116 [13:10:28] RECOVERY - bast1 PowerDNS Recursor on bast1 is OK: DNS OK: 0.228 seconds response time. wikiforge.net returns 208.110.84.235,2604:4300:a:37::116 [13:10:43] RECOVERY - ns2 Auth DNS on ns2 is OK: DNS OK: 0.232 seconds response time. inside.wf returns 208.110.84.235,2604:4300:a:37::116 [13:10:53] RECOVERY - cp4 SSH on cp4 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u2 (protocol 2.0) [13:10:57] RECOVERY - swiftobject1 PowerDNS Recursor on swiftobject1 is OK: DNS OK: 1.752 second response time. wikiforge.net returns 2001:41d0:601:1100::2d57,51.38.134.38 [13:11:02] RECOVERY - ns2 SSH on ns2 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u2 (protocol 2.0) [13:11:05] RECOVERY - swiftproxy1 PowerDNS Recursor on swiftproxy1 is OK: DNS OK: 0.241 seconds response time. wikiforge.net returns 2001:41d0:601:1100::2d57,51.38.134.38 [13:11:34] RECOVERY - jobrunner1 PowerDNS Recursor on jobrunner1 is OK: DNS OK: 0.234 seconds response time. wikiforge.net returns 2001:41d0:601:1100::2d57,51.38.134.38 [13:11:34] RECOVERY - swiftac1 PowerDNS Recursor on swiftac1 is OK: DNS OK: 1.737 second response time. wikiforge.net returns 2001:41d0:601:1100::2d57,51.38.134.38 [13:11:49] RECOVERY - phorge1 PowerDNS Recursor on phorge1 is OK: DNS OK: 0.228 seconds response time. wikiforge.net returns 2001:41d0:601:1100::2d57,51.38.134.38 [13:11:51] RECOVERY - mail1 PowerDNS Recursor on mail1 is OK: DNS OK: 0.237 seconds response time. wikiforge.net returns 2001:41d0:601:1100::2d57,51.38.134.38 [13:11:52] RECOVERY - mw2 MediaWiki Rendering on mw2 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.152 second response time [13:11:54] RECOVERY - phorge1 phorge-static.wikiforge.net HTTPS on phorge1 is OK: HTTP OK: Status line output matched "HTTP/1.1 200" - 53041 bytes in 3.829 second response time [13:11:56] RECOVERY - mw2 PowerDNS Recursor on mw2 is OK: DNS OK: 0.232 seconds response time. wikiforge.net returns 2001:41d0:601:1100::2d57,51.38.134.38 [13:12:00] RECOVERY - jobrunner1 MediaWiki Rendering on jobrunner1 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.177 second response time [13:12:14] RECOVERY - mon1 PowerDNS Recursor on mon1 is OK: DNS OK: 0.028 seconds response time. wikiforge.net returns 208.110.84.235,2604:4300:a:37::116 [13:12:26] RECOVERY - puppet1 PowerDNS Recursor on puppet1 is OK: DNS OK: 0.234 seconds response time. wikiforge.net returns 2001:41d0:601:1100::2d57,51.38.134.38 [13:13:03] RECOVERY - phorge1 support.wikiforge.net HTTPS on phorge1 is OK: HTTP OK: HTTP/1.1 200 OK - 6345 bytes in 0.064 second response time [13:14:22] RECOVERY - wc.wikiforge.net on sslhost is OK: OK - Certificate 'wikiforge.net' will expire on Wed 10 Apr 2024 07:15:27 PM GMT +0000. [13:14:33] RECOVERY - Host ns1 is UP: PING OK - Packet loss = 0%, RTA = 36.29 ms [13:14:59] RECOVERY - Host cp5 is UP: PING OK - Packet loss = 0%, RTA = 450.85 ms [13:15:00] RECOVERY - Host cp2 is UP: PING OK - Packet loss = 0%, RTA = 36.30 ms [13:15:14] RECOVERY - Host cp6 is UP: PING OK - Packet loss = 0%, RTA = 188.22 ms [13:15:16] RECOVERY - ns1 SSH on ns1 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u2 (protocol 2.0) [13:15:28] RECOVERY - cp2 SSH on cp2 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u2 (protocol 2.0) [13:15:58] RECOVERY - ns1 Auth DNS on ns1 is OK: DNS OK: 0.102 seconds response time. inside.wf returns 2001:41d0:801:2000::2200,51.68.214.246 [13:16:03] RECOVERY - cp6 SSH on cp6 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u2 (protocol 2.0) [13:16:23] RECOVERY - cp5 SSH on cp5 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u2 (protocol 2.0) [13:17:01] RECOVERY - www.theharrypotter.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'theharrypotter.wiki' will expire on Tue 26 Mar 2024 03:34:20 AM GMT +0000. [13:17:08] RECOVERY - rippaverse.wikiforge.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wikiforge.net' will expire on Wed 10 Apr 2024 07:15:27 PM GMT +0000. [13:17:33] RECOVERY - mon1 Puppet on mon1 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [13:18:05] RECOVERY - director.wikiforge.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wikiforge.net' will expire on Wed 10 Apr 2024 07:15:27 PM GMT +0000. [13:19:38] RECOVERY - www.wikiforge.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wikiforge.net' will expire on Wed 10 Apr 2024 07:15:27 PM GMT +0000. [13:24:43] RECOVERY - beaconspace.unrestrictedlorefare.com - LetsEncrypt on sslhost is OK: OK - Certificate 'beaconspace.unrestrictedlorefare.com' will expire on Wed 10 Apr 2024 08:22:51 PM GMT +0000. [13:24:48] RECOVERY - churchof0days.com - LetsEncrypt on sslhost is OK: OK - Certificate 'churchof0days.com' will expire on Sat 13 Apr 2024 06:35:48 PM GMT +0000. [13:25:45] RECOVERY - avid.wikiforge.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wikiforge.net' will expire on Wed 10 Apr 2024 07:15:27 PM GMT +0000. [13:26:19] RECOVERY - wikiforge.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wikiforge.net' will expire on Wed 10 Apr 2024 07:15:27 PM GMT +0000. [13:27:41] RECOVERY - meta.wikiforge.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wikiforge.net' will expire on Wed 10 Apr 2024 07:15:27 PM GMT +0000. [13:29:46] RECOVERY - www.harrypotter.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'harrypotter.wiki' will expire on Tue 26 Mar 2024 03:35:56 AM GMT +0000. [13:30:26] RECOVERY - dc.wikiforge.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wikiforge.net' will expire on Wed 10 Apr 2024 07:15:27 PM GMT +0000. [13:32:48] RECOVERY - harrypotter.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'harrypotter.wiki' will expire on Tue 26 Mar 2024 03:35:56 AM GMT +0000. [13:35:05] RECOVERY - dcmultiverse.wikiforge.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wikiforge.net' will expire on Wed 10 Apr 2024 07:15:27 PM GMT +0000. [13:35:15] RECOVERY - theharrypotter.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'theharrypotter.wiki' will expire on Tue 26 Mar 2024 03:34:20 AM GMT +0000. [13:37:57] RECOVERY - avid.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'avid.wiki' will expire on Tue 26 Mar 2024 03:37:23 AM GMT +0000. [13:38:04] RECOVERY - ghostmachinewiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'ghostmachinewiki.com' will expire on Tue 16 Apr 2024 01:42:03 PM GMT +0000. [13:39:13] RECOVERY - dcmultiversewiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'dcmultiversewiki.com' will expire on Tue 26 Mar 2024 03:36:42 AM GMT +0000. [13:39:44] RECOVERY - www.dcmultiversewiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'dcmultiversewiki.com' will expire on Tue 26 Mar 2024 03:36:42 AM GMT +0000. [13:50:01] [02WikiForge/ssl] 07WikiForgeSSLBot pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/WikiForge/ssl/compare/cb08f2e640a3...00f2400e290e [13:50:04] [02WikiForge/ssl] 07WikiForgeSSLBot 0300f2400 - Bot: Add SSL cert for wiki.fishonmc.net Certificate committed by agent [14:08:53] PROBLEM - mw1 APT on mw1 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [14:10:52] RECOVERY - mw1 APT on mw1 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [14:11:54] RECOVERY - mw2 APT on mw2 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [14:21:36] PROBLEM - wiki.fishonmc.net - pointing to WikiForge on sslhost is CRITICAL: DNS CRITICAL - expected 'mw-lb.wikiforge.net.' but got '2607:5300:201:3100::175d,51.68.214.246' [23:15:39] [02WikiForge/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiForge/mw-config/compare/5a831664c912...dab2e3878cce [23:15:42] [02WikiForge/mw-config] 07Universal-Omega 03dab2e38 - Fix wgImportDumpScriptCommand [23:16:38] WikiForge/mw-config - Universal-Omega the build passed. [23:38:07] !log [@jobrunner1] starting deploy of {'config': True, 'versions': ['']} to [mw1, mw2, mwdedi1, mwdedi2, jobrunner1] [23:38:14] !log [@jobrunner1] finished deploy of {'config': True, 'versions': ['']} to [mw1, mw2, mwdedi1, mwdedi2, jobrunner1] - SUCCESS in 6s [23:39:07] oh sweet, autodeploy works [23:42:32] !log [universalomega@jobrunner1] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/importDump.php --wiki=ghostmachinewiki --no-updates /home/universalomega/ghostmachinewiki-20240121230234.xml (START) [23:42:33] !log [universalomega@jobrunner1] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/importDump.php --wiki=ghostmachinewiki --no-updates /home/universalomega/ghostmachinewiki-20240121230234.xml (END - exit=256) [23:42:34] !log [universalomega@jobrunner1] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/rebuildall.php --wiki=ghostmachinewiki (START) [23:42:35] !log [universalomega@jobrunner1] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/rebuildall.php --wiki=ghostmachinewiki (END - exit=0) [23:42:36] !log [universalomega@jobrunner1] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/initSiteStats.php --wiki=ghostmachinewiki --active --update (END - exit=0) [23:42:57] well that failed... [23:45:32] [1/3] universalomega@jobrunner1:~$ . /etc/swift-env.sh [23:45:33] [2/3] -bash: /etc/swift-env.sh: No such file or directory [23:45:33] [3/3] @agentisai [23:46:06] ah