[01:06:57] [02mw-config] 07Universal-Omega closed pull request 03#32: Remove MatomoAnalytics - 13https://github.com/WikiTideOrg/mw-config/pull/32 [01:06:59] [02WikiTideOrg/mw-config] 07Universal-Omega pushed 031 commit to 03main [+0/-0/±1] 13https://github.com/WikiTideOrg/mw-config/compare/ad35e0a3060e...747b6b97017f [01:07:01] [02WikiTideOrg/mw-config] 07Universal-Omega 03747b6b9 - Remove MatomoAnalytics (#32) [01:07:02] [02WikiTideOrg/mw-config] 07Universal-Omega deleted branch 03remove-MatomoAnalytics [01:07:05] [02mw-config] 07Universal-Omega deleted branch 03remove-MatomoAnalytics - 13https://github.com/WikiTideOrg/mw-config [01:07:59] WikiTideOrg/mw-config - Universal-Omega the build passed. [01:11:17] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03decom [+0/-1/±0] 13https://github.com/WikiTideOrg/puppet/commit/51ef775f2e1a [01:11:18] [02WikiTideOrg/puppet] 07Universal-Omega 0351ef775 - Delete hieradata/hosts/bast1.yaml [01:11:21] [02puppet] 07Universal-Omega created branch 03decom - 13https://github.com/WikiTideOrg/puppet [01:11:31] [02puppet] 07Universal-Omega opened pull request 03#49: Decom servers - 13https://github.com/WikiTideOrg/puppet/pull/49 [01:11:49] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03decom [+0/-1/±0] 13https://github.com/WikiTideOrg/puppet/compare/51ef775f2e1a...49c71ea5419a [01:11:52] [02WikiTideOrg/puppet] 07Universal-Omega 0349c71ea - Delete hieradata/hosts/bots1.yaml [01:11:53] [02puppet] 07Universal-Omega synchronize pull request 03#49: Decom servers - 13https://github.com/WikiTideOrg/puppet/pull/49 [01:12:03] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03decom [+0/-1/±0] 13https://github.com/WikiTideOrg/puppet/compare/49c71ea5419a...9c75df415142 [01:12:04] [02WikiTideOrg/puppet] 07Universal-Omega 039c75df4 - Delete hieradata/hosts/cp1.yaml [01:12:06] [02puppet] 07Universal-Omega synchronize pull request 03#49: Decom servers - 13https://github.com/WikiTideOrg/puppet/pull/49 [01:12:13] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03decom [+0/-1/±0] 13https://github.com/WikiTideOrg/puppet/compare/9c75df415142...21be59a1f14e [01:12:16] [02WikiTideOrg/puppet] 07Universal-Omega 0321be59a - Delete hieradata/hosts/cp2.yaml [01:12:17] [02puppet] 07Universal-Omega synchronize pull request 03#49: Decom servers - 13https://github.com/WikiTideOrg/puppet/pull/49 [01:12:22] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03decom [+0/-1/±0] 13https://github.com/WikiTideOrg/puppet/compare/21be59a1f14e...9309ff20ae18 [01:12:25] [02WikiTideOrg/puppet] 07Universal-Omega 039309ff2 - Delete hieradata/hosts/cp3.yaml [01:12:28] [02puppet] 07Universal-Omega synchronize pull request 03#49: Decom servers - 13https://github.com/WikiTideOrg/puppet/pull/49 [01:12:35] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03decom [+0/-1/±0] 13https://github.com/WikiTideOrg/puppet/compare/9309ff20ae18...98d2a7ba4dc4 [01:12:37] [02WikiTideOrg/puppet] 07Universal-Omega 0398d2a7b - Delete hieradata/hosts/graylog1.yaml [01:12:39] [02puppet] 07Universal-Omega synchronize pull request 03#49: Decom servers - 13https://github.com/WikiTideOrg/puppet/pull/49 [01:12:49] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03decom [+0/-1/±0] 13https://github.com/WikiTideOrg/puppet/compare/98d2a7ba4dc4...03b9094cb5c0 [01:12:52] [02WikiTideOrg/puppet] 07Universal-Omega 0303b9094 - Delete hieradata/hosts/ldap1.yaml [01:12:55] [02puppet] 07Universal-Omega synchronize pull request 03#49: Decom servers - 13https://github.com/WikiTideOrg/puppet/pull/49 [01:12:59] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03decom [+0/-1/±0] 13https://github.com/WikiTideOrg/puppet/compare/03b9094cb5c0...a9b3f94ad4ad [01:13:01] [02WikiTideOrg/puppet] 07Universal-Omega 03a9b3f94 - Delete hieradata/hosts/matomo1.yaml [01:13:02] [02puppet] 07Universal-Omega synchronize pull request 03#49: Decom servers - 13https://github.com/WikiTideOrg/puppet/pull/49 [01:13:16] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03decom [+0/-1/±0] 13https://github.com/WikiTideOrg/puppet/compare/a9b3f94ad4ad...25331c9656ed [01:13:18] [02WikiTideOrg/puppet] 07Universal-Omega 0325331c9 - Delete hieradata/hosts/mon1.yaml [01:13:21] [02puppet] 07Universal-Omega synchronize pull request 03#49: Decom servers - 13https://github.com/WikiTideOrg/puppet/pull/49 [01:13:26] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03decom [+0/-1/±0] 13https://github.com/WikiTideOrg/puppet/compare/25331c9656ed...44fc47c22403 [01:13:27] [02WikiTideOrg/puppet] 07Universal-Omega 0344fc47c - Delete hieradata/hosts/os1.yaml [01:13:30] [02puppet] 07Universal-Omega synchronize pull request 03#49: Decom servers - 13https://github.com/WikiTideOrg/puppet/pull/49 [01:13:37] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03decom [+0/-1/±0] 13https://github.com/WikiTideOrg/puppet/compare/44fc47c22403...178ca7f86dc0 [01:13:40] [02WikiTideOrg/puppet] 07Universal-Omega 03178ca7f - Delete hieradata/hosts/phorge1.yaml [01:13:41] [02puppet] 07Universal-Omega synchronize pull request 03#49: Decom servers - 13https://github.com/WikiTideOrg/puppet/pull/49 [01:13:47] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03decom [+0/-1/±0] 13https://github.com/WikiTideOrg/puppet/compare/178ca7f86dc0...af28006cb872 [01:13:50] [02WikiTideOrg/puppet] 07Universal-Omega 03af28006 - Delete hieradata/hosts/prometheus1.yaml [01:13:51] [02puppet] 07Universal-Omega synchronize pull request 03#49: Decom servers - 13https://github.com/WikiTideOrg/puppet/pull/49 [01:14:01] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03decom [+0/-1/±0] 13https://github.com/WikiTideOrg/puppet/compare/af28006cb872...15e0ed1e9f3f [01:14:03] [02WikiTideOrg/puppet] 07Universal-Omega 0315e0ed1 - Delete hieradata/hosts/reports1.yaml [01:14:04] [02puppet] 07Universal-Omega synchronize pull request 03#49: Decom servers - 13https://github.com/WikiTideOrg/puppet/pull/49 [01:14:13] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03decom [+0/-1/±0] 13https://github.com/WikiTideOrg/puppet/compare/15e0ed1e9f3f...6cefcf463a88 [01:14:14] [02WikiTideOrg/puppet] 07Universal-Omega 036cefcf4 - Delete hieradata/hosts/services1.yaml [01:14:17] [02puppet] 07Universal-Omega synchronize pull request 03#49: Decom servers - 13https://github.com/WikiTideOrg/puppet/pull/49 [01:14:28] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03decom [+0/-1/±0] 13https://github.com/WikiTideOrg/puppet/compare/6cefcf463a88...1b784c3733f8 [01:14:31] [02WikiTideOrg/puppet] 07Universal-Omega 031b784c3 - Delete hieradata/hosts/test1.yaml [01:14:33] [02puppet] 07Universal-Omega synchronize pull request 03#49: Decom servers - 13https://github.com/WikiTideOrg/puppet/pull/49 [01:16:46] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03decom [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/1b784c3733f8...f0bce4d78c45 [01:16:48] [02WikiTideOrg/puppet] 07Universal-Omega 03f0bce4d - Update common.yaml [01:16:51] [02puppet] 07Universal-Omega synchronize pull request 03#49: Decom servers - 13https://github.com/WikiTideOrg/puppet/pull/49 [01:17:09] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03decom [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/f0bce4d78c45...3aa2dc88dbf1 [01:17:10] [02WikiTideOrg/puppet] 07Universal-Omega 033aa2dc8 - Update common.yaml [01:17:13] [02puppet] 07Universal-Omega synchronize pull request 03#49: Decom servers - 13https://github.com/WikiTideOrg/puppet/pull/49 [01:49:26] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03decom [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/3aa2dc88dbf1...4520d23486fd [01:49:28] [02WikiTideOrg/puppet] 07Universal-Omega 034520d23 - Update site.pp [01:49:31] [02puppet] 07Universal-Omega synchronize pull request 03#49: Decom servers - 13https://github.com/WikiTideOrg/puppet/pull/49 [02:18:21] PROBLEM - puppet1 APT on puppet1 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [02:20:23] PROBLEM - puppet1 APT on puppet1 is CRITICAL: APT CRITICAL: 42 packages available for upgrade (12 critical updates). [02:52:04] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03decom [+1/-0/±0] 13https://github.com/WikiTideOrg/puppet/compare/4520d23486fd...677495eb3195 [02:52:05] [02WikiTideOrg/puppet] 07Universal-Omega 03677495e - Create cp1.yaml [02:52:07] [02puppet] 07Universal-Omega synchronize pull request 03#49: Decom servers - 13https://github.com/WikiTideOrg/puppet/pull/49 [02:52:23] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03decom [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/677495eb3195...08c657e8eb6b [02:52:25] [02WikiTideOrg/puppet] 07Universal-Omega 0308c657e - Update site.pp [02:52:27] [02puppet] 07Universal-Omega synchronize pull request 03#49: Decom servers - 13https://github.com/WikiTideOrg/puppet/pull/49 [03:43:20] PROBLEM - bots1 IRC-Discord Relay Bot on bots1 is CRITICAL: PROCS CRITICAL: 0 processes with args 'relaybot' [04:15:01] PROBLEM - puppet1 Backups SSLKeys on puppet1 is CRITICAL: FILE_AGE CRITICAL: File not found - /var/log/sslkeys-backup.log [04:15:09] PROBLEM - puppet1 Backups Private on puppet1 is CRITICAL: FILE_AGE CRITICAL: File not found - /var/log/private-backup.log [04:17:07] [02puppet] 07Universal-Omega closed pull request 03#45: mwdeploy: support using -private in rsync command - 13https://github.com/WikiTideOrg/puppet/pull/45 [04:17:09] [02WikiTideOrg/puppet] 07Universal-Omega deleted branch 03mwdeploy-private-rsync [04:17:12] [02puppet] 07Universal-Omega deleted branch 03mwdeploy-private-rsync - 13https://github.com/WikiTideOrg/puppet [04:17:20] [02puppet] 07Universal-Omega closed pull request 03#39: Add raid status checks - 13https://github.com/WikiTideOrg/puppet/pull/39 [04:17:23] [02WikiTideOrg/puppet] 07Universal-Omega deleted branch 03raid [04:17:25] [02puppet] 07Universal-Omega deleted branch 03raid - 13https://github.com/WikiTideOrg/puppet [04:17:34] [02puppet] 07Universal-Omega closed pull request 03#23: Add thumbor and haproxy support - 13https://github.com/WikiTideOrg/puppet/pull/23 [04:17:36] [02WikiTideOrg/puppet] 07Universal-Omega deleted branch 03thumbor [04:17:39] [02puppet] 07Universal-Omega deleted branch 03thumbor - 13https://github.com/WikiTideOrg/puppet [04:17:44] [02puppet] 07Universal-Omega closed pull request 03#21: cloud: add monitoring - 13https://github.com/WikiTideOrg/puppet/pull/21 [04:17:47] [02WikiTideOrg/puppet] 07Universal-Omega deleted branch 03cloud-monitoring [04:17:50] [02puppet] 07Universal-Omega deleted branch 03cloud-monitoring - 13https://github.com/WikiTideOrg/puppet [04:19:42] PROBLEM - bots1 NTP time on bots1 is CRITICAL: connect to address 10.0.0.123 port 5666: No route to hostconnect to host 10.0.0.123 port 5666: No route to host [04:20:08] PROBLEM - Host bots1 is DOWN: CRITICAL - Host Unreachable (10.0.0.123) [04:27:47] PROBLEM - puppet1 HTTPS on puppet1 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 7 - Failed to connect to puppet1.wikitide.net port 443 after 0 ms: Couldn't connect to server [04:28:16] PROBLEM - puppet1 puppetdb on puppet1 is CRITICAL: connect to address 10.0.0.100 and port 8081: Connection refused [04:28:19] PROBLEM - puppet1 conntrack_table_size on puppet1 is CRITICAL: connect to address 10.0.0.100 port 5666: Connection refusedconnect to host 10.0.0.100 port 5666: Connection refused [04:28:45] PROBLEM - puppet1 PowerDNS Recursor on puppet1 is CRITICAL: connect to address 10.0.0.100 port 5666: Connection refusedconnect to host 10.0.0.100 port 5666: Connection refused [04:29:02] PROBLEM - puppet1 SSH on puppet1 is CRITICAL: connect to address 10.0.0.100 and port 22: Connection refused [04:29:45] PROBLEM - puppet1 NTP time on puppet1 is CRITICAL: connect to address 10.0.0.100 port 5666: Connection refusedconnect to host 10.0.0.100 port 5666: Connection refused [04:29:49] PROBLEM - puppet1 puppetserver on puppet1 is CRITICAL: connect to address 10.0.0.100 and port 8140: Connection refused [04:29:49] PROBLEM - puppet1 ferm_active on puppet1 is CRITICAL: connect to address 10.0.0.100 port 5666: Connection refusedconnect to host 10.0.0.100 port 5666: Connection refused [04:29:49] PROBLEM - puppet1 Current Load on puppet1 is CRITICAL: connect to address 10.0.0.100 port 5666: Connection refusedconnect to host 10.0.0.100 port 5666: Connection refused [04:29:53] [02mw-config] 07Universal-Omega closed pull request 03#24: Update Wikibase.php for lhmnwiki/lhmndatawiki - 13https://github.com/WikiTideOrg/mw-config/pull/24 [04:29:56] [02mw-config] 07Universal-Omega commented on pull request 03#24: Update Wikibase.php for lhmnwiki/lhmndatawiki - 13https://github.com/WikiTideOrg/mw-config/pull/24#issuecomment-1933346608 [04:32:45] RECOVERY - puppet1 PowerDNS Recursor on puppet1 is OK: DNS OK: 0.386 seconds response time. wikitide.org returns 146.59.44.171,2001:41d0:601:1100::4062 [04:33:02] RECOVERY - puppet1 SSH on puppet1 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u2 (protocol 2.0) [04:33:46] RECOVERY - puppet1 NTP time on puppet1 is OK: NTP OK: Offset 0.00181645155 secs [04:33:49] RECOVERY - puppet1 puppetserver on puppet1 is OK: TCP OK - 0.000 second response time on 10.0.0.100 port 8140 [04:33:49] RECOVERY - puppet1 ferm_active on puppet1 is OK: OK ferm input default policy is set [04:33:49] RECOVERY - puppet1 Current Load on puppet1 is OK: LOAD OK - total load average: 2.65, 1.02, 0.37 [04:34:18] RECOVERY - puppet1 puppetdb on puppet1 is OK: TCP OK - 0.001 second response time on 10.0.0.100 port 8081 [04:34:19] RECOVERY - puppet1 conntrack_table_size on puppet1 is OK: OK: nf_conntrack is 0 % full [04:35:33] RECOVERY - mem1 Puppet on mem1 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [04:35:55] RECOVERY - graylog1 Puppet on graylog1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:37:03] RECOVERY - services1 Puppet on services1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:37:11] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [04:40:06] RECOVERY - cp4 Puppet on cp4 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [04:41:52] RECOVERY - prometheus1 Puppet on prometheus1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:44:30] RECOVERY - mon1 Puppet on mon1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:44:52] [02puppet] 07Universal-Omega closed pull request 03#49: Decom servers - 13https://github.com/WikiTideOrg/puppet/pull/49 [04:44:54] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-14/±2] 13https://github.com/WikiTideOrg/puppet/compare/55e2db5fb24e...e204343d5f03 [04:44:56] [02WikiTideOrg/puppet] 07Universal-Omega 03e204343 - Decom servers [04:44:59] [02WikiTideOrg/puppet] 07Universal-Omega deleted branch 03decom [04:45:00] [02puppet] 07Universal-Omega deleted branch 03decom - 13https://github.com/WikiTideOrg/puppet [04:45:25] RECOVERY - swiftobject1 Puppet on swiftobject1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:47:15] RECOVERY - jobchron1 Puppet on jobchron1 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [04:48:09] RECOVERY - swiftac1 Puppet on swiftac1 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [04:48:09] RECOVERY - swiftproxy1 Puppet on swiftproxy1 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [04:48:42] PROBLEM - puppet1 puppetdb on puppet1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:48:51] RECOVERY - db1 Puppet on db1 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [04:48:55] PROBLEM - mw1 MediaWiki Rendering on mw1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:49:09] RECOVERY - jobrunner1 Puppet on jobrunner1 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [04:49:17] PROBLEM - jobrunner1 MediaWiki Rendering on jobrunner1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:49:27] PROBLEM - puppet1 SSH on puppet1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:49:28] PROBLEM - mw2 MediaWiki Rendering on mw2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:49:48] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:50:17] PROBLEM - cp6 Nginx Backend for matomo1 on cp6 is CRITICAL: connect to address localhost and port 8204: Connection refused [04:50:19] PROBLEM - db1 MariaDB Connections on db1 is UNKNOWN: PHP Fatal error: Uncaught mysqli_sql_exception: Connection timed out in /usr/lib/nagios/plugins/check_mysql_connections.php:47Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(47): mysqli_real_connect(Object(mysqli), 'db1.wikitide.ne...', 'icinga', Object(SensitiveParameterValue), NULL, NULL, NULL, false)#1 {main} thrown in /usr/lib/nagios/plugins/check_mysql_c [04:50:19] line 47Fatal error: Uncaught mysqli_sql_exception: Connection timed out in /usr/lib/nagios/plugins/check_mysql_connections.php:47Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(47): mysqli_real_connect(Object(mysqli), 'db1.wikitide.ne...', 'icinga', Object(SensitiveParameterValue), NULL, NULL, NULL, false)#1 {main} thrown in /usr/lib/nagios/plugins/check_mysql_connections.php on line 47 [04:50:20] PROBLEM - jobchron1 SSH on jobchron1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:50:23] PROBLEM - cp6 Nginx Backend for phorge1 on cp6 is CRITICAL: connect to address localhost and port 8200: Connection refused [04:50:37] PROBLEM - swiftproxy1 SSH on swiftproxy1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:50:41] PROBLEM - swiftproxy1 memcached on swiftproxy1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:50:43] PROBLEM - cp1 Nginx Backend for matomo1 on cp1 is CRITICAL: connect to address localhost and port 8204: Connection refused [04:50:44] PROBLEM - cp6 Nginx Backend for test1 on cp6 is CRITICAL: connect to address localhost and port 8180: Connection refused [04:50:49] PROBLEM - swiftac1 SSH on swiftac1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:50:53] PROBLEM - swiftobject1 SSH on swiftobject1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:50:56] PROBLEM - mem1 memcached on mem1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:51:16] PROBLEM - swiftac1 Swift Account Service on swiftac1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:51:19] PROBLEM - swiftproxy1 Swift Proxy on swiftproxy1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:51:28] PROBLEM - jobrunner1 SSH on jobrunner1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:51:31] PROBLEM - mw1 HTTPS on mw1 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Connection timed out after 10004 milliseconds [04:51:32] PROBLEM - mw2 SSH on mw2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:51:36] PROBLEM - cp1 Nginx Backend for test1 on cp1 is CRITICAL: connect to address localhost and port 8180: Connection refused [04:51:37] PROBLEM - cp1 SSH on cp1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:51:46] PROBLEM - cp1 Nginx Backend for phorge1 on cp1 is CRITICAL: connect to address localhost and port 8200: Connection refused [04:51:57] PROBLEM - cp6 SSH on cp6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:51:59] PROBLEM - swiftac1 Swift Container Service on swiftac1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:52:02] PROBLEM - db1 SSH on db1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:52:05] PROBLEM - swiftobject1 Swift Object Service on swiftobject1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:52:05] PROBLEM - mem1 SSH on mem1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:52:09] PROBLEM - mw1 SSH on mw1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:52:25] PROBLEM - mw2 HTTPS on mw2 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Connection timed out after 10004 milliseconds [04:52:28] PROBLEM - jobrunner1 HTTPS on jobrunner1 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Connection timed out after 10003 milliseconds [04:53:07] PROBLEM - cp5 Nginx Backend for matomo1 on cp5 is CRITICAL: connect to address localhost and port 8204: Connection refused [04:53:12] PROBLEM - cp4 Nginx Backend for matomo1 on cp4 is CRITICAL: connect to address localhost and port 8204: Connection refused [04:53:12] PROBLEM - cp4 Nginx Backend for phorge1 on cp4 is CRITICAL: connect to address localhost and port 8200: Connection refused [04:53:20] PROBLEM - db1 MariaDB on db1 is UNKNOWN: [04:53:20] PROBLEM - cp4 SSH on cp4 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:53:23] PROBLEM - cp5 SSH on cp5 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:53:44] PROBLEM - cp4 Puppet on cp4 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[load-new-vcl-file] [04:53:45] PROBLEM - cp4 Nginx Backend for test1 on cp4 is CRITICAL: connect to address localhost and port 8180: Connection refused [04:53:58] PROBLEM - cp5 Nginx Backend for test1 on cp5 is CRITICAL: connect to address localhost and port 8180: Connection refused [04:54:10] PROBLEM - cp5 Nginx Backend for phorge1 on cp5 is CRITICAL: connect to address localhost and port 8200: Connection refused [04:55:47] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/e204343d5f03...9c8eb953ddac [04:55:48] [02WikiTideOrg/puppet] 07Universal-Omega 039c8eb95 - Don't always use bastion public ips [05:02:03] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/9c8eb953ddac...e7c3ad656722 [05:02:04] [02WikiTideOrg/puppet] 07Universal-Omega 03e7c3ad6 - Revert [05:07:51] RECOVERY - puppet1 puppetdb on puppet1 is OK: TCP OK - 0.000 second response time on 10.0.0.100 port 8081 [05:08:15] RECOVERY - puppet1 SSH on puppet1 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u2 (protocol 2.0) [05:18:02] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/e7c3ad656722...04706617e34b [05:18:05] [02WikiTideOrg/puppet] 07Universal-Omega 030470661 - Add cloud1 [05:19:46] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/04706617e34b...96cddc240d4f [05:19:47] [02WikiTideOrg/puppet] 07Universal-Omega 0396cddc2 - -syslog [05:23:36] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/96cddc240d4f...28c23a43547c [05:23:56] [02WikiTideOrg/puppet] 07Universal-Omega 0328c23a4 - Add 10.0.0.1 [05:57:44] [02WikiTideOrg/mw-config] 07Universal-Omega pushed 031 commit to 03main [+0/-0/±1] 13https://github.com/WikiTideOrg/mw-config/compare/747b6b97017f...db17c24fb46e [05:57:46] [02WikiTideOrg/mw-config] 07Universal-Omega 03db17c24 - Remove IRC feed [05:58:40] WikiTideOrg/mw-config - Universal-Omega the build passed.