[04:27:49] [02WikiForge/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiForge/puppet/compare/c88e8e4a13c6...6450efdaf6a7 [04:27:52] [02WikiForge/puppet] 07Universal-Omega 036450efd - matomo: use systemd::timer::job [04:28:26] [02WikiForge/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-1/±0] 13https://github.com/WikiForge/puppet/compare/6450efdaf6a7...be75c1c9039b [04:28:29] [02WikiForge/puppet] 07Universal-Omega 03be75c1c - Remove runMatomoArchive [04:31:45] [02WikiForge/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiForge/puppet/compare/be75c1c9039b...4608826defa9 [04:31:47] [02WikiForge/puppet] 07Universal-Omega 034608826 - No IPv6 [04:32:54] WikiForge/puppet - Universal-Omega the build has errored. [04:39:44] [02WikiForge/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiForge/puppet/compare/4608826defa9...5180916b7bf2 [04:39:45] [02WikiForge/puppet] 07Universal-Omega 035180916 - Add to monitoring [05:10:52] RECOVERY - www.dcmultiversewiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'dcmultiversewiki.com' will expire on Sat 03 Feb 2024 09:45:36 PM GMT +0000. [05:11:03] RECOVERY - dcmultiversewiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'dcmultiversewiki.com' will expire on Mon 12 Feb 2024 07:02:02 PM GMT +0000. [05:11:05] PROBLEM - www.mwcosmos.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wikitide.org' expires in 14 day(s) (Mon 04 Dec 2023 02:58:50 PM GMT +0000). [05:11:46] PROBLEM - mw2 Puppet on mw2 is UNKNOWN: NRPE: Unable to read output [05:11:48] PROBLEM - phorge1 APT on phorge1 is CRITICAL: APT CRITICAL: 1 packages available for upgrade (1 critical updates). [05:11:50] PROBLEM - mon1 Puppet on mon1 is UNKNOWN: NRPE: Unable to read output [05:11:57] PROBLEM - mw3 MediaWiki Rendering on mw3 is UNKNOWN: HTTP UNKNOWN: Failed to unchunk message body [05:11:57] PROBLEM - mon1 APT on mon1 is CRITICAL: APT CRITICAL: 21 packages available for upgrade (3 critical updates). [05:12:00] PROBLEM - jobrunner1 MediaWiki Rendering on jobrunner1 is UNKNOWN: HTTP UNKNOWN: Failed to unchunk message body [05:12:00] PROBLEM - ldap1 APT on ldap1 is CRITICAL: APT CRITICAL: 4 packages available for upgrade (3 critical updates). [05:12:02] PROBLEM - mon1 grafana.inside.wf HTTPS on mon1 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 295 bytes in 0.007 second response time [05:12:05] PROBLEM - mw2 MediaWiki Rendering on mw2 is UNKNOWN: HTTP UNKNOWN: Failed to unchunk message body [05:12:12] PROBLEM - wikiforge.work - pointing to WF/WT on sslhost is CRITICAL: DNS CRITICAL - expected 'mw-lb.wikiforge.net.' but got '104.21.36.88,172.67.190.178,2606:4700:3036::6815:2458,2606:4700:3036::ac43:beb2' [05:12:15] There we go, I only had to delete db1 from icinga lol, now to figure out why it borked everything... [05:12:21] PROBLEM - jobrunner1 APT on jobrunner1 is CRITICAL: APT CRITICAL: 48 packages available for upgrade (15 critical updates). [05:12:24] PROBLEM - mon1 icinga.inside.wf HTTPS on mon1 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 295 bytes in 0.010 second response time [05:12:27] PROBLEM - puppet1 puppetdb on puppet1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:12:31] PROBLEM - jobrunner1 Puppet on jobrunner1 is UNKNOWN: NRPE: Unable to read output [05:12:32] PROBLEM - os1 APT on os1 is CRITICAL: APT CRITICAL: 8 packages available for upgrade (7 critical updates). [05:12:33] PROBLEM - mw1 MediaWiki Rendering on mw1 is UNKNOWN: HTTP UNKNOWN: Failed to unchunk message body [05:12:34] PROBLEM - graylog1 APT on graylog1 is CRITICAL: APT CRITICAL: 15 packages available for upgrade (7 critical updates). [05:12:35] PROBLEM - mw1 Puppet on mw1 is UNKNOWN: NRPE: Unable to read output [05:13:06] PROBLEM - Host mw3 is DOWN: PING CRITICAL - Packet loss = 100% [05:13:07] PROBLEM - puppet1 APT on puppet1 is CRITICAL: APT CRITICAL: 18 packages available for upgrade (6 critical updates). [05:13:14] PROBLEM - cloud1 APT on cloud1 is CRITICAL: APT CRITICAL: 39 packages available for upgrade (10 critical updates). [05:13:14] PROBLEM - Host mail1 is DOWN: PING CRITICAL - Packet loss = 100% [05:13:15] PROBLEM - jobrunner11 APT on jobrunner11 is CRITICAL: APT CRITICAL: 48 packages available for upgrade (15 critical updates). [05:17:40] RECOVERY - mon1 Puppet on mon1 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [05:18:55] PROBLEM - mon1 Check correctness of the icinga configuration on mon1 is CRITICAL: Icinga configuration contains errors [05:19:29] [02WikiForge/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiForge/puppet/compare/5180916b7bf2...105b02cb9921 [05:19:32] [02WikiForge/puppet] 07Universal-Omega 03105b02c - Use monitoring [05:22:12] [02WikiForge/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiForge/puppet/compare/105b02cb9921...8b92aa5778fe [05:22:14] [02WikiForge/puppet] 07Universal-Omega 038b92aa5 - Add icinga-wikiforge to relay ignore [06:49:08] !log [universalomega@puppet1] Upgraded packages libssl3, libpq5, openjdk-17-jdk, openjdk-17-jdk-headless, openjdk-17-jre, openjdk-17-jre-headless, and openssl on graylog1 [06:50:04] RECOVERY - graylog1 APT on graylog1 is OK: APT OK: 8 packages available for upgrade (0 critical updates). [06:56:26] PROBLEM - jobrunner1 MediaWiki Rendering on jobrunner1 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 1867 bytes in 0.022 second response time [06:56:28] RECOVERY - jobrunner11 APT on jobrunner11 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [06:57:20] RECOVERY - jobrunner1 APT on jobrunner1 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [06:58:26] PROBLEM - jobrunner1 MediaWiki Rendering on jobrunner1 is UNKNOWN: HTTP UNKNOWN: Failed to unchunk message body [07:25:45] [02WikiForge/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiForge/puppet/compare/8b92aa5778fe...9ae2c9124789 [07:25:47] [02WikiForge/puppet] 07Universal-Omega 039ae2c91 - Fix hostname [07:27:04] [02WikiForge/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiForge/puppet/compare/9ae2c9124789...f75c5fa3287e [07:27:06] [02WikiForge/puppet] 07Universal-Omega 03f75c5fa - Fix certs [16:55:11] [02WikiForge/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiForge/puppet/compare/f75c5fa3287e...437575e40108 [16:55:14] [02WikiForge/puppet] 07Universal-Omega 03437575e - Use exec directly to avoid running every time [16:59:23] [02WikiForge/puppet] 07Universal-Omega pushed 031 commit to 03absent-ircrcbot [+0/-0/±1] 13https://github.com/WikiForge/puppet/commit/0d5da93cc4fe [16:59:25] [02WikiForge/puppet] 07Universal-Omega 030d5da93 - Absent ircrcbot [16:59:27] [02puppet] 07Universal-Omega created branch 03absent-ircrcbot - 13https://github.com/WikiForge/puppet [16:59:28] [02puppet] 07Universal-Omega opened pull request 03#203: Absent ircrcbot - 13https://github.com/WikiForge/puppet/pull/203 [17:00:11] [02WikiForge/puppet] 07github-actions[bot] pushed 031 commit to 03absent-ircrcbot [+0/-0/±1] 13https://github.com/WikiForge/puppet/compare/0d5da93cc4fe...3a82fdb8e8b7 [17:00:13] [02WikiForge/puppet] 07github-actions 033a82fdb - CI: lint puppet code to standards Check commit and GitHub actions for more details [17:00:16] [02puppet] 07github-actions[bot] synchronize pull request 03#203: Absent ircrcbot - 13https://github.com/WikiForge/puppet/pull/203 [17:03:32] [02puppet] 07Universal-Omega closed pull request 03#203: Absent ircrcbot - 13https://github.com/WikiForge/puppet/pull/203 [17:03:34] [02WikiForge/puppet] 07Universal-Omega deleted branch 03absent-ircrcbot [17:03:37] [02WikiForge/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/WikiForge/puppet/compare/437575e40108...a72c5850ea7e [17:03:39] [02WikiForge/puppet] 07Universal-Omega 03a72c585 - Absent ircrcbot [17:03:42] [02puppet] 07Universal-Omega deleted branch 03absent-ircrcbot - 13https://github.com/WikiForge/puppet [17:06:06] [02WikiForge/puppet] 07Universal-Omega pushed 031 commit to 03remove-ircrcbot [+0/-0/±1] 13https://github.com/WikiForge/puppet/commit/610a61d228f2 [17:06:08] [02WikiForge/puppet] 07Universal-Omega 03610a61d - Remove ircrcbot [17:06:10] [02puppet] 07Universal-Omega created branch 03remove-ircrcbot - 13https://github.com/WikiForge/puppet [17:06:14] [02puppet] 07Universal-Omega opened pull request 03#204: Remove ircrcbot - 13https://github.com/WikiForge/puppet/pull/204 [17:06:52] [02WikiForge/puppet] 07Universal-Omega pushed 031 commit to 03remove-ircrcbot [+0/-1/±0] 13https://github.com/WikiForge/puppet/compare/610a61d228f2...f59677acc875 [17:06:55] [02WikiForge/puppet] 07Universal-Omega 03f59677a - Remove [17:06:56] [02puppet] 07Universal-Omega synchronize pull request 03#204: Remove ircrcbot - 13https://github.com/WikiForge/puppet/pull/204 [17:08:22] [02WikiForge/puppet] 07Universal-Omega pushed 031 commit to 03remove-ircrcbot [+0/-1/±0] 13https://github.com/WikiForge/puppet/compare/f59677acc875...08b9ef915b5a [17:08:23] [02WikiForge/puppet] 07Universal-Omega 0308b9ef9 - Remove [17:08:24] [02puppet] 07Universal-Omega synchronize pull request 03#204: Remove ircrcbot - 13https://github.com/WikiForge/puppet/pull/204 [17:15:27] [02WikiForge/puppet] 07Universal-Omega pushed 031 commit to 03remove-ircrcbot [+1/-1/±0] 13https://github.com/WikiForge/puppet/compare/08b9ef915b5a...3e6748964a3f [17:15:28] [02WikiForge/puppet] 07Universal-Omega 033e67489 - Rename [17:15:31] [02puppet] 07Universal-Omega synchronize pull request 03#204: Remove ircrcbot - 13https://github.com/WikiForge/puppet/pull/204 [17:16:06] [02WikiForge/puppet] 07Universal-Omega pushed 031 commit to 03remove-ircrcbot [+0/-0/±1] 13https://github.com/WikiForge/puppet/compare/3e6748964a3f...0cc557310a9d [17:16:07] [02WikiForge/puppet] 07Universal-Omega 030cc5573 - Update template [17:16:10] [02puppet] 07Universal-Omega synchronize pull request 03#204: Remove ircrcbot - 13https://github.com/WikiForge/puppet/pull/204 [17:17:30] WikiForge/puppet - Universal-Omega the build has errored. [17:19:07] WikiForge/puppet - Universal-Omega the build has errored. [19:50:14] RECOVERY - mon1 Check correctness of the icinga configuration on mon1 is OK: Icinga configuration is correct [19:50:26] PROBLEM - db1 MariaDB Connections on db1 is UNKNOWN: PHP Fatal error: Uncaught mysqli_sql_exception: Access denied for user 'icinga'@'10.0.0.110' (using password: YES) in /usr/lib/nagios/plugins/check_mysql_connections.php:66Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(66): mysqli_real_connect(Object(mysqli), 'db1.inside.wf', 'icinga', Object(SensitiveParameterValue), NULL, NULL, NULL, true)#1 {main} thrown in /u [19:50:26] s/plugins/check_mysql_connections.php on line 66Fatal error: Uncaught mysqli_sql_exception: Access denied for user 'icinga'@'10.0.0.110' (using password: YES) in /usr/lib/nagios/plugins/check_mysql_connections.php:66Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(66): mysqli_real_connect(Object(mysqli), 'db1.inside.wf', 'icinga', Object(SensitiveParameterValue), NULL, NULL, NULL, true)#1 {main} thrown in /usr/lib/na [19:50:26] /check_mysql_connections.php on line 66 [19:51:02] PROBLEM - db1 MariaDB on db1 is CRITICAL: Access denied for user 'icinga'@'10.0.0.110' (using password: YES) [19:51:14] PROBLEM - db1 Backups SQL on db1 is CRITICAL: FILE_AGE CRITICAL: File not found - /var/log/backup-logs/sql-backup.log [20:49:58] PROBLEM - mon1 Check correctness of the icinga configuration on mon1 is CRITICAL: Icinga configuration contains errors