[00:00:23] <icinga-wikiforge>	 RECOVERY - mw2 MediaWiki Rendering on mw2 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.261 second response time
[00:00:43] <icinga-wikiforge>	 RECOVERY - jobrunner1 MediaWiki Rendering on jobrunner1 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 3.515 second response time
[00:01:25] <icinga-wikiforge>	 RECOVERY - mw1 MediaWiki Rendering on mw1 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.163 second response time
[01:20:13] <botifico-49a823b>	 [02WikiForge/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiForge/puppet/compare/57ca5ac7840b...c53f949bce17
[01:20:14] <botifico-49a823b>	 [02WikiForge/puppet] 07Universal-Omega 03c53f949 - Fix
[01:21:27] <botifico-49a823b>	 [02puppet] 07Universal-Omega closed pull request 03#211: phorge: remove support-archive - 13https://github.com/WikiForge/puppet/pull/211
[01:21:29] <botifico-49a823b>	 [02WikiForge/puppet] 07Universal-Omega deleted branch 03remove-support-archive
[01:21:32] <botifico-49a823b>	 [02WikiForge/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-3/±4] 13https://github.com/WikiForge/puppet/compare/c53f949bce17...9241f111e46c
[01:21:35] <botifico-49a823b>	 [02WikiForge/puppet] 07Universal-Omega 039241f11 - phorge: remove support-archive
[01:21:36] <botifico-49a823b>	 [02puppet] 07Universal-Omega deleted branch 03remove-support-archive - 13https://github.com/WikiForge/puppet
[01:42:56] <botifico-49a823b>	 [02WikiForge/ssl] 07Universal-Omega pushed 031 commit to 03master [+0/-22/±1] 13https://github.com/WikiForge/ssl/compare/20eb0bd36320...0196e0d67f1c
[01:42:57] <botifico-49a823b>	 [02WikiForge/ssl] 07Universal-Omega 030196e0d - Remove certs not pointing to WikiForge
[01:50:16] <icinga-wikiforge>	 PROBLEM - phorge1 APT on phorge1 is CRITICAL: APT CRITICAL: 19 packages available for upgrade (4 critical updates).
[01:50:41] <icinga-wikiforge>	 PROBLEM - phorge1 PowerDNS Recursor on phorge1 is CRITICAL: Domain 'wikiforge.net' was not found by the server
[01:50:53] <icinga-wikiforge>	 PROBLEM - phorge1 phorge-static.wikiforge.net HTTPS on phorge1 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/1.1 404 Not Found
[02:07:56] <botifico-49a823b>	 [02WikiForge/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiForge/puppet/compare/9241f111e46c...46071800b591
[02:07:58] <botifico-49a823b>	 [02WikiForge/puppet] 07Universal-Omega 034607180 - Remove 1.39
[02:10:31] <icinga-wikiforge>	 RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[02:10:43] <WikiForgeRelay>	 <agentisai> @cosmicalpha have we confirmed no outlier remains on MW 1.39?
[02:14:02] <icinga-wikiforge>	 RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures
[02:14:31] <icinga-wikiforge>	 PROBLEM - mw1 Puppet on mw1 is UNKNOWN: UNKNOWN: Failed to check. Reason is: failed_to_parse_summary_file
[02:17:01] <WikiForgeLSBot>	 !log [universalomega@mw1] sudo chmod +r /opt/puppetlabs/puppet/public/last_run_summary.yaml
[02:17:52] <CosmicAlpha>	 @agentisai it is already missing on servers I checked anyway the directory did not exist
[02:18:03] <WikiForgeRelay>	 <agentisai> haha
[02:18:07] <WikiForgeRelay>	 <agentisai> that's great then
[02:19:38] <icinga-wikiforge>	 RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[02:20:26] <WikiForgeLSBot>	 !log [universalomega@mw2] sudo chmod +r /opt/puppetlabs/puppet/public/last_run_summary.yaml
[02:22:51] <WikiForgeLSBot>	 !log [universalomega@mw3] sudo chmod +r /opt/puppetlabs/puppet/public/last_run_summary.yaml
[02:25:20] <WikiForgeLSBot>	 !log [universalomega@mail1] sudo chmod +r /opt/puppetlabs/puppet/public/last_run_summary.yaml
[02:39:33] <botifico-49a823b>	 [02WikiForge/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±8] 13https://github.com/WikiForge/puppet/compare/46071800b591...d7dfc6ef3fa9
[02:39:35] <botifico-49a823b>	 [02WikiForge/puppet] 07Universal-Omega 03d7dfc6e - PostgreSQL: fix support for bookworm
[02:55:19] <icinga-wikiforge>	 RECOVERY - puppet1 APT on puppet1 is OK: APT OK: 0 packages available for upgrade (0 critical updates).
[02:57:11] <CosmicAlpha>	 oh.... whoops @agentisai I just realized puppet1 is still on bullseye but I just removed some support for it lol. I am going to upgrade to bookworm if your okay with that?
[02:57:36] <WikiForgeRelay>	 <agentisai, replying to CosmicAlpha> wait what? it's on bullseye?
[02:57:43] <WikiForgeRelay>	 <agentisai> 🤔
[02:57:56] <CosmicAlpha>	 lol yeah
[02:58:09] <WikiForgeRelay>	 <agentisai> I'm not sure how that happened...
[02:59:25] <icinga-wikiforge>	 PROBLEM - bast1 Puppet on bast1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[02:59:32] <CosmicAlpha>	 Not anymore lol
[03:00:50] <Agent>	 great!
[03:01:24] <icinga-wikiforge>	 RECOVERY - bast1 Puppet on bast1 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures
[03:05:29] <icinga-wikiforge>	 RECOVERY - puppet1 Puppet on puppet1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[03:11:48] <icinga-wikiforge>	 PROBLEM - mw2 Puppet on mw2 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 27 minutes ago with 0 failures
[03:17:42] <icinga-wikiforge>	 RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[04:29:03] <botifico-49a823b>	 [02WikiForge/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiForge/puppet/compare/d7dfc6ef3fa9...5b97b708e876
[04:29:04] <botifico-49a823b>	 [02WikiForge/puppet] 07Universal-Omega 035b97b70 - Add full URL
[05:19:41] <icinga-wikiforge>	 RECOVERY - phorge1 phorge-static.wikiforge.net HTTPS on phorge1 is OK: HTTP OK: Status line output matched "HTTP/1.1 200" - 53041 bytes in 0.287 second response time
[06:53:12] <icinga-wikiforge>	 PROBLEM - jobrunner1 Current Load on jobrunner1 is CRITICAL: LOAD CRITICAL - total load average: 11.31, 10.20, 5.00
[06:55:10] <icinga-wikiforge>	 PROBLEM - jobrunner1 Current Load on jobrunner1 is WARNING: LOAD WARNING - total load average: 2.95, 7.43, 4.60
[06:57:08] <icinga-wikiforge>	 RECOVERY - jobrunner1 Current Load on jobrunner1 is OK: LOAD OK - total load average: 1.21, 5.29, 4.15
[07:11:52] <icinga-wikiforge>	 PROBLEM - jobrunner1 Current Load on jobrunner1 is CRITICAL: LOAD CRITICAL - total load average: 8.74, 10.90, 6.63
[07:13:52] <icinga-wikiforge>	 PROBLEM - jobrunner1 Current Load on jobrunner1 is WARNING: LOAD WARNING - total load average: 2.07, 7.70, 5.98
[07:15:52] <icinga-wikiforge>	 RECOVERY - jobrunner1 Current Load on jobrunner1 is OK: LOAD OK - total load average: 0.80, 5.39, 5.34
[07:35:40] <icinga-wikiforge>	 PROBLEM - jobrunner1 MediaWiki Rendering on jobrunner1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[07:37:37] <icinga-wikiforge>	 PROBLEM - jobrunner1 JobRunner Service on jobrunner1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[07:38:44] <icinga-wikiforge>	 PROBLEM - jobrunner1 Current Load on jobrunner1 is CRITICAL: LOAD CRITICAL - total load average: 26.38, 18.20, 9.81
[07:39:09] <icinga-wikiforge>	 PROBLEM - jobrunner1 Disk Space on jobrunner1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[07:39:33] <icinga-wikiforge>	 RECOVERY - jobrunner1 JobRunner Service on jobrunner1 is OK: PROCS OK: 1 process with args 'redisJobRunnerService'
[07:39:44] <icinga-wikiforge>	 RECOVERY - jobrunner1 MediaWiki Rendering on jobrunner1 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 3.253 second response time
[07:41:07] <icinga-wikiforge>	 RECOVERY - jobrunner1 Disk Space on jobrunner1 is OK: DISK OK - free space: / 12835MiB (25% inode=71%);
[07:44:38] <icinga-wikiforge>	 PROBLEM - jobrunner1 Current Load on jobrunner1 is WARNING: LOAD WARNING - total load average: 0.93, 6.35, 7.09
[07:46:36] <icinga-wikiforge>	 RECOVERY - jobrunner1 Current Load on jobrunner1 is OK: LOAD OK - total load average: 1.53, 4.82, 6.44
[08:08:08] <WikiForgeLSBot>	 !log [agent@jobrunner1] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/rebuildall.php --wiki=harrypotterwiki (END - exit=35072)
[08:08:59] <WikiForgeRelay>	 <cosmicalpha> @agentisai ^
[08:09:12] <icinga-wikiforge>	 PROBLEM - jobrunner1 Current Load on jobrunner1 is CRITICAL: LOAD CRITICAL - total load average: 10.04, 9.00, 5.74
[08:09:47] <WikiForgeRelay>	 <agentisai> huh, got killed off
[08:09:57] <WikiForgeLSBot>	 !log [agent@jobrunner1] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/refreshLinks.php --wiki=harrypotterwiki 193500 (START)
[08:11:10] <icinga-wikiforge>	 RECOVERY - jobrunner1 Current Load on jobrunner1 is OK: LOAD OK - total load average: 2.22, 6.36, 5.17
[09:35:27] <WikiForgeLSBot>	 !log [agent@jobrunner1] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/refreshLinks.php --wiki=harrypotterwiki 193500 (END - exit=0)
[19:02:00] <botifico-49a823b>	 [02WikiForge/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiForge/puppet/compare/5b97b708e876...1126c7cc5a71
[19:02:01] <botifico-49a823b>	 [02WikiForge/puppet] 07Universal-Omega 031126c7c - matomo: remove DB SSL
[19:31:48] <botifico-49a823b>	 [02WikiForge/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiForge/puppet/compare/1126c7cc5a71...b69c7a5e45c6
[19:31:50] <botifico-49a823b>	 [02WikiForge/puppet] 07Universal-Omega 03b69c7a5 - matomo: upgrade to 4.16.0
[23:26:31] <botifico-49a823b>	 [02WikiForge/puppet] 07Universal-Omega created branch 03grafana-sqlite 13https://github.com/WikiForge/puppet/commit/b69c7a5e45c6826fdf47d9921f1fe47ba293af36
[23:26:33] <botifico-49a823b>	 [02puppet] 07Universal-Omega created branch 03grafana-sqlite - 13https://github.com/WikiForge/puppet
[23:28:17] <botifico-49a823b>	 [02WikiForge/puppet] 07Universal-Omega pushed 031 commit to 03grafana-sqlite [+0/-0/±5] 13https://github.com/WikiForge/puppet/compare/b69c7a5e45c6...be9fcfde8b9b
[23:28:20] <botifico-49a823b>	 [02WikiForge/puppet] 07Universal-Omega 03be9fcfd - grafana: use sqlite
[23:28:27] <botifico-49a823b>	 [02puppet] 07Universal-Omega opened pull request 03#212: grafana: use sqlite - 13https://github.com/WikiForge/puppet/pull/212
[23:29:26] <botifico-49a823b>	 [02puppet] 07Universal-Omega closed pull request 03#212: grafana: use sqlite - 13https://github.com/WikiForge/puppet/pull/212
[23:29:28] <botifico-49a823b>	 [02WikiForge/puppet] 07Universal-Omega deleted branch 03grafana-sqlite
[23:29:30] <botifico-49a823b>	 [02WikiForge/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±5] 13https://github.com/WikiForge/puppet/compare/b69c7a5e45c6...289634b979bf
[23:29:31] <botifico-49a823b>	 [02WikiForge/puppet] 07Universal-Omega 03289634b - grafana: use sqlite
[23:29:32] <botifico-49a823b>	 [02puppet] 07Universal-Omega deleted branch 03grafana-sqlite - 13https://github.com/WikiForge/puppet
[23:50:08] <icinga-wikiforge>	 PROBLEM - mon1 Backups Grafana on mon1 is CRITICAL: FILE_AGE CRITICAL: File not found - /var/log/grafana-backup.log