[00:01:18] <MacFanBot>	 [02statichelp] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/statichelp/commit/6e5ce2801337cc5f8c0a8f69ab4aad94c52f12dd
[00:01:18] <MacFanBot>	 02statichelp/03main 07WikiTideBot 036e5ce28 Bot: Auto-update Tech namespace pages 2025-12-21 00:01:15
[00:17:10] <icinga-miraheze>	 RECOVERY - cp201 Disk Space on cp201 is OK: DISK OK - free space: / 71245MiB (15% inode=99%);
[03:02:57] <icinga-miraheze>	 PROBLEM - puppet181 Disk Space on puppet181 is CRITICAL: DISK CRITICAL - free space: / 1808MiB (5% inode=79%);
[03:06:57] <icinga-miraheze>	 RECOVERY - puppet181 Disk Space on puppet181 is OK: DISK OK - free space: / 4174MiB (13% inode=79%);
[04:07:31] <icinga-miraheze>	 PROBLEM - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is WARNING: WARNING - NGINX Error Rate is 42%
[04:09:31] <icinga-miraheze>	 PROBLEM - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is CRITICAL: CRITICAL - NGINX Error Rate is 65%
[04:09:52] <MacFanBot>	 [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/5c3514115f00f041b5e90b7774aebd00eb0463eb
[04:09:52] <MacFanBot>	 02puppet/03main 07CosmicAlpha 035c35141 Add new SSH key
[04:11:31] <icinga-miraheze>	 RECOVERY - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is OK: OK - NGINX Error Rate is 29%
[04:15:31] <icinga-miraheze>	 PROBLEM - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is WARNING: WARNING - NGINX Error Rate is 53%
[04:17:31] <icinga-miraheze>	 RECOVERY - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is OK: OK - NGINX Error Rate is 32%
[11:21:10] <icinga-miraheze>	 PROBLEM - cp201 Disk Space on cp201 is WARNING: DISK WARNING - free space: / 49802MiB (10% inode=99%);
[12:31:24] <MirahezeRelay>	 <paladox> !log [root@matomo151:/srv/matomo]# ./console core:purge-old-archive-data
[12:31:28] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[12:50:08] <MirahezeRelay>	 <paladox> !log mwtask181: sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.php /srv/mediawiki/1.44/maintenance/run.php /srv/mediawiki/1.44/extensions/MatomoAnalytics/maintenance/CleanupMatomos.php
[12:50:12] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[12:51:47] <MirahezeRelay>	 <paladox> !log cancel
[12:51:51] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[12:52:15] <MirahezeRelay>	 <paladox> !log mwtask181: sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.php /srv/mediawiki/1.44/maintenance/run.php /srv/mediawiki/1.44/extensions/MatomoAnalytics/maintenance/ModifyMatomo.php --remove
[12:52:19] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[14:19:09] <icinga-miraheze>	 PROBLEM - cp191 Disk Space on cp191 is WARNING: DISK WARNING - free space: / 49855MiB (10% inode=99%);
[14:43:07] <miraheze-github>	 miraheze/MatomoAnalytics - paladox the build passed.
[14:48:32] <miraheze-github>	 miraheze/MatomoAnalytics - paladox the build passed.
[14:50:28] <MirahezeRelay>	 <paladox> !log erase matomo on db182
[14:50:32] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[14:52:18] <icinga-miraheze>	 RECOVERY - db182 Disk Space on db182 is OK: DISK OK - free space: / 526802MiB (89% inode=99%);
[14:53:01] <icinga-miraheze>	 PROBLEM - matomo151 Check unit status of matomo-queuedtracking-1 on matomo151 is CRITICAL: CRITICAL: Status of the systemd unit matomo-queuedtracking-1
[14:53:55] <icinga-miraheze>	 PROBLEM - matomo151 Check unit status of matomo-archiver-1 on matomo151 is CRITICAL: CRITICAL: Status of the systemd unit matomo-archiver-1
[14:54:04] <icinga-miraheze>	 PROBLEM - matomo151 Check unit status of matomo-archiver-3 on matomo151 is CRITICAL: CRITICAL: Status of the systemd unit matomo-archiver-3
[14:54:10] <icinga-miraheze>	 PROBLEM - matomo151 Check unit status of matomo-archiver-2 on matomo151 is CRITICAL: CRITICAL: Status of the systemd unit matomo-archiver-2
[14:54:20] <icinga-miraheze>	 PROBLEM - matomo151 Check unit status of matomo-queuedtracking-2 on matomo151 is CRITICAL: CRITICAL: Status of the systemd unit matomo-queuedtracking-2
[14:54:31] <icinga-miraheze>	 PROBLEM - matomo151 Check unit status of matomo-queuedtracking-0 on matomo151 is CRITICAL: CRITICAL: Status of the systemd unit matomo-queuedtracking-0
[15:01:54] <icinga-miraheze>	 PROBLEM - matomo151 Puppet on matomo151 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[matomo_composer]
[15:03:57] <icinga-miraheze>	 PROBLEM - matomo151 Puppet on matomo151 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 5 minutes ago with 1 failures
[15:09:12] <icinga-miraheze>	 PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 15.44, 12.24, 9.38
[15:11:12] <icinga-miraheze>	 PROBLEM - matomo151 Current Load on matomo151 is CRITICAL: LOAD CRITICAL - total load average: 16.52, 13.26, 10.08
[15:13:12] <icinga-miraheze>	 PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 13.82, 13.27, 10.47
[15:15:12] <icinga-miraheze>	 RECOVERY - matomo151 Current Load on matomo151 is OK: LOAD OK - total load average: 13.08, 12.99, 10.70
[15:19:12] <icinga-miraheze>	 PROBLEM - matomo151 Current Load on matomo151 is CRITICAL: LOAD CRITICAL - total load average: 16.50, 14.63, 11.87
[15:20:39] <icinga-miraheze>	 PROBLEM - cp171 Disk Space on cp171 is WARNING: DISK WARNING - free space: / 49824MiB (10% inode=99%);
[15:21:12] <icinga-miraheze>	 PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 15.80, 14.78, 12.25
[15:23:12] <icinga-miraheze>	 PROBLEM - matomo151 Current Load on matomo151 is CRITICAL: LOAD CRITICAL - total load average: 17.96, 16.04, 13.02
[15:25:12] <icinga-miraheze>	 PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 11.99, 14.71, 12.92
[15:31:12] <icinga-miraheze>	 RECOVERY - matomo151 Current Load on matomo151 is OK: LOAD OK - total load average: 11.66, 13.34, 13.00
[15:36:22] <icinga-miraheze>	 RECOVERY - matomo151 Puppet on matomo151 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[15:37:12] <icinga-miraheze>	 PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 14.96, 15.13, 13.77
[15:41:13] <icinga-miraheze>	 PROBLEM - matomo151 Current Load on matomo151 is CRITICAL: LOAD CRITICAL - total load average: 21.02, 17.11, 14.75
[15:42:09] <MirahezeRelay>	 <paladox> downtimed ^
[15:42:19] <icinga-miraheze>	 PROBLEM - ldap171 LDAP SSL check on ldap171 is WARNING: WARNING - Certificate 'wikitide.net' expires in 7 day(s) (Mon Dec 29 15:38:37 2025 +0000).
[15:49:15] <icinga-miraheze>	 RECOVERY - ldap171 LDAP SSL check on ldap171 is OK: OK - Certificate 'wikitide.net' will expire on Fri Mar 13 15:30:27 2026 +0000.TCP OK - 0.011 second response time on localhost port 636
[15:49:24] <MirahezeRelay>	 <paladox> !log restart slapd on ldap171
[15:49:28] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[21:09:57] <MirahezeRelay>	 <paladox> !log mwtask181: sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.php /srv/mediawiki/1.44/maintenance/run.php /srv/mediawiki/1.44/extensions/MatomoAnalytics/maintenance/ModifyMatomo.php
[21:10:02] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[21:15:08] <icinga-miraheze>	 RECOVERY - cp191 Disk Space on cp191 is OK: DISK OK - free space: / 76240MiB (16% inode=99%);
[21:53:10] <icinga-miraheze>	 PROBLEM - cp201 Disk Space on cp201 is CRITICAL: DISK CRITICAL - free space: / 27156MiB (5% inode=99%);
[22:10:09] <MirahezeLSBot>	 !log [somerandomdeveloper@test151] starting deploy of {'versions': '1.44', 'upgrade_extensions': 'Flow'} to test151
[22:10:10] <MirahezeLSBot>	 !log [somerandomdeveloper@test151] finished deploy of {'versions': '1.44', 'upgrade_extensions': 'Flow'} to test151 - SUCCESS in 1s
[22:10:13] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:10:16] <MirahezeLSBot>	 !log [somerandomdeveloper@mwtask181] starting deploy of {'versions': '1.44', 'upgrade_extensions': 'Flow'} to all
[22:10:17] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:10:21] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:10:46] <MirahezeLSBot>	 !log [somerandomdeveloper@mwtask181] finished deploy of {'versions': '1.44', 'upgrade_extensions': 'Flow'} to all - SUCCESS in 30s
[22:10:51] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[23:14:39] <icinga-miraheze>	 RECOVERY - cp171 Disk Space on cp171 is OK: DISK OK - free space: / 75974MiB (16% inode=99%);