[00:01:21] <MacFanBot>	 [02statichelp] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/statichelp/commit/9a82107ae989f0607cdf10afe1706b2c2e6ba12f
[00:01:21] <MacFanBot>	 02statichelp/03main 07WikiTideBot 039a82107 Bot: Auto-update Tech namespace pages 2025-10-26 00:01:19
[00:01:34] <icinga-miraheze>	 PROBLEM - cp201 Disk Space on cp201 is CRITICAL: DISK CRITICAL - free space: / 26934MiB (5% inode=99%);
[00:17:27] <icinga-miraheze>	 RECOVERY - cp201 Disk Space on cp201 is OK: DISK OK - free space: / 70202MiB (15% inode=99%);
[00:52:36] <MirahezeLSBot>	 !log [skye@mwtask171] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php MirahezeMagic:RenameDatabase --wiki=loginwiki --rename --old=piggyrobloxwikiwiki --new=piggywiki --user=Skye (END - exit=0)
[00:52:40] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[01:04:55] <MirahezeLSBot>	 !log [skye@mwtask171] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php CreateWiki:SetContainersAccess --wiki=piggywiki (END - exit=0)
[01:04:59] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[02:21:18] <icinga-miraheze>	 PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 13.23, 9.22, 5.75
[02:23:16] <icinga-miraheze>	 RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 3.91, 6.94, 5.33
[03:05:04] <icinga-miraheze>	 PROBLEM - puppet181 Disk Space on puppet181 is WARNING: DISK WARNING - free space: / 2427MiB (7% inode=80%);
[03:07:04] <icinga-miraheze>	 RECOVERY - puppet181 Disk Space on puppet181 is OK: DISK OK - free space: / 4659MiB (15% inode=80%);
[03:18:35] <icinga-miraheze>	 PROBLEM - db171 Current Load on db171 is WARNING: LOAD WARNING - total load average: 10.06, 10.74, 7.66
[03:20:35] <icinga-miraheze>	 PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 12.00, 12.01, 8.54
[03:22:35] <icinga-miraheze>	 RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 5.92, 9.97, 8.23
[10:18:40] <icinga-miraheze>	 PROBLEM - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is WARNING: WARNING - NGINX Error Rate is 45%
[10:20:40] <icinga-miraheze>	 RECOVERY - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is OK: OK - NGINX Error Rate is 31%
[11:31:27] <icinga-miraheze>	 PROBLEM - cp201 Disk Space on cp201 is WARNING: DISK WARNING - free space: / 49753MiB (10% inode=99%);
[13:42:20] <icinga-miraheze>	 PROBLEM - cp191 Disk Space on cp191 is WARNING: DISK WARNING - free space: / 49900MiB (10% inode=99%);
[14:16:35] <icinga-miraheze>	 PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 14.17, 10.85, 7.86
[14:18:10] <icinga-miraheze>	 PROBLEM - cp171 Disk Space on cp171 is WARNING: DISK WARNING - free space: / 49779MiB (10% inode=99%);
[14:18:35] <icinga-miraheze>	 PROBLEM - db171 Current Load on db171 is WARNING: LOAD WARNING - total load average: 10.73, 11.42, 8.49
[14:19:22] <icinga-miraheze>	 PROBLEM - puppet181 Check unit status of listdomains_github_push on puppet181 is CRITICAL: CRITICAL: Status of the systemd unit listdomains_github_push
[14:20:16] <MacFanBot>	 [02puppet] 07paladox created 03paladox-patch-3 (+1 new commit) 13https://github.com/miraheze/puppet/commit/10d14c86507a
[14:20:16] <MacFanBot>	 02puppet/03paladox-patch-3 07paladox 0310d14c8 mariadb: Change transaction isolation level to REPEATABLE-READ…
[14:20:20] <MacFanBot>	 [02puppet] 07paladox opened pull request #4587: mariadb: Change transaction isolation level to REPEATABLE-READ (03main...03paladox-patch-3) 13https://github.com/miraheze/puppet/pull/4587
[14:20:33] <MacFanBot>	 [02puppet] 07coderabbitai[bot] commented on pull request #4587: <details> […] 13https://github.com/miraheze/puppet/pull/4587#issuecomment-3448583788
[14:20:35] <icinga-miraheze>	 PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 22.88, 15.32, 10.25
[14:22:35] <icinga-miraheze>	 PROBLEM - db171 Current Load on db171 is WARNING: LOAD WARNING - total load average: 6.84, 11.78, 9.57
[14:24:35] <icinga-miraheze>	 RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 4.45, 9.35, 8.95
[14:25:22] <icinga-miraheze>	 RECOVERY - puppet181 Check unit status of listdomains_github_push on puppet181 is OK: OK: Status of the systemd unit listdomains_github_push
[14:28:35] <icinga-miraheze>	 PROBLEM - db171 Current Load on db171 is WARNING: LOAD WARNING - total load average: 5.85, 10.36, 9.82
[14:30:35] <icinga-miraheze>	 RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 7.62, 10.10, 9.81
[14:34:30] <MirahezeRelay>	 <paladox> !log on db* SET GLOBAL tx_isolation='REPEATABLE-READ'; (on db182/172 it's SET GLOBAL transaction_isolation='REPEATABLE-READ';)
[14:34:34] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[14:43:48] <MacFanBot>	 [02puppet] 07paladox merged pull request #4587: mariadb: Change transaction isolation level to REPEATABLE-READ (03main...03paladox-patch-3) 13https://github.com/miraheze/puppet/pull/4587
[14:43:48] <MacFanBot>	 [02puppet] 07paladox pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/e90d2d926c99a1ff327e930cda69c6565bf385ef
[14:43:48] <MacFanBot>	 02puppet/03main 07paladox 03e90d2d9 mariadb: Change transaction isolation level to REPEATABLE-READ (#4587)…
[14:43:49] <MacFanBot>	 [02puppet] 07paladox 04deleted 03paladox-patch-3 at 0310d14c8 13https://api.github.com/repos/miraheze/puppet/commit/10d14c8
[14:47:17] <icinga-miraheze>	 PROBLEM - matomo151 Check unit status of matomo-archiver-2 on matomo151 is CRITICAL: CRITICAL: Status of the systemd unit matomo-archiver-2
[14:48:17] <icinga-miraheze>	 PROBLEM - matomo151 Check unit status of matomo-archiver-1 on matomo151 is CRITICAL: CRITICAL: Status of the systemd unit matomo-archiver-1
[14:48:33] <icinga-miraheze>	 PROBLEM - matomo151 Check unit status of matomo-archiver-3 on matomo151 is CRITICAL: CRITICAL: Status of the systemd unit matomo-archiver-3
[15:06:24] <MacFanBot>	 [02puppet] 07paladox created 03paladox-patch-3 (+1 new commit) 13https://github.com/miraheze/puppet/commit/7857c1002edb
[15:06:24] <MacFanBot>	 02puppet/03paladox-patch-3 07paladox 037857c10 mariadb: remove innodb_thread_concurrency…
[15:06:28] <MacFanBot>	 [02puppet] 07paladox opened pull request #4588: mariadb: remove innodb_thread_concurrency (03main...03paladox-patch-3) 13https://github.com/miraheze/puppet/pull/4588
[15:06:35] <MacFanBot>	 [02puppet] 07paladox merged pull request #4588: mariadb: remove innodb_thread_concurrency (03main...03paladox-patch-3) 13https://github.com/miraheze/puppet/pull/4588
[15:06:37] <MacFanBot>	 [02puppet] 07paladox pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/2e621b5bea5569ca8c2c35bac431b91077845430
[15:06:39] <MacFanBot>	 [02puppet] 07paladox 04deleted 03paladox-patch-3 at 037857c10 13https://api.github.com/repos/miraheze/puppet/commit/7857c10
[15:06:40] <MacFanBot>	 02puppet/03main 07paladox 032e621b5 mariadb: remove innodb_thread_concurrency (#4588)…
[15:06:41] <MacFanBot>	 [02puppet] 07coderabbitai[bot] commented on pull request #4588: <details> […] 13https://github.com/miraheze/puppet/pull/4588#issuecomment-3448614821
[15:07:39] <MacFanBot>	 [02puppet] 07paladox created 03paladox-patch-3 (+1 new commit) 13https://github.com/miraheze/puppet/commit/014041c8b11d
[15:07:39] <MacFanBot>	 02puppet/03paladox-patch-3 07paladox 03014041c mariadb: Increase innodb_io_capacity from 500 to 1000…
[15:07:43] <MacFanBot>	 [02puppet] 07paladox opened pull request #4589: mariadb: Increase innodb_io_capacity from 500 to 1000 (03main...03paladox-patch-3) 13https://github.com/miraheze/puppet/pull/4589
[15:07:55] <MacFanBot>	 [02puppet] 07coderabbitai[bot] commented on pull request #4589: <details> […] 13https://github.com/miraheze/puppet/pull/4589#issuecomment-3448615789
[15:11:02] <MacFanBot>	 [02puppet] 07paladox commented on pull request #4051: I've reverted this in https://github.com/miraheze/puppet/pull/4589. Note to self when changing it, explain. 13https://github.com/miraheze/puppet/pull/4051#issuecomment-3448618333
[15:11:41] <MacFanBot>	 [02puppet] 07paladox merged pull request #4589: mariadb: Increase innodb_io_capacity from 500 to 1000 (03main...03paladox-patch-3) 13https://github.com/miraheze/puppet/pull/4589
[15:11:41] <MacFanBot>	 [02puppet] 07paladox pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/b99e8d5a8f050ac5524f8a0d62250c5e53abb2ce
[15:11:41] <MacFanBot>	 02puppet/03main 07paladox 03b99e8d5 mariadb: Increase innodb_io_capacity from 500 to 1000 (#4589)…
[15:11:43] <MacFanBot>	 [02puppet] 07paladox 04deleted 03paladox-patch-3 at 03014041c 13https://api.github.com/repos/miraheze/puppet/commit/014041c
[15:13:23] <MirahezeRelay>	 <paladox> !log db*: set global innodb_io_capacity = 1000;
[15:13:27] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[15:40:12] <icinga-miraheze>	 PROBLEM - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is WARNING: WARNING - NGINX Error Rate is 41%
[15:42:12] <icinga-miraheze>	 RECOVERY - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is OK: OK - NGINX Error Rate is 23%
[16:00:17] <icinga-miraheze>	 RECOVERY - matomo151 Check unit status of matomo-archiver-1 on matomo151 is OK: OK: Status of the systemd unit matomo-archiver-1
[16:01:17] <icinga-miraheze>	 RECOVERY - matomo151 Check unit status of matomo-archiver-2 on matomo151 is OK: OK: Status of the systemd unit matomo-archiver-2
[16:02:33] <icinga-miraheze>	 RECOVERY - matomo151 Check unit status of matomo-archiver-3 on matomo151 is OK: OK: Status of the systemd unit matomo-archiver-3
[17:08:12] <icinga-miraheze>	 PROBLEM - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is WARNING: WARNING - NGINX Error Rate is 51%
[17:10:12] <icinga-miraheze>	 RECOVERY - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is OK: OK - NGINX Error Rate is 11%
[22:16:20] <icinga-miraheze>	 RECOVERY - cp191 Disk Space on cp191 is OK: DISK OK - free space: / 74471MiB (16% inode=99%);
[22:33:26] <MirahezeLSBot>	 !log [somerandomdeveloper@test151] starting deploy of {'versions': ['1.44', '1.45'], 'upgrade_skins': 'Monaco'} to test151
[22:33:28] <MirahezeLSBot>	 !log [somerandomdeveloper@test151] finished deploy of {'versions': ['1.44', '1.45'], 'upgrade_skins': 'Monaco'} to test151 - SUCCESS in 1s
[22:33:30] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:33:32] <MirahezeLSBot>	 !log [somerandomdeveloper@mwtask181] starting deploy of {'versions': '1.44', 'upgrade_skins': 'Monaco'} to all
[22:33:35] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:33:39] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:33:58] <MirahezeLSBot>	 !log [somerandomdeveloper@mwtask181] finished deploy of {'versions': '1.44', 'upgrade_skins': 'Monaco'} to all - SUCCESS in 25s
[22:34:02] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:39:35] <MirahezeRelay>	 <bartomelow, replying to MirahezeLSBot> it works!!!!
[22:39:51] <MirahezeRelay>	 <abaddriverlol> nice
[22:47:40] <icinga-miraheze>	 PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 15.22, 13.40, 8.49
[22:49:39] <icinga-miraheze>	 RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 4.63, 10.04, 7.86
[22:51:27] <icinga-miraheze>	 PROBLEM - cp201 Disk Space on cp201 is CRITICAL: DISK CRITICAL - free space: / 27182MiB (5% inode=99%);
[22:57:31] <icinga-miraheze>	 PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 25.96, 14.38, 9.83
[23:03:26] <icinga-miraheze>	 RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 4.18, 9.57, 9.35
[23:14:10] <icinga-miraheze>	 RECOVERY - cp171 Disk Space on cp171 is OK: DISK OK - free space: / 74506MiB (16% inode=99%);
[23:32:57] <MirahezeRelay>	 <paladox> !log upgrade phorge on phorge171
[23:33:00] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[23:50:56] <icinga-miraheze>	 PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 20.78, 16.81, 10.37
[23:52:56] <icinga-miraheze>	 RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 17.06, 16.65, 11.10