[00:01:22] <MacFanBot>	 [02statichelp] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/statichelp/commit/acf1c9bad5c239c7bf9bbe151de41d9bc1d07dbf
[00:01:22] <MacFanBot>	 02statichelp/03main 07WikiTideBot 03acf1c9b Bot: Auto-update Tech namespace pages 2026-09-27 00:01:19
[01:02:16] <MacFanBot>	 [02TSPortal] 07renovate[bot] created 03renovate/lock-file-maintenance (+1 new commit) 13https://github.com/miraheze/TSPortal/commit/391b86751ae0
[01:02:16] <MacFanBot>	 02TSPortal/03renovate/lock-file-maintenance 07renovate[bot] 03391b867 Lock file maintenance
[01:05:32] <icinga-miraheze>	 RECOVERY - cp201 Disk Space on cp201 is OK: DISK OK - free space: / 84624MiB (18.7% inode=100%);
[01:44:18] <MacFanBot>	 [02mediawiki-repos] 07lihaohong6 opened pull request #169: T13532: pin ConfirmEdit and DiscussionTools (07miraheze:03main...07lihaohong6:03captcha) 13https://github.com/miraheze/mediawiki-repos/pull/169
[01:44:32] <MacFanBot>	 [02mediawiki-repos] 07lihaohong6 merged pull request #168: T16102: remove minimum cache TTL set by core (07miraheze:03main...07lihaohong6:03cache) 13https://github.com/miraheze/mediawiki-repos/pull/168
[01:44:35] <MacFanBot>	 [02mediawiki-repos] 07lihaohong6 pushed 1 new commit to 03main 13https://github.com/miraheze/mediawiki-repos/commit/20155a5a4202838d3fb0e542ec3171d23fa3388e
[01:44:36] <MacFanBot>	 02mediawiki-repos/03main 07Peter Li 0320155a5 T16102: remove minimum cache TTL set by core (#168)
[01:49:26] <MirahezeLSBot>	 !log [universalomega@mwtask181] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/deleted.php /srv/mediawiki/1.46/maintenance/run.php MirahezeMagic:UpgradeWiki --json=/srv/mediawiki/1.46.json --force (END - exit=0)
[01:49:31] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[02:25:22] <MacFanBot>	 [02puppet] 07Universal-Omega pushed 1 new commit to 03mwdeploy-rework 13https://github.com/miraheze/puppet/commit/a8f3aca76a02b50961cefb4d7689f69839526192
[02:25:22] <MacFanBot>	 02puppet/03mwdeploy-rework 07CosmicAlpha 03a8f3aca Check reverse to see if patch is already applied
[02:26:37] <miraheze-github>	 miraheze/puppet - Universal-Omega the build passed.
[02:33:10] <MacFanBot>	 [02puppet] 07Universal-Omega pushed 1 new commit to 03mwdeploy-rework 13https://github.com/miraheze/puppet/commit/9e6fc64dc41d83a65b3e6a3e531cfc144dc74600
[02:33:10] <MacFanBot>	 02puppet/03mwdeploy-rework 07CosmicAlpha 039e6fc64 Fixes to patches
[02:34:26] <miraheze-github>	 miraheze/puppet - Universal-Omega the build has errored.
[02:38:08] <MacFanBot>	 [02puppet] 07Universal-Omega pushed 1 new commit to 03mwdeploy-rework 13https://github.com/miraheze/puppet/commit/00f00b4a6fe2ebe3de661debe5ab066392afa306
[02:38:08] <MacFanBot>	 02puppet/03mwdeploy-rework 07CosmicAlpha 0300f00b4 Three-way patch support
[02:39:29] <miraheze-github>	 miraheze/puppet - Universal-Omega the build has errored.
[02:46:49] <MacFanBot>	 [02puppet] 07Universal-Omega pushed 1 new commit to 03mwdeploy-rework 13https://github.com/miraheze/puppet/commit/9d52c7296625e614fa734699a2f16bb91de8b109
[02:46:49] <MacFanBot>	 02puppet/03mwdeploy-rework 07CosmicAlpha 039d52c72 Cleanup
[02:48:00] <miraheze-github>	 miraheze/puppet - Universal-Omega the build has errored.
[03:29:03] <MacFanBot>	 [02puppet] 07Universal-Omega pushed 1 new commit to 03mwdeploy-rework 13https://github.com/miraheze/puppet/commit/4dd02202fdf2a6f6e9a50ed404eaeed57697ab34
[03:29:03] <MacFanBot>	 02puppet/03mwdeploy-rework 07CosmicAlpha 034dd0220 Colored console + progress bars
[03:30:17] <miraheze-github>	 miraheze/puppet - Universal-Omega the build has errored.
[03:41:29] <MacFanBot>	 [02puppet] 07Universal-Omega pushed 1 new commit to 03mwdeploy-rework 13https://github.com/miraheze/puppet/commit/115bbb77adf0c9a157077844e8e53c2abfd35d7f
[03:41:29] <MacFanBot>	 02puppet/03mwdeploy-rework 07CosmicAlpha 03115bbb7 Output improvements
[03:42:36] <miraheze-github>	 miraheze/puppet - Universal-Omega the build has errored.
[04:02:48] <MacFanBot>	 [02puppet] 07Universal-Omega pushed 1 new commit to 03mwdeploy-rework 13https://github.com/miraheze/puppet/commit/4e7e119395d052b240e3df821ffbad442eabdfda
[04:02:48] <MacFanBot>	 02puppet/03mwdeploy-rework 07CosmicAlpha 034e7e119 Use subprocess
[04:04:08] <miraheze-github>	 miraheze/puppet - Universal-Omega the build has errored.
[06:29:10] <MacFanBot>	 [02TSPortal] 07renovate[bot] merged 03renovate/lock-file-maintenance into 03main 13https://github.com/miraheze/TSPortal/compare/e0a01969b718...391b86751ae0
[06:29:12] <MacFanBot>	 [02TSPortal] 07renovate[bot] 04deleted 03renovate/lock-file-maintenance at 03391b867 13https://api.github.com/repos/miraheze/TSPortal/commit/391b867
[07:24:42] <MacFanBot>	 [02puppet] 07Universal-Omega pushed 1 new commit to 03mwdeploy-rework 13https://github.com/miraheze/puppet/commit/42f1d1bdd11a91afcada82328ce21053c732b0f2
[07:24:42] <MacFanBot>	 02puppet/03mwdeploy-rework 07CosmicAlpha 0342f1d1b Update tests
[07:26:08] <miraheze-github>	 miraheze/puppet - Universal-Omega the build has errored.
[07:27:43] <icinga-miraheze>	 PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 12.93, 10.86, 6.85
[07:28:30] <RhinosF1>	 Nope
[07:28:31] <RhinosF1>	 No
[07:28:47] * RhinosF1 stares at the database and suggests it behaves
[07:29:43] <icinga-miraheze>	 PROBLEM - db171 Current Load on db171 is WARNING: LOAD WARNING - total load average: 11.28, 10.58, 7.21
[07:30:19] <MacFanBot>	 [02puppet] 07Universal-Omega pushed 1 new commit to 03mwdeploy-rework 13https://github.com/miraheze/puppet/commit/3b5b8bd37238d94e978e19b938b9fe6d8dbc2e78
[07:30:19] <MacFanBot>	 02puppet/03mwdeploy-rework 07CosmicAlpha 033b5b8bd Fix
[07:31:30] <miraheze-github>	 miraheze/puppet - Universal-Omega the build has errored.
[07:31:43] <icinga-miraheze>	 PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 22.68, 14.32, 8.94
[07:32:05] <icinga-miraheze>	 PROBLEM - mw183 Current Load on mw183 is WARNING: LOAD WARNING - total load average: 23.03, 20.39, 15.96
[07:36:02] <icinga-miraheze>	 RECOVERY - mw183 Current Load on mw183 is OK: LOAD OK - total load average: 14.85, 18.36, 16.23
[07:36:54] <MacFanBot>	 [02puppet] 07Universal-Omega pushed 1 new commit to 03mwdeploy-rework 13https://github.com/miraheze/puppet/commit/18adf6d236b4a68c32f07ce38e853994dc385ba3
[07:36:54] <MacFanBot>	 02puppet/03mwdeploy-rework 07CosmicAlpha 0318adf6d Fix
[07:37:43] <icinga-miraheze>	 PROBLEM - db171 Current Load on db171 is WARNING: LOAD WARNING - total load average: 9.75, 11.40, 9.42
[07:38:10] <MirahezeRelay>	 <posix_memalign> Probably caused by scrapers on philosophyballwiki
[07:38:34] <miraheze-github>	 miraheze/puppet - Universal-Omega the build passed.
[07:39:21] <MirahezeRelay>	 <posix_memalign> [1/2] Very neat cloudflare feature
[07:39:21] <MirahezeRelay>	 <posix_memalign> [2/2] https://cdn.discordapp.com/attachments/808001911868489748/1553672379434143767/image.png?ex=6aba19a9&is=6ab8c829&hm=861fdeb7da1fe07502a1cb3f41795a996681d0ddbed422efb11802e7bd290cdf&
[07:39:56] <MirahezeRelay>	 <posix_memalign> CF rule deployed
[07:41:43] <icinga-miraheze>	 RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 6.74, 9.21, 9.00
[07:51:01] <MirahezeRelay>	 <posix_memalign> [1/2] Turns out @orduin already has a more general rule and that's why I'm seeing 0 hits on the new rule.
[07:51:01] <MirahezeRelay>	 <posix_memalign> [2/2] But the bots have figured out how to bypass CF challenges so I'll make it stricter.
[08:09:45] <MacFanBot>	 [02puppet] 07Universal-Omega merged 07dependabot[bot]'s pull request #5050: build(deps-dev): bump rubyzip from 2.4.1 to 3.4.0 in /modules/graylog (03main...03dependabot/bundler/modules/graylog/rubyzip-3.4.0) 13https://github.com/miraheze/puppet/pull/5050
[08:09:45] <MacFanBot>	 [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/abc7b2b3739700dea18fc854a05816505e682f71
[08:09:45] <MacFanBot>	 02puppet/03main 07dependabot[bot] 03abc7b2b build(deps-dev): bump rubyzip from 2.4.1 to 3.4.0 in /modules/graylog (#5050)…
[08:09:47] <MacFanBot>	 [02puppet] 07Universal-Omega 04deleted 03dependabot/bundler/modules/graylog/rubyzip-3.4.0 at 03fe71af4 13https://api.github.com/repos/miraheze/puppet/commit/fe71af4
[10:06:30] <MacFanBot>	 [02ManageWiki] 07Reception123 pushed 1 new commit to 03undelete 13https://github.com/miraheze/ManageWiki/commit/803fbe00745d5aa83b0d1f0e4610b555cf7a22bf
[10:06:30] <MacFanBot>	 02ManageWiki/03undelete 07Reception123 03803fbe0 Update SpecialUndeleteWiki.php
[11:28:12] <icinga-miraheze>	 PROBLEM - mw173 MediaWiki Rendering on mw173 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[11:28:24] <icinga-miraheze>	 PROBLEM - mw173 HTTPS on mw173 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Connection timed out after 10005 milliseconds
[11:28:48] <icinga-miraheze>	 PROBLEM - mw173.fsslc.wtnet SSL Check on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[11:28:54] <icinga-miraheze>	 PROBLEM - mw173 SSH on mw173 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[11:30:51] <icinga-miraheze>	 RECOVERY - mw173 SSH on mw173 is OK: SSH OK - OpenSSH_10.0p2 (protocol 2.0)
[11:31:03] <icinga-miraheze>	 PROBLEM - mw173 Current Load on mw173 is CRITICAL: LOAD CRITICAL - total load average: 50.88, 45.68, 24.23
[11:32:14] <icinga-miraheze>	 RECOVERY - mw173 MediaWiki Rendering on mw173 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.189 second response time
[11:32:23] <icinga-miraheze>	 RECOVERY - mw173 HTTPS on mw173 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 340 bytes in 0.075 second response time
[11:35:03] <icinga-miraheze>	 PROBLEM - mw173 Current Load on mw173 is WARNING: LOAD WARNING - total load average: 6.07, 23.36, 19.91
[11:37:03] <icinga-miraheze>	 RECOVERY - mw173 Current Load on mw173 is OK: LOAD OK - total load average: 6.41, 17.77, 18.28
[11:58:11] <icinga-miraheze>	 RECOVERY - mw173.fsslc.wtnet SSL Check on sslhost is OK: OK - Certificate 'CloudFlare Origin Certificate' will expire on Tue 03 Apr 2040 12:14:00 PM GMT +0000.
[14:24:39] <MacFanBot>	 [02python-functions] 07renovate[bot] created 03renovate/all-deps (+1 new commit) 13https://github.com/miraheze/python-functions/commit/e2f2cc2ba15f
[14:24:39] <MacFanBot>	 02python-functions/03renovate/all-deps 07renovate[bot] 03e2f2cc2 chore(deps): update dependency coverage to v7.16.2
[15:22:46] <icinga-miraheze>	 PROBLEM - mw202 Current Load on mw202 is WARNING: LOAD WARNING - total load average: 22.00, 19.39, 14.84
[15:22:55] <icinga-miraheze>	 PROBLEM - mw192 Current Load on mw192 is WARNING: LOAD WARNING - total load average: 21.63, 19.03, 14.38
[15:23:05] <icinga-miraheze>	 PROBLEM - mw201 Current Load on mw201 is WARNING: LOAD WARNING - total load average: 22.10, 19.74, 14.96
[15:23:10] <icinga-miraheze>	 PROBLEM - mw183 Current Load on mw183 is CRITICAL: LOAD CRITICAL - total load average: 24.62, 20.52, 15.48
[15:24:46] <icinga-miraheze>	 RECOVERY - mw202 Current Load on mw202 is OK: LOAD OK - total load average: 18.05, 18.59, 15.09
[15:24:55] <icinga-miraheze>	 RECOVERY - mw192 Current Load on mw192 is OK: LOAD OK - total load average: 18.55, 18.72, 14.84
[15:25:05] <icinga-miraheze>	 RECOVERY - mw201 Current Load on mw201 is OK: LOAD OK - total load average: 19.83, 20.07, 15.69
[15:25:10] <icinga-miraheze>	 PROBLEM - mw183 Current Load on mw183 is WARNING: LOAD WARNING - total load average: 22.91, 20.97, 16.24
[15:27:10] <icinga-miraheze>	 RECOVERY - mw183 Current Load on mw183 is OK: LOAD OK - total load average: 18.41, 20.06, 16.48
[15:32:55] <icinga-miraheze>	 PROBLEM - mw192 Current Load on mw192 is CRITICAL: LOAD CRITICAL - total load average: 24.08, 21.11, 17.32
[15:33:10] <icinga-miraheze>	 PROBLEM - mw183 Current Load on mw183 is WARNING: LOAD WARNING - total load average: 21.31, 20.71, 17.78
[15:34:46] <icinga-miraheze>	 PROBLEM - mw202 Current Load on mw202 is WARNING: LOAD WARNING - total load average: 21.33, 19.72, 17.09
[15:34:55] <icinga-miraheze>	 PROBLEM - mw192 Current Load on mw192 is WARNING: LOAD WARNING - total load average: 23.50, 21.83, 18.05
[15:35:05] <icinga-miraheze>	 PROBLEM - mw201 Current Load on mw201 is WARNING: LOAD WARNING - total load average: 23.60, 21.19, 17.85
[15:35:10] <icinga-miraheze>	 PROBLEM - mw183 Current Load on mw183 is CRITICAL: LOAD CRITICAL - total load average: 25.62, 22.31, 18.71
[15:36:46] <icinga-miraheze>	 RECOVERY - mw202 Current Load on mw202 is OK: LOAD OK - total load average: 18.74, 19.42, 17.31
[15:37:05] <icinga-miraheze>	 RECOVERY - mw201 Current Load on mw201 is OK: LOAD OK - total load average: 16.80, 20.01, 17.86
[15:37:10] <icinga-miraheze>	 PROBLEM - mw183 Current Load on mw183 is WARNING: LOAD WARNING - total load average: 19.43, 22.05, 19.12
[15:38:55] <icinga-miraheze>	 RECOVERY - mw192 Current Load on mw192 is OK: LOAD OK - total load average: 9.73, 16.79, 17.01
[15:39:10] <icinga-miraheze>	 RECOVERY - mw183 Current Load on mw183 is OK: LOAD OK - total load average: 10.27, 17.68, 17.86
[16:32:43] <MacFanBot>	 [02python-functions] 07renovate[bot] merged 03renovate/all-deps into 03main 13https://github.com/miraheze/python-functions/compare/8d2bc915183f...e2f2cc2ba15f
[16:32:44] <MacFanBot>	 [02python-functions] 07renovate[bot] 04deleted 03renovate/all-deps at 03e2f2cc2 13https://api.github.com/repos/miraheze/python-functions/commit/e2f2cc2
[16:36:51] <miraheze-github>	 miraheze/python-functions - renovate[bot] the build passed.
[19:16:46] <icinga-miraheze>	 PROBLEM - mw202 Current Load on mw202 is WARNING: LOAD WARNING - total load average: 21.72, 16.62, 11.60
[19:16:55] <icinga-miraheze>	 PROBLEM - mw192 Current Load on mw192 is WARNING: LOAD WARNING - total load average: 22.79, 16.64, 11.70
[19:17:03] <icinga-miraheze>	 PROBLEM - mw173 Current Load on mw173 is WARNING: LOAD WARNING - total load average: 21.59, 16.57, 11.41
[19:17:10] <icinga-miraheze>	 PROBLEM - mw183 Current Load on mw183 is WARNING: LOAD WARNING - total load average: 23.35, 18.33, 13.16
[19:17:56] <icinga-miraheze>	 PROBLEM - mw193 Current Load on mw193 is WARNING: LOAD WARNING - total load average: 21.57, 17.17, 11.79
[19:18:02] <icinga-miraheze>	 PROBLEM - mw163 Current Load on mw163 is WARNING: LOAD WARNING - total load average: 22.96, 17.88, 12.28
[19:18:20] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.55, 17.49, 12.47
[19:18:21] <icinga-miraheze>	 PROBLEM - mw203 Current Load on mw203 is WARNING: LOAD WARNING - total load average: 20.97, 17.41, 12.35
[19:19:06] <icinga-miraheze>	 PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.49, 17.76, 12.59
[19:19:06] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 21.62, 17.80, 12.14
[19:19:20] <icinga-miraheze>	 PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 21.11, 18.04, 12.70
[19:21:03] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 21.10, 17.78, 12.28
[19:21:06] <icinga-miraheze>	 RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 20.15, 18.56, 13.11
[19:21:08] <icinga-miraheze>	 PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 20.60, 18.19, 12.67
[19:21:24] <MirahezeRelay>	 <tsybangd> wat
[19:22:20] <icinga-miraheze>	 RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 16.33, 18.04, 13.94
[19:22:48] <icinga-miraheze>	 PROBLEM - mw191 Current Load on mw191 is WARNING: LOAD WARNING - total load average: 21.64, 19.61, 14.06
[19:23:03] <icinga-miraheze>	 RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 17.21, 17.97, 13.06
[19:23:03] <icinga-miraheze>	 RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 17.07, 18.18, 13.34
[19:23:10] <icinga-miraheze>	 PROBLEM - mw183 Current Load on mw183 is CRITICAL: LOAD CRITICAL - total load average: 24.45, 22.09, 16.52
[19:23:20] <icinga-miraheze>	 RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 16.95, 18.50, 14.18
[19:24:02] <icinga-miraheze>	 RECOVERY - mw163 Current Load on mw163 is OK: LOAD OK - total load average: 18.45, 19.39, 14.89
[19:24:11] <icinga-miraheze>	 PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 20.92, 19.21, 14.20
[19:24:21] <icinga-miraheze>	 RECOVERY - mw203 Current Load on mw203 is OK: LOAD OK - total load average: 19.93, 19.68, 15.10
[19:24:44] <icinga-miraheze>	 RECOVERY - mw191 Current Load on mw191 is OK: LOAD OK - total load average: 20.35, 19.66, 14.72
[19:24:55] <icinga-miraheze>	 RECOVERY - mw192 Current Load on mw192 is OK: LOAD OK - total load average: 20.37, 20.21, 15.61
[19:25:10] <icinga-miraheze>	 PROBLEM - mw183 Current Load on mw183 is WARNING: LOAD WARNING - total load average: 22.84, 22.53, 17.38
[19:26:11] <icinga-miraheze>	 RECOVERY - mw161 Current Load on mw161 is OK: LOAD OK - total load average: 19.49, 19.32, 14.86
[19:27:06] <icinga-miraheze>	 PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 21.07, 19.68, 15.27
[19:27:44] <icinga-miraheze>	 PROBLEM - mw191 Check unit status of php8.4-fpm_check_restart on mw191 is CRITICAL: CRITICAL: Status of the systemd unit php8.4-fpm_check_restart
[19:27:56] <icinga-miraheze>	 RECOVERY - mw193 Current Load on mw193 is OK: LOAD OK - total load average: 19.86, 20.28, 16.16
[19:28:02] <icinga-miraheze>	 PROBLEM - mw163 Current Load on mw163 is WARNING: LOAD WARNING - total load average: 22.43, 20.75, 16.45
[19:28:36] <icinga-miraheze>	 PROBLEM - mw191 Current Load on mw191 is WARNING: LOAD WARNING - total load average: 20.55, 20.34, 16.13
[19:28:46] <icinga-miraheze>	 RECOVERY - mw202 Current Load on mw202 is OK: LOAD OK - total load average: 19.98, 20.16, 16.45
[19:28:55] <icinga-miraheze>	 PROBLEM - mw192 Current Load on mw192 is WARNING: LOAD WARNING - total load average: 19.63, 20.48, 16.83
[19:29:03] <icinga-miraheze>	 PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 21.14, 19.74, 15.33
[19:29:06] <icinga-miraheze>	 RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 19.60, 19.94, 15.92
[19:29:10] <icinga-miraheze>	 PROBLEM - mw183 Current Load on mw183 is CRITICAL: LOAD CRITICAL - total load average: 24.10, 22.83, 18.64
[19:30:34] <MacFanBot>	 [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/f3bfaf76bc1d9dd7289bb51674705ba7ce897bb1
[19:30:35] <MacFanBot>	 02ssl/03main 07WikiTideBot 03f3bfaf7 Bot: Auto-update domain lists
[19:31:03] <icinga-miraheze>	 RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 18.88, 19.14, 15.64
[19:31:03] <icinga-miraheze>	 RECOVERY - mw173 Current Load on mw173 is OK: LOAD OK - total load average: 20.27, 20.30, 17.06
[19:31:10] <icinga-miraheze>	 PROBLEM - mw183 Current Load on mw183 is WARNING: LOAD WARNING - total load average: 23.35, 23.02, 19.22
[19:31:56] <icinga-miraheze>	 PROBLEM - mw193 Current Load on mw193 is WARNING: LOAD WARNING - total load average: 22.06, 21.05, 17.42
[19:32:46] <icinga-miraheze>	 PROBLEM - mw202 Current Load on mw202 is WARNING: LOAD WARNING - total load average: 21.04, 20.56, 17.45
[19:34:23] <icinga-miraheze>	 RECOVERY - mw191 Current Load on mw191 is OK: LOAD OK - total load average: 19.99, 20.29, 17.47
[19:34:46] <icinga-miraheze>	 RECOVERY - mw202 Current Load on mw202 is OK: LOAD OK - total load average: 19.15, 19.84, 17.55
[19:35:10] <icinga-miraheze>	 PROBLEM - mw183 Current Load on mw183 is CRITICAL: LOAD CRITICAL - total load average: 24.66, 23.35, 20.19
[19:36:20] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.66, 20.69, 17.76
[19:36:48] <icinga-miraheze>	 PROBLEM - mw201 Current Load on mw201 is WARNING: LOAD WARNING - total load average: 20.03, 20.50, 17.81
[19:36:55] <icinga-miraheze>	 RECOVERY - mw192 Current Load on mw192 is OK: LOAD OK - total load average: 19.63, 20.23, 18.25
[19:37:56] <icinga-miraheze>	 RECOVERY - mw193 Current Load on mw193 is OK: LOAD OK - total load average: 13.37, 18.59, 17.70
[19:38:02] <icinga-miraheze>	 RECOVERY - mw163 Current Load on mw163 is OK: LOAD OK - total load average: 12.36, 18.37, 17.74
[19:38:20] <icinga-miraheze>	 RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 10.96, 17.30, 16.90
[19:38:46] <icinga-miraheze>	 RECOVERY - mw201 Current Load on mw201 is OK: LOAD OK - total load average: 9.25, 16.46, 16.68
[19:39:05] <icinga-miraheze>	 RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 8.56, 16.04, 16.90
[19:39:10] <icinga-miraheze>	 RECOVERY - mw183 Current Load on mw183 is OK: LOAD OK - total load average: 9.83, 18.66, 19.26
[19:47:18] <MirahezeLSBot>	 !log [wwr@mwtask171] curl -X DELETE https://opensearch-mw.wikitide.net/samandmaxwiki_content https://opensearch-mw.wikitide.net/samandmaxwiki_general
[19:47:23] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:47:33] <MirahezeLSBot>	 !log [wwr@mwtask171] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php CirrusSearch:UpdateSearchIndexConfig --wiki=samandmaxwiki --startOver (END - exit=0)
[19:47:39] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:58:07] <MirahezeLSBot>	 !log [petramagna@test151] starting deploy of {'folders': '1.46/extensions/DiscussionTools,1.46/extensions/ConfirmEdit'} to test151
[19:58:08] <MirahezeLSBot>	 !log [petramagna@test151] DEPLOY ABORTED: Canary check failed for meta.mirabeta.org@localhost
[19:58:13] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:58:18] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:59:06] <MirahezeLSBot>	 !log [petramagna@test151] starting deploy of {'folders': '1.46/extensions/DiscussionTools,1.46/extensions/ConfirmEdit'} to test151
[19:59:07] <MirahezeLSBot>	 !log [petramagna@test151] DEPLOY ABORTED: Canary check failed for meta.mirabeta.org@localhost
[19:59:11] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:59:15] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:59:50] <MirahezeLSBot>	 !log [petramagna@test151] starting deploy of {'folders': '1.46/extensions/DiscussionTools,1.46/extensions/ConfirmEdit'} to test151
[19:59:51] <MirahezeLSBot>	 !log [petramagna@test151] finished deploy of {'folders': '1.46/extensions/DiscussionTools,1.46/extensions/ConfirmEdit'} to test151 - SUCCESS in 0s
[19:59:55] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[20:00:00] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[20:13:39] <MacFanBot>	 [02puppet] 07Universal-Omega merged pull request #4920: Massive rework to mwdeploy (03main...03mwdeploy-rework) 13https://github.com/miraheze/puppet/pull/4920
[20:13:39] <MacFanBot>	 [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/e117dd2435834f29465a7e65dbfc6ba6fb7a85cc
[20:13:39] <MacFanBot>	 02puppet/03main 07CosmicAlpha 03e117dd2 Massive rework to mwdeploy
[20:13:41] <MacFanBot>	 [02puppet] 07Universal-Omega 04deleted 03mwdeploy-rework at 0318adf6d 13https://api.github.com/repos/miraheze/puppet/commit/18adf6d
[20:15:39] <miraheze-github>	 miraheze/puppet - Universal-Omega the build passed.
[20:17:12] <MirahezeLSBot>	 !log [wwr@mwtask171] sudo -u www-data php /srv/mediawiki/1.45/maintenance/run.php CirrusSearch:ForceSearchIndex --wiki=samandmaxwiki (END - exit=0)
[20:17:17] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[20:21:32] <icinga-miraheze>	 PROBLEM - cp201 Disk Space on cp201 is WARNING: DISK WARNING - free space: / 45230MiB (10.0% inode=100%);
[20:30:21] <MacFanBot>	 [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/514149804dc45f795e3c34e06370b033912e3e5d
[20:30:21] <MacFanBot>	 02ssl/03main 07WikiTideBot 035141498 Bot: Auto-update domain lists
[20:33:21] <MacFanBot>	 [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/9b1072a4f4126178ec5cf7a68a12916e6b967fe8
[20:33:21] <MacFanBot>	 02puppet/03main 07CosmicAlpha 039b1072a mwscript: support upgrade-wikis dblist
[20:34:51] <miraheze-github>	 miraheze/puppet - Universal-Omega the build passed.
[20:47:27] <MirahezeLSBot>	 !log [universalomega@mwtask181] finished deploy of {'versions': ['1.45', '1.46'], 'upgrade_extensions': 'MirahezeMagic'} to all - SUCCESS in 127s
[20:47:31] <MirahezeLSBot>	 !log [universalomega@mwtask181] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/upgrade-wikis.php /srv/mediawiki/1.46/maintenance/run.php MirahezeMagic:UpgradeWiki --json=/srv/mediawiki/1.46.json (START)
[20:47:31] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[20:47:37] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[20:48:59] <MacFanBot>	 [02mediawiki-repos] 07lihaohong6 merged pull request #169: T13532: pin ConfirmEdit and DiscussionTools (07miraheze:03main...07lihaohong6:03captcha) 13https://github.com/miraheze/mediawiki-repos/pull/169
[20:48:59] <MacFanBot>	 [02mediawiki-repos] 07lihaohong6 pushed 1 new commit to 03main 13https://github.com/miraheze/mediawiki-repos/commit/31a28ed529f0383710797d7bc7eaf0c6fe2b4b22
[20:48:59] <MacFanBot>	 02mediawiki-repos/03main 07Peter Li 0331a28ed T13532: pin ConfirmEdit and DiscussionTools (#169)…
[20:57:24] <MirahezeLSBot>	 !log [universalomega@mwtask171] sudo -u www-data php /srv/mediawiki/1.46/maintenance/run.php MirahezeMagic:ChangeMediaWikiVersion --wiki=metawiki --closed --mwversion=1.46 (END - exit=0)
[20:57:32] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[20:58:46] <MirahezeLSBot>	 !log [universalomega@mwtask171] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/upgrade-wikis.php /srv/mediawiki/1.46/maintenance/run.php MirahezeMagic:UpgradeWiki --json=/srv/mediawiki/1.46.json (START)
[20:58:52] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[21:00:22] <MacFanBot>	 [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/6ce2fd0cafe7019146ef38a7fd3d8d2f068738cc
[21:00:23] <MacFanBot>	 02ssl/03main 07WikiTideBot 036ce2fd0 Bot: Auto-update domain lists
[21:23:29] <icinga-miraheze>	 PROBLEM - db151 Current Load on db151 is CRITICAL: LOAD CRITICAL - total load average: 40.40, 18.62, 8.14
[21:25:44] <MacFanBot>	 [02puppet] 07Universal-Omega created 03mwdeploy-fixes (+1 new commit) 13https://github.com/miraheze/puppet/commit/fdf2257bcb2b
[21:25:44] <MacFanBot>	 02puppet/03mwdeploy-fixes 07CosmicAlpha 03fdf2257 mwdeploy: some improvements/fixes
[21:25:50] <MacFanBot>	 [02puppet] 07Universal-Omega opened pull request #5051: mwdeploy: some improvements/fixes (03main...03mwdeploy-fixes) 13https://github.com/miraheze/puppet/pull/5051
[21:27:29] <icinga-miraheze>	 PROBLEM - db151 Current Load on db151 is WARNING: LOAD WARNING - total load average: 2.97, 10.66, 7.41
[21:29:29] <icinga-miraheze>	 RECOVERY - db151 Current Load on db151 is OK: LOAD OK - total load average: 1.65, 7.63, 6.69
[21:29:30] <MacFanBot>	 [02puppet] 07Universal-Omega pushed 1 new commit to 03mwdeploy-fixes 13https://github.com/miraheze/puppet/commit/34b3cae99523fb6cb8055750b1e88658502ce89e
[21:29:31] <MacFanBot>	 02puppet/03mwdeploy-fixes 07CosmicAlpha 0334b3cae Update tests
[21:31:00] <miraheze-github>	 miraheze/puppet - Universal-Omega the build passed.
[21:38:12] <icinga-miraheze>	 PROBLEM - mwtask181 Puppet on mwtask181 is CRITICAL: CRITICAL: Puppet has 4 failures. Last run 3 minutes ago with 4 failures. Failed resources (up to 3 shown): Exec[git_checkout_MediaWiki-REL1_45 ConfirmEdit],Exec[git_checkout_MediaWiki-REL1_45 DiscussionTools],Exec[git_checkout_MediaWiki-REL1_46 ConfirmEdit],Exec[git_checkout_MediaWiki-REL1_46 DiscussionTools]
[21:39:43] <MirahezeLSBot>	 !log [petramagna@test151] == deploy of {'versions': '1.46', 'upgrade_extensions': ['ConfirmEdit', 'DiscussionTools']} to test151
[21:39:45] <MirahezeLSBot>	 !log [petramagna@test151] DEPLOY ABORTED: Non-Zero Exit Code in prep, see output.
[21:39:49] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[21:39:53] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[21:41:01] <MirahezeLSBot>	 !log [petramagna@test151] DEPLOY ABORTED: Non-Zero Exit Code in prep, see output.
[21:41:06] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[21:41:20] <MirahezeLSBot>	 !log [petramagna@test151] DEPLOY ABORTED: Non-Zero Exit Code in prep, see output.
[21:41:24] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[21:50:08] <MacFanBot>	 [02puppet] 07Universal-Omega merged pull request #5051: mwdeploy: some improvements/fixes (03main...03mwdeploy-fixes) 13https://github.com/miraheze/puppet/pull/5051
[21:50:08] <MacFanBot>	 [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/4e8935670a20164bd2359ade656f34d66799ff08
[21:50:08] <MacFanBot>	 02puppet/03main 07CosmicAlpha 034e89356 mwdeploy: some improvements/fixes
[21:50:10] <MacFanBot>	 [02puppet] 07Universal-Omega 04deleted 03mwdeploy-fixes at 0334b3cae 13https://api.github.com/repos/miraheze/puppet/commit/34b3cae
[21:52:44] <MirahezeLSBot>	 !log [universalomega@test151] DEPLOY ABORTED: Non-Zero Exit Code in prep, see output.
[21:52:49] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[21:53:02] <MirahezeLSBot>	 !log [universalomega@test151] finished deploy of {'force': True, 'versions': 'all', 'upgrade_extensions': ['ConfirmEdit', 'DiscussionTools']} to test151 - SUCCESS in 9s
[21:53:07] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:14:57] <MirahezeLSBot>	 !log [somerandomdeveloper@speedscope211] sudo docker image prune -a
[22:15:03] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:17:20] <MacFanBot>	 [02puppet] 07Universal-Omega created 03mwdeploy-debug (+1 new commit) 13https://github.com/miraheze/puppet/commit/817a83d364b3
[22:17:20] <MacFanBot>	 02puppet/03mwdeploy-debug 07CosmicAlpha 03817a83d mwdeploy: add debug argument
[22:17:26] <MacFanBot>	 [02puppet] 07Universal-Omega opened pull request #5052: mwdeploy: add debug argument (03main...03mwdeploy-debug) 13https://github.com/miraheze/puppet/pull/5052
[22:17:40] <icinga-miraheze>	 RECOVERY - speedscope211 Disk Space on speedscope211 is OK: DISK OK - free space: / 1331MiB (10.5% inode=88%);
[22:18:07] <MacFanBot>	 [02puppet] 07Universal-Omega pushed 1 new commit to 03mwdeploy-debug 13https://github.com/miraheze/puppet/commit/939d58e3c3f6cb7fe8bb4feefe78d6d7f80b3818
[22:18:08] <MacFanBot>	 02puppet/03mwdeploy-debug 07CosmicAlpha 03939d58e Update
[22:18:36] <miraheze-github>	 miraheze/puppet - Universal-Omega the build has errored.
[22:19:42] <miraheze-github>	 miraheze/puppet - Universal-Omega the build passed.
[22:20:44] <MirahezeRelay>	 <abaddriverlol> !log ran "DELETE FROM AggregatedProfile WHERE type = 'HOURLY';" on speedscope211 for the sqlite DB
[22:20:49] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:21:36] <MirahezeRelay>	 <abaddriverlol> !log ran "VACUUM;" on speedscope211 for the sqlite DB
[22:21:40] <MacFanBot>	 [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/9a1447f258bcc03ce576bd3072fa2a831bd5830e
[22:21:40] <MacFanBot>	 02puppet/03main 07CosmicAlpha 039a1447f mwdeploy: add debug argument
[22:21:40] <MacFanBot>	 [02puppet] 07Universal-Omega merged pull request #5052: mwdeploy: add debug argument (03main...03mwdeploy-debug) 13https://github.com/miraheze/puppet/pull/5052
[22:21:41] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:21:41] <MacFanBot>	 [02puppet] 07Universal-Omega 04deleted 03mwdeploy-debug at 03939d58e 13https://api.github.com/repos/miraheze/puppet/commit/939d58e
[22:23:04] <miraheze-github>	 miraheze/puppet - Universal-Omega the build passed.
[22:24:30] <icinga-miraheze>	 RECOVERY - speedscope211 Puppet on speedscope211 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures
[22:26:10] <MacFanBot>	 [02puppet] 07SomeMWDev created 03update-speedscope (+1 new commit) 13https://github.com/miraheze/puppet/commit/bfb0aed6d1d9
[22:26:10] <MacFanBot>	 02puppet/03update-speedscope 07SomeRandomDeveloper 03bfb0aed speedscope: update image
[22:26:22] <MacFanBot>	 [02puppet] 07SomeMWDev opened pull request #5053: speedscope: update image (03main...03update-speedscope) 13https://github.com/miraheze/puppet/pull/5053
[22:27:19] <MacFanBot>	 [02puppet] 07SomeMWDev merged pull request #5053: speedscope: update image (03main...03update-speedscope) 13https://github.com/miraheze/puppet/pull/5053
[22:27:21] <MacFanBot>	 [02puppet] 07SomeMWDev pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/151c72a0f6c25d13b11a685f829b745beb541e28
[22:27:23] <MacFanBot>	 [02puppet] 07SomeMWDev 04deleted 03update-speedscope at 03bfb0aed 13https://api.github.com/repos/miraheze/puppet/commit/bfb0aed
[22:27:24] <MacFanBot>	 02puppet/03main 07SomeRandomDeveloper 03151c72a speedscope: update image (#5053)…
[22:45:54] <icinga-miraheze>	 PROBLEM - speedscope211 APT on speedscope211 is CRITICAL: APT CRITICAL: 80 packages available for upgrade (2 critical updates).
[23:00:51] <icinga-miraheze>	 RECOVERY - speedscope211 Check unit status of speedscope_hourly_aggregation on speedscope211 is OK: OK: Status of the systemd unit speedscope_hourly_aggregation
[23:30:19] <MacFanBot>	 [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/fc7ad721c636c33d69c3cac9efffd9b613ff8d39
[23:30:19] <MacFanBot>	 02ssl/03main 07WikiTideBot 03fc7ad72 Bot: Auto-update domain lists