[00:03:56] PROBLEM - mwtask181 Check unit status of update-static-tech-docs on mwtask181 is CRITICAL: CRITICAL: Status of the systemd unit update-static-tech-docs [00:23:59] RECOVERY - cp171 Disk Space on cp171 is OK: DISK OK - free space: / 86686MiB (19.1% inode=100%); [00:30:15] RECOVERY - cp191 Disk Space on cp191 is OK: DISK OK - free space: / 84102MiB (18.5% inode=100%); [00:31:07] RECOVERY - speedscope211 Check unit status of speedscope_daily_aggregation on speedscope211 is OK: OK: Status of the systemd unit speedscope_daily_aggregation [00:32:36] RECOVERY - cp201 Disk Space on cp201 is OK: DISK OK - free space: / 83522MiB (18.4% inode=100%); [00:46:17] PROBLEM - speedscope211 HTTPS on speedscope211 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Connection timed out after 10003 milliseconds [00:46:29] PROBLEM - speedscope211 speedscope.wikitide.net HTTPS on speedscope211 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:47:05] PROBLEM - speedscope211 SSH on speedscope211 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:48:31] PROBLEM - speedscope211 chrony on speedscope211 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [00:48:43] PROBLEM - speedscope211 APT on speedscope211 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [00:49:03] PROBLEM - speedscope211 PowerDNS Recursor on speedscope211 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [00:49:10] PROBLEM - speedscope211 speedscope 3000 check on speedscope211 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [00:49:15] PROBLEM - speedscope211 Disk Space on speedscope211 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [00:49:39] PROBLEM - speedscope211 conntrack_table_size on speedscope211 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [00:49:39] PROBLEM - speedscope211 Check unit status of speedscope_hourly_aggregation on speedscope211 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [00:49:40] PROBLEM - speedscope211 Puppet on speedscope211 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [00:50:01] PROBLEM - speedscope211 nftables_active on speedscope211 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [00:50:07] PROBLEM - speedscope211 Check unit status of speedscope_daily_aggregation on speedscope211 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [00:50:25] PROBLEM - speedscope211 Current Load on speedscope211 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [01:04:54] RECOVERY - speedscope211 speedscope.wikitide.net HTTPS on speedscope211 is OK: HTTP OK: HTTP/1.1 200 OK - 8781 bytes in 2.559 second response time [01:04:58] RECOVERY - speedscope211 chrony on speedscope211 is OK: chrony OK: System Time 0.000009593 seconds slow of NTP time, Stratum 4 [01:05:15] RECOVERY - speedscope211 HTTPS on speedscope211 is OK: HTTP OK: HTTP/2 200 - 354 bytes in 0.011 second response time [01:05:42] RECOVERY - speedscope211 APT on speedscope211 is OK: APT OK: 78 packages available for upgrade (0 critical updates). [01:05:46] RECOVERY - speedscope211 SSH on speedscope211 is OK: SSH OK - OpenSSH_10.0p2 (protocol 2.0) [01:05:53] RECOVERY - speedscope211 PowerDNS Recursor on speedscope211 is OK: DNS OK: 0.052 seconds response time. speedscope211.fsslc.wtnet returns 10.0.21.172 [01:05:57] RECOVERY - speedscope211 Disk Space on speedscope211 is OK: DISK OK - free space: / 4466MiB (35.3% inode=83%); [01:05:59] RECOVERY - speedscope211 speedscope 3000 check on speedscope211 is OK: TCP OK - 0.000 second response time on 127.0.0.1 port 3000 [01:06:20] RECOVERY - speedscope211 conntrack_table_size on speedscope211 is OK: OK: nf_conntrack is 0 % full [01:06:22] RECOVERY - speedscope211 Check unit status of speedscope_hourly_aggregation on speedscope211 is OK: OK: Status of the systemd unit speedscope_hourly_aggregation [01:06:31] RECOVERY - speedscope211 nftables_active on speedscope211 is OK: OK nftables input default policy is set [01:06:38] RECOVERY - speedscope211 Puppet on speedscope211 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:35:35] PROBLEM - speedscope211 Current Load on speedscope211 is WARNING: LOAD WARNING - total load average: 0.04, 0.20, 7.26 [01:37:30] RECOVERY - speedscope211 Current Load on speedscope211 is OK: LOAD OK - total load average: 0.00, 0.13, 6.41 [08:30:23] [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/4205b92e507f21a44ed8afb26e48b4c054d410a5 [08:30:23] 02ssl/03main 07WikiTideBot 034205b92 Bot: Auto-update domain lists [10:05:04] PROBLEM - wc.miraheze.org on sslhost is WARNING: WARNING - Certificate 'miraheze.org' expires in 30 day(s) (Mon 19 Oct 2026 10:02:41 AM GMT +0000). [13:30:24] [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/6c8e16f895b37ea67f3281607195b3dce3d00ce8 [13:30:24] 02ssl/03main 07WikiTideBot 036c8e16f Bot: Auto-update domain lists [13:33:00] PROBLEM - mwtask151 HTTPS on mwtask151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Connection timed out after 10004 milliseconds [13:33:22] PROBLEM - mwtask151 MediaWiki Rendering on mwtask151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:33:35] PROBLEM - mwtask151 SSH on mwtask151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:33:58] PROBLEM - mwtask151 videoscaler.svc.fsslc.wtnet HTTP on mwtask151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10006 milliseconds with 0 bytes received [13:34:27] PROBLEM - mwtask151 jobrunner-high.svc.fsslc.wtnet HTTP on mwtask151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10006 milliseconds with 0 bytes received [13:34:41] PROBLEM - mwtask151 jobrunner.svc.fsslc.wtnet HTTP on mwtask151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10005 milliseconds with 0 bytes received [13:35:32] PROBLEM - mwtask151 Disk Space on mwtask151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:35:41] PROBLEM - mwtask151 PowerDNS Recursor on mwtask151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:35:46] PROBLEM - mwtask151 APT on mwtask151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:35:47] PROBLEM - mwtask151 php-fpm on mwtask151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:36:08] PROBLEM - mwtask151 Current Load on mwtask151 is CRITICAL: LOAD CRITICAL - total load average: 122.60, 83.38, 39.44 [13:36:26] RECOVERY - mwtask151 jobrunner-high.svc.fsslc.wtnet HTTP on mwtask151 is OK: HTTP OK: HTTP/1.1 204 No Content - 167 bytes in 0.001 second response time [13:36:27] PROBLEM - mwtask151 Puppet on mwtask151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:36:29] PROBLEM - mwtask151 chrony on mwtask151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:36:38] RECOVERY - mwtask151 jobrunner.svc.fsslc.wtnet HTTP on mwtask151 is OK: HTTP OK: HTTP/1.1 204 No Content - 168 bytes in 0.002 second response time [13:37:02] RECOVERY - mwtask151 HTTPS on mwtask151 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 344 bytes in 0.082 second response time [13:37:28] RECOVERY - mwtask151 Disk Space on mwtask151 is OK: DISK OK - free space: / 203489MiB (86.0% inode=96%); [13:37:31] RECOVERY - mwtask151 MediaWiki Rendering on mwtask151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.530 second response time [13:37:36] RECOVERY - mwtask151 SSH on mwtask151 is OK: SSH OK - OpenSSH_10.0p2 (protocol 2.0) [13:37:40] RECOVERY - mwtask151 PowerDNS Recursor on mwtask151 is OK: DNS OK: 0.059 seconds response time. mwtask151.fsslc.wtnet returns 10.0.15.150 [13:37:46] RECOVERY - mwtask151 php-fpm on mwtask151 is OK: PROCS OK: 25 processes with command name 'php-fpm8.4' [13:37:49] RECOVERY - mwtask151 APT on mwtask151 is OK: APT OK: 190 packages available for upgrade (0 critical updates). [13:37:59] RECOVERY - mwtask151 videoscaler.svc.fsslc.wtnet HTTP on mwtask151 is OK: HTTP OK: HTTP/1.1 204 No Content - 167 bytes in 0.002 second response time [13:38:28] RECOVERY - mwtask151 chrony on mwtask151 is OK: chrony OK: System Time 0.000134311 seconds fast of NTP time, Stratum 4 [13:38:29] RECOVERY - mwtask151 Puppet on mwtask151 is OK: OK: Puppet is currently enabled, last run 24 minutes ago with 0 failures [13:46:01] PROBLEM - mwtask151 Current Load on mwtask151 is WARNING: LOAD WARNING - total load average: 4.78, 15.60, 23.07 [13:50:01] RECOVERY - mwtask151 Current Load on mwtask151 is OK: LOAD OK - total load average: 4.23, 9.41, 18.82 [14:12:50] [02mw-config] 07dependabot[bot] created 03dependabot/composer/phpunit/phpunit-13.3.4 (+1 new commit) 13https://github.com/miraheze/mw-config/commit/919145757e95 [14:12:50] 02mw-config/03dependabot/composer/phpunit/phpunit-13.3.4 07dependabot[bot] 039191457 Update phpunit/phpunit requirement from 13.3.3 to 13.3.4… [14:12:51] [02mw-config] 07dependabot[bot] added the label 'dependencies' to pull request #6551 (Update phpunit/phpunit requirement from 13.3.3 to 13.3.4) 13https://github.com/miraheze/mw-config/pull/6551 [14:12:52] [02mw-config] 07dependabot[bot] added the label 'php' to pull request #6551 (Update phpunit/phpunit requirement from 13.3.3 to 13.3.4) 13https://github.com/miraheze/mw-config/pull/6551 [14:12:54] [02mw-config] 07dependabot[bot] opened pull request #6551: Update phpunit/phpunit requirement from 13.3.3 to 13.3.4 (03main...03dependabot/composer/phpunit/phpunit-13.3.4) 13https://github.com/miraheze/mw-config/pull/6551 [14:12:56] [02mw-config] 07dependabot[bot] added the label 'php' to pull request #6551 (Update phpunit/phpunit requirement from 13.3.3 to 13.3.4) 13https://github.com/miraheze/mw-config/pull/6551 [14:12:58] [02mw-config] 07dependabot[bot] added the label 'dependencies' to pull request #6551 (Update phpunit/phpunit requirement from 13.3.3 to 13.3.4) 13https://github.com/miraheze/mw-config/pull/6551 [14:14:03] miraheze/mw-config - dependabot[bot] the build passed. [15:30:25] [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/02d479c1202bb3a6f96313339b95f349d27b258b [15:30:25] 02ssl/03main 07WikiTideBot 0302d479c Bot: Auto-update domain lists [16:45:54] [02mw-config] 07Universal-Omega merged 07dependabot[bot]'s pull request #6551: Update phpunit/phpunit requirement from 13.3.3 to 13.3.4 (03main...03dependabot/composer/phpunit/phpunit-13.3.4) 13https://github.com/miraheze/mw-config/pull/6551 [16:45:54] [02mw-config] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/mw-config/commit/841ba4b58cf3a190dc076d4a4bfba284fb763c9d [16:45:56] 02mw-config/03main 07dependabot[bot] 03841ba4b Update phpunit/phpunit requirement from 13.3.3 to 13.3.4 (#6551)… [16:45:58] [02mw-config] 07Universal-Omega 04deleted 03dependabot/composer/phpunit/phpunit-13.3.4 at 039191457 13https://api.github.com/repos/miraheze/mw-config/commit/9191457 [16:47:07] miraheze/mw-config - Universal-Omega the build passed. [16:55:40] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/dd27bbdc1e7e570bdd4e1c514280451d85a15745 [16:55:40] 02puppet/03main 07CosmicAlpha 03dd27bbd mattermost: update to 11.11.0 [19:06:59] [02MirahezeMagic] 07Universal-Omega created 03upgradewiki-check (+1 new commit) 13https://github.com/miraheze/MirahezeMagic/commit/6997f0588850 [19:06:59] 02MirahezeMagic/03upgradewiki-check 07CosmicAlpha 036997f05 UpgradeWiki: add a check if it has already been upgraded [19:07:06] [02MirahezeMagic] 07Universal-Omega opened pull request #669: UpgradeWiki: add a check if it has already been upgraded (03main...03upgradewiki-check) 13https://github.com/miraheze/MirahezeMagic/pull/669 [19:10:21] [02MirahezeMagic] 07Universal-Omega pushed 1 new commit to 03upgradewiki-check 13https://github.com/miraheze/MirahezeMagic/commit/0a7768e41e882617d400ded9d96282050f28d7f4 [19:10:21] 02MirahezeMagic/03upgradewiki-check 07CosmicAlpha 030a7768e Add output [19:10:30] miraheze/MirahezeMagic - Universal-Omega the build has errored. [19:14:43] miraheze/MirahezeMagic - Universal-Omega the build has errored. [19:19:20] [02MirahezeMagic] 07Universal-Omega pushed 1 new commit to 03upgradewiki-check 13https://github.com/miraheze/MirahezeMagic/commit/2dd7ffddfaf4227b5e4141b6f81a35b894dd1b8b [19:19:21] 02MirahezeMagic/03upgradewiki-check 07CosmicAlpha 032dd7ffd Fix order [19:25:20] miraheze/MirahezeMagic - Universal-Omega the build passed. [19:33:00] [02MirahezeMagic] 07Universal-Omega merged pull request #669: UpgradeWiki: add a check if it has already been upgraded (03main...03upgradewiki-check) 13https://github.com/miraheze/MirahezeMagic/pull/669 [19:33:00] [02MirahezeMagic] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/MirahezeMagic/commit/aa1fd821939c817be89bfb227365a6d102c97f33 [19:33:00] 02MirahezeMagic/03main 07CosmicAlpha 03aa1fd82 UpgradeWiki: add a check if it has already been upgraded [19:33:02] [02MirahezeMagic] 07Universal-Omega 04deleted 03upgradewiki-check at 032dd7ffd 13https://api.github.com/repos/miraheze/MirahezeMagic/commit/2dd7ffd [19:39:00] miraheze/MirahezeMagic - Universal-Omega the build passed. [19:49:31] [02ManageWiki] 07Universal-Omega pushed 1 new commit to 03emptygroups 13https://github.com/miraheze/ManageWiki/commit/ff1111772f4a1929b29952342d28d6025caa9610 [19:49:31] 02ManageWiki/03emptygroups 07CosmicAlpha 03ff11117 - [19:50:01] [02ManageWiki] 07Universal-Omega pushed 1 new commit to 03emptygroups 13https://github.com/miraheze/ManageWiki/commit/58ef2cb112c7f1a0becb2a959598829d74b1f78a [19:50:01] 02ManageWiki/03emptygroups 07CosmicAlpha 0358ef2cb - [19:51:15] [02ManageWiki] 07Universal-Omega pushed 1 new commit to 03emptygroups 13https://github.com/miraheze/ManageWiki/commit/35c915175d715fb4bdb4f0d40fcc161076ae0b49 [19:51:15] 02ManageWiki/03emptygroups 07CosmicAlpha 0335c9151 - [21:00:27] [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/c2fd06eb1ef198cef0e1b62362cef8606bfa26ee [21:00:27] 02ssl/03main 07WikiTideBot 03c2fd06e Bot: Auto-update domain lists [21:16:30] [1/4] https://cdn.discordapp.com/attachments/808001911868489748/1550616530419449906/image.jpg?ex=6aaefbad&is=6aadaa2d&hm=0135d0e69b0120452d44946ccbb042449fabf6697819c821440393cebc949de4& [21:16:31] [2/4] https://cdn.discordapp.com/attachments/808001911868489748/1550616530876895373/image.jpg?ex=6aaefbae&is=6aadaa2e&hm=63ff3d14353726b02a33c3f799b9d18daabde6c19c6cad72d5a4521a87dcd83f& [21:16:31] [3/4] https://cdn.discordapp.com/attachments/808001911868489748/1550616531254116402/image.jpg?ex=6aaefbae&is=6aadaa2e&hm=039dcf8c609f79ac668ac51c39412aa0a66dc4266adfae3d225701d969df10f4& [21:16:31] [4/4] https://cdn.discordapp.com/attachments/808001911868489748/1550616531711561819/image.jpg?ex=6aaefbae&is=6aadaa2e&hm=a1e0418b6afc35f62b7e2c96c7429bdd610642836f0da2ddfad04022ab79e90c& [21:30:23] [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/0538a5f34d95c8a4960d687c651d92db02dffa42 [21:30:23] 02ssl/03main 07WikiTideBot 030538a5f Bot: Auto-update domain lists [21:53:04] RECOVERY - wc.miraheze.org on sslhost is OK: OK - Certificate 'miraheze.org' will expire on Thu 17 Dec 2026 09:51:50 PM GMT +0000. [23:39:51] PROBLEM - cp191 Disk Space on cp191 is WARNING: DISK WARNING - free space: / 44453MiB (9.8% inode=100%); [23:42:00] PROBLEM - cp201 Disk Space on cp201 is WARNING: DISK WARNING - free space: / 45215MiB (10.0% inode=100%); [23:43:44] PROBLEM - cp171 Disk Space on cp171 is WARNING: DISK WARNING - free space: / 45262MiB (10.0% inode=100%); [23:49:51] RECOVERY - cp191 Disk Space on cp191 is OK: DISK OK - free space: / 86894MiB (19.1% inode=100%);