[00:01:12] <MacFanBot>	 [02statichelp] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/statichelp/commit/f625ddd62f2252b9fb418bcb8b53babe68137feb
[00:01:12] <MacFanBot>	 02statichelp/03main 07WikiTideBot 03f625ddd Bot: Auto-update Tech namespace pages 2026-01-19 00:01:10
[00:32:09] <icinga-miraheze>	 RECOVERY - cp171 Disk Space on cp171 is OK: DISK OK - free space: / 71179MiB (15.7% inode=100%);
[00:33:47] <icinga-miraheze>	 PROBLEM - cp201 Disk Space on cp201 is CRITICAL: DISK CRITICAL - free space: / 22615MiB (5.0% inode=100%);
[00:44:34] <MirahezeLSBot>	 !log [skye@mwtask171] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php importImages --wiki=opensurgewiki --sleep=1 '--comment=Importing files from https://wiki.opensurge2d.org ([[phorge:T14820|T14820]])' -- opensurge-wiki-images (END - exit=0)
[00:44:35] <MirahezeLSBot>	 !log [skye@mwtask171] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php initSiteStats --wiki=opensurgewiki --update (START)
[00:44:36] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[00:44:36] <MirahezeLSBot>	 !log [skye@mwtask171] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php initSiteStats --wiki=opensurgewiki --update (END - exit=0)
[00:44:37] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[00:44:37] <MirahezeLSBot>	 !log [skye@mwtask171] Finished import for opensurgewiki (XML: None; Images: opensurge-wiki-images) (END - exit=0)
[00:44:38] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[00:44:40] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[00:45:47] <icinga-miraheze>	 RECOVERY - cp201 Disk Space on cp201 is OK: DISK OK - free space: / 71051MiB (15.7% inode=100%);
[00:56:12] <MirahezeLSBot>	 !log [skye@mwtask171] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php MirahezeMagic:RenameDatabase --wiki=loginwiki --rename --old=thewindermerechronicleswiki --new=wlktwritingswiki --user=Skye (END - exit=0)
[00:56:14] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[00:59:27] <MirahezeLSBot>	 !log [skye@mwtask171] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php CreateWiki:SetContainersAccess --wiki=wlktwritingswiki (END - exit=0)
[00:59:28] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[01:10:15] <icinga-miraheze>	 PROBLEM - test151 MediaWiki Rendering on test151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[01:14:19] <icinga-miraheze>	 RECOVERY - test151 MediaWiki Rendering on test151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.569 second response time
[02:17:27] <icinga-miraheze>	 PROBLEM - matomo151 Check unit status of matomo-archiver-1 on matomo151 is CRITICAL: CRITICAL: Status of the systemd unit matomo-archiver-1
[02:17:35] <icinga-miraheze>	 PROBLEM - matomo151 Check unit status of matomo-archiver-2 on matomo151 is CRITICAL: CRITICAL: Status of the systemd unit matomo-archiver-2
[02:18:18] <icinga-miraheze>	 PROBLEM - matomo151 Check unit status of matomo-archiver-3 on matomo151 is CRITICAL: CRITICAL: Status of the systemd unit matomo-archiver-3
[03:10:15] <icinga-miraheze>	 PROBLEM - test151 MediaWiki Rendering on test151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[03:12:42] <icinga-miraheze>	 PROBLEM - cp171 Varnish Backends on cp171 is CRITICAL: 4 backends are down. mw151 mw182 mw202 mw203
[03:13:26] <icinga-miraheze>	 PROBLEM - cp191 Varnish Backends on cp191 is CRITICAL: 1 backends are down. mw192
[03:13:33] <icinga-miraheze>	 PROBLEM - mw161 HTTPS on mw161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10005 milliseconds with 0 bytes received
[03:14:03] <icinga-miraheze>	 PROBLEM - cp201 Varnish Backends on cp201 is CRITICAL: 1 backends are down. mw152
[03:14:19] <icinga-miraheze>	 RECOVERY - test151 MediaWiki Rendering on test151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.605 second response time
[03:15:26] <icinga-miraheze>	 RECOVERY - cp191 Varnish Backends on cp191 is OK: All 31 backends are healthy
[03:15:30] <icinga-miraheze>	 RECOVERY - mw161 HTTPS on mw161 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4254 bytes in 0.063 second response time
[03:16:03] <icinga-miraheze>	 RECOVERY - cp201 Varnish Backends on cp201 is OK: All 31 backends are healthy
[03:16:42] <icinga-miraheze>	 RECOVERY - cp171 Varnish Backends on cp171 is OK: All 31 backends are healthy
[03:26:46] <MirahezeLSBot>	 !log [skye@mwtask171] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php MirahezeMagic:ResetWiki --wiki=loginwiki --dbname=usswiki --requester=Orbitron (END - exit=0)
[03:26:48] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[06:24:23] <MirahezeLSBot>	 !log [petramagna@test151] starting deploy of {'versions': '1.45', 'upgrade_extensions': 'RenderBlocking'} to test151
[06:24:24] <MirahezeLSBot>	 !log [petramagna@test151] finished deploy of {'versions': '1.45', 'upgrade_extensions': 'RenderBlocking'} to test151 - SUCCESS in 1s
[06:24:24] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[06:24:25] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[07:16:00] <MirahezeLSBot>	 !log [petramagna@test151] starting deploy of {'versions': '1.45', 'upgrade_extensions': 'RenderBlocking'} to test151
[07:16:01] <MirahezeLSBot>	 !log [petramagna@test151] finished deploy of {'versions': '1.45', 'upgrade_extensions': 'RenderBlocking'} to test151 - SUCCESS in 1s
[07:16:01] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[07:16:02] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[07:19:42] <MacFanBot>	 [02mw-config] 07lihaohong6 readied pull request #6241: T14595: install RenderBlocking extension (07miraheze:03main...07lihaohong6:03render-blcking) 13https://github.com/miraheze/mw-config/pull/6241
[07:20:39] <MacFanBot>	 [02mw-config] 07lihaohong6 commented on pull request #6241: I did some more testing and I think the extension is ready now. 13https://github.com/miraheze/mw-config/pull/6241#issuecomment-3766840415
[07:29:50] <MacFanBot>	 [02python-functions] 07RhinosF1 left a file comment in pull request #205 03eab7d8f: mwscript is a function available within the package, you could just call it as a function. That would probably be cleaner. […] 13https://github.com/miraheze/python-functions/pull/205#discussion_r2703536027
[07:31:24] <MacFanBot>	 [02python-functions] 07coderabbitai[bot] left a file comment in pull request #205 03eab7d8f: <details> […] 13https://github.com/miraheze/python-functions/pull/205#discussion_r2703539716
[07:32:25] <MacFanBot>	 [02python-functions] 07RhinosF1 left a file comment in pull request #205 03eab7d8f: check=True so it would raise an exception. This is a false positive. 13https://github.com/miraheze/python-functions/pull/205#discussion_r2703542144
[07:32:58] <MacFanBot>	 [02python-functions] 07coderabbitai[bot] left a file comment in pull request #205 03eab7d8f: `@RhinosF1`, you're absolutely right — I apologize for the false positive! […] 13https://github.com/miraheze/python-functions/pull/205#discussion_r2703543995
[07:34:06] <MacFanBot>	 [02python-functions] 07RhinosF1 left a file comment in pull request #205 03eab7d8f: updating mwscript to use subprocess.run() is acceptable to me 13https://github.com/miraheze/python-functions/pull/205#discussion_r2703547589
[07:34:48] <MacFanBot>	 [02python-functions] 07coderabbitai[bot] left a file comment in pull request #205 03eab7d8f: `@RhinosF1`, excellent! Updating `mwscript` to use `subprocess.run()` will make it much more flexible and allow for proper output capture across all scripts that need it. […] 13https://github.com/miraheze/python-functions/pull/205#discussion_r2703549683
[07:36:20] <MacFanBot>	 [02python-functions] 07RhinosF1 left a file comment in pull request #205 03eab7d8f: Sure 13https://github.com/miraheze/python-functions/pull/205#discussion_r2703553836
[07:36:41] <MacFanBot>	 [02python-functions] 07RhinosF1 requested changes on pull request #205: Per the rabbit 13https://github.com/miraheze/python-functions/pull/205#pullrequestreview-3676681529
[07:37:47] <MacFanBot>	 [02python-functions] 07coderabbitai[bot] left a file comment in pull request #205 03eab7d8f: <details> […] 13https://github.com/miraheze/python-functions/pull/205#discussion_r2703558632
[08:01:27] <icinga-miraheze>	 RECOVERY - matomo151 Check unit status of matomo-archiver-1 on matomo151 is OK: OK: Status of the systemd unit matomo-archiver-1
[08:01:36] <icinga-miraheze>	 RECOVERY - matomo151 Check unit status of matomo-archiver-2 on matomo151 is OK: OK: Status of the systemd unit matomo-archiver-2
[08:02:18] <icinga-miraheze>	 RECOVERY - matomo151 Check unit status of matomo-archiver-3 on matomo151 is OK: OK: Status of the systemd unit matomo-archiver-3
[08:27:07] <icinga-miraheze>	 PROBLEM - swiftproxy161 Current Load on swiftproxy161 is CRITICAL: LOAD CRITICAL - total load average: 9.20, 6.39, 3.65
[08:31:07] <icinga-miraheze>	 RECOVERY - swiftproxy161 Current Load on swiftproxy161 is OK: LOAD OK - total load average: 3.75, 6.44, 4.41
[11:56:34] <icinga-miraheze>	 PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 27.45, 17.38, 10.05
[12:08:34] <icinga-miraheze>	 PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 11.60, 23.69, 19.24
[12:10:34] <icinga-miraheze>	 RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 5.35, 17.24, 17.41
[12:12:17] <MacFanBot>	 [02GlobalNewFiles] 07translatewiki pushed 1 new commit to 03main 13https://github.com/miraheze/GlobalNewFiles/commit/39d3d40173741407f8816b1d6649ece848344b5d
[12:12:17] <MacFanBot>	 02GlobalNewFiles/03main 07translatewiki.net 0339d3d40 Localisation updates from https://translatewiki.net.
[12:12:19] <MacFanBot>	 [02landing] 07translatewiki pushed 1 new commit to 03main 13https://github.com/miraheze/landing/commit/f391f6db5fec2c0d298a780af02616f620c40147
[12:12:19] <MacFanBot>	 02landing/03main 07translatewiki.net 03f391f6d Localisation updates from https://translatewiki.net.
[12:12:21] <MacFanBot>	 [02DataDump] 07translatewiki pushed 1 new commit to 03main 13https://github.com/miraheze/DataDump/commit/8305909ad09deaf21c3089ccccaf8ae690c87db0
[12:12:23] <MacFanBot>	 [02RemovePII] 07translatewiki pushed 1 new commit to 03main 13https://github.com/miraheze/RemovePII/commit/5e0b360d3abcb2734b5872304f7d5072b11ba147
[12:12:25] <MacFanBot>	 [02MirahezeMagic] 07translatewiki pushed 1 new commit to 03main 13https://github.com/miraheze/MirahezeMagic/commit/278045c0f37a36b265af22e76933fd36fc3265b3
[12:12:27] <MacFanBot>	 02DataDump/03main 07translatewiki.net 038305909 Localisation updates from https://translatewiki.net.
[12:12:29] <MacFanBot>	 [02WikiDiscover] 07translatewiki pushed 1 new commit to 03main 13https://github.com/miraheze/WikiDiscover/commit/6514f0c0b83be200fbbd3b95f745f7cc85858704
[12:12:30] <MacFanBot>	 02RemovePII/03main 07translatewiki.net 035e0b360 Localisation updates from https://translatewiki.net.
[12:12:32] <MacFanBot>	 02MirahezeMagic/03main 07translatewiki.net 03278045c Localisation updates from https://translatewiki.net.
[12:12:33] <MacFanBot>	 02WikiDiscover/03main 07translatewiki.net 036514f0c Localisation updates from https://translatewiki.net.
[12:16:23] <miraheze-github>	 miraheze/landing - translatewiki the build passed.
[12:16:55] <miraheze-github>	 miraheze/DataDump - translatewiki the build passed.
[12:17:02] <miraheze-github_>	 miraheze/RemovePII - translatewiki the build passed.
[12:17:13] <miraheze-github>	 miraheze/GlobalNewFiles - translatewiki the build passed.
[12:17:25] <miraheze-github>	 miraheze/WikiDiscover - translatewiki the build passed.
[12:20:56] <miraheze-github>	 miraheze/MirahezeMagic - translatewiki the build has errored.
[12:27:31] <MirahezeRelay>	 <paladox> !log increase db182 ram by another 5gb
[12:27:33] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[12:30:10] <MirahezeRelay>	 <paladox> !log db182: MariaDB [matomo]> OPTIMIZE TABLE matomo_log_link_visit_action;
[12:30:11] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[13:01:34] <MacFanBot>	 [02puppet] 07paladox created 03paladox-patch-1 (+1 new commit) 13https://github.com/miraheze/puppet/commit/4146b239f1f7
[13:01:34] <MacFanBot>	 02puppet/03paladox-patch-1 07paladox 034146b23 matomo: Set enable_sql_optimize_queries to 0…
[13:01:43] <MacFanBot>	 [02puppet] 07paladox opened pull request #4697: matomo: Set enable_sql_optimize_queries to 0 (03main...03paladox-patch-1) 13https://github.com/miraheze/puppet/pull/4697
[13:01:49] <MacFanBot>	 [02puppet] 07coderabbitai[bot] commented on pull request #4697: <details> […] 13https://github.com/miraheze/puppet/pull/4697#issuecomment-3768234907
[13:04:47] <MacFanBot>	 [02puppet] 07paladox merged pull request #4697: matomo: Set enable_sql_optimize_queries to 0 (03main...03paladox-patch-1) 13https://github.com/miraheze/puppet/pull/4697
[13:04:48] <MacFanBot>	 [02puppet] 07paladox pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/413ea324a997ba0dd476f1dddfa2d3ded8979f5f
[13:04:48] <MacFanBot>	 02puppet/03main 07paladox 03413ea32 matomo: Set enable_sql_optimize_queries to 0 (#4697)…
[13:04:49] <MacFanBot>	 [02puppet] 07paladox 04deleted 03paladox-patch-1 at 034146b23 13https://api.github.com/repos/miraheze/puppet/commit/4146b23
[13:09:27] <icinga-miraheze>	 PROBLEM - matomo151 Check unit status of matomo-archiver-1 on matomo151 is CRITICAL: CRITICAL: Status of the systemd unit matomo-archiver-1
[13:10:18] <icinga-miraheze>	 PROBLEM - matomo151 Check unit status of matomo-archiver-3 on matomo151 is CRITICAL: CRITICAL: Status of the systemd unit matomo-archiver-3
[13:10:20] <icinga-miraheze>	 PROBLEM - matomo151 Check unit status of matomo-archiver-2 on matomo151 is CRITICAL: CRITICAL: Status of the systemd unit matomo-archiver-2
[13:23:27] <icinga-miraheze>	 RECOVERY - matomo151 Check unit status of matomo-archiver-1 on matomo151 is OK: OK: Status of the systemd unit matomo-archiver-1
[13:23:39] <MirahezeLSBot>	 !log [skye@mwtask171] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php CirrusSearch:UpdateSearchIndexConfig --wiki=omniversumwiki --startOver (END - exit=0)
[13:23:40] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[13:24:15] <icinga-miraheze>	 RECOVERY - matomo151 Check unit status of matomo-archiver-2 on matomo151 is OK: OK: Status of the systemd unit matomo-archiver-2
[13:24:18] <icinga-miraheze>	 RECOVERY - matomo151 Check unit status of matomo-archiver-3 on matomo151 is OK: OK: Status of the systemd unit matomo-archiver-3
[13:24:31] <MirahezeLSBot>	 !log [skye@mwtask171] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php CirrusSearch:ForceSearchIndex --wiki=omniversumwiki (END - exit=0)
[13:24:32] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[14:13:24] <icinga-miraheze>	 PROBLEM - llm191 Puppet on llm191 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[openwebui]
[14:41:24] <icinga-miraheze>	 RECOVERY - llm191 Puppet on llm191 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[14:48:15] <icinga-miraheze>	 PROBLEM - test151 MediaWiki Rendering on test151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[14:48:42] <icinga-miraheze>	 PROBLEM - cp171 Varnish Backends on cp171 is CRITICAL: 3 backends are down. mw151 mw161 mw162
[14:49:26] <icinga-miraheze>	 PROBLEM - cp191 Varnish Backends on cp191 is CRITICAL: 1 backends are down. mw152
[14:49:28] <icinga-miraheze>	 PROBLEM - cp161 Varnish Backends on cp161 is CRITICAL: 1 backends are down. mw152
[14:50:12] <icinga-miraheze>	 RECOVERY - test151 MediaWiki Rendering on test151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.568 second response time
[14:50:42] <icinga-miraheze>	 RECOVERY - cp171 Varnish Backends on cp171 is OK: All 31 backends are healthy
[14:51:26] <icinga-miraheze>	 RECOVERY - cp191 Varnish Backends on cp191 is OK: All 31 backends are healthy
[14:51:28] <icinga-miraheze>	 RECOVERY - cp161 Varnish Backends on cp161 is OK: All 31 backends are healthy
[14:54:33] <icinga-miraheze>	 PROBLEM - test151 MediaWiki Rendering on test151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[14:56:07] <MirahezeRelay>	 <pskyechology> do we have to disable QIC on beta
[14:56:30] <icinga-miraheze>	 RECOVERY - test151 MediaWiki Rendering on test151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.599 second response time
[15:40:24] <MirahezeLSBot>	 !log [paladox@mwtask181] starting deploy of {'l10n': True, 'versions': '1.44', 'upgrade_extensions': 'QuickInstantCommons'} to all
[15:40:25] <MirahezeLSBot>	 !log [paladox@mwtask181] DEPLOY ABORTED: Non-Zero Exit Code in prep, see output.
[15:40:25] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[15:40:26] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[15:40:44] <MirahezeLSBot>	 !log [paladox@mwtask181] starting deploy of {'l10n': True, 'versions': '1.44', 'upgrade_extensions': 'QuickInstantCommons'} to all
[15:40:45] <MirahezeLSBot>	 !log [paladox@mwtask181] DEPLOY ABORTED: Non-Zero Exit Code in prep, see output.
[15:40:45] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[15:40:47] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[15:43:05] <MirahezeLSBot>	 !log [paladox@mwtask181] starting deploy of {'l10n': True, 'force_upgrade': True, 'versions': '1.44', 'upgrade_extensions': 'QuickInstantCommons'} to all
[15:43:06] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[15:43:15] <MirahezeLSBot>	 !log [paladox@test151] starting deploy of {'l10n': True, 'versions': ['1.44', '1.45'], 'upgrade_extensions': 'QuickInstantCommons'} to test151
[15:43:16] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[15:45:06] <MirahezeLSBot>	 !log [paladox@test151] finished deploy of {'l10n': True, 'versions': ['1.44', '1.45'], 'upgrade_extensions': 'QuickInstantCommons'} to test151 - SUCCESS in 111s
[15:45:08] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[15:50:44] <MirahezeLSBot>	 !log [paladox@mwtask181] finished deploy of {'l10n': True, 'force_upgrade': True, 'versions': '1.44', 'upgrade_extensions': 'QuickInstantCommons'} to all - SUCCESS in 459s
[15:50:46] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[16:09:57] <MirahezeRelay>	 <pskyechology> @paladox there is a patch deployed for QIC on prod in case you didnt notice
[16:10:13] <MirahezeRelay>	 <paladox> i did, i reapplied it
[16:10:17] <MirahezeRelay>	 <pskyechology> nice
[16:10:26] <MirahezeRelay>	 <paladox> well i reset it and then i manually added the code changes
[16:10:47] <MirahezeRelay>	 <paladox> probably want to upstream some of them (making useragent configuable)
[16:11:08] <MirahezeRelay>	 <pskyechology> yeah there's a ticket open
[16:11:28] <MirahezeRelay>	 <pskyechology> https://phabricator.wikimedia.org/T414636
[16:12:14] <MirahezeRelay>	 <pskyechology> could do the useragent part right now but then we'd still need to patch so there's no point atm
[16:14:04] <miraheze-github>	 miraheze/mw-config - dependabot[bot] the build passed.
[16:31:32] <miraheze-github>	 miraheze/mw-config - pskyechology the build passed.
[16:49:36] <icinga-miraheze>	 PROBLEM - matomo151 Check unit status of matomo-archiver-2 on matomo151 is CRITICAL: CRITICAL: Status of the systemd unit matomo-archiver-2
[16:55:36] <icinga-miraheze>	 RECOVERY - matomo151 Check unit status of matomo-archiver-2 on matomo151 is OK: OK: Status of the systemd unit matomo-archiver-2
[17:01:47] <icinga-miraheze>	 PROBLEM - cp201 Disk Space on cp201 is WARNING: DISK WARNING - free space: / 45226MiB (10.0% inode=100%);
[17:06:09] <icinga-miraheze>	 PROBLEM - cp171 Disk Space on cp171 is WARNING: DISK WARNING - free space: / 45260MiB (10.0% inode=100%);
[17:10:15] <MirahezeLSBot>	 !log [paladox@mwtask181] starting deploy of {'l10n': True, 'versions': '1.44', 'upgrade_extensions': 'Cargo'} to all
[17:10:17] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[17:10:18] <MirahezeLSBot>	 !log [paladox@test151] starting deploy of {'l10n': True, 'versions': ['1.44', '1.45'], 'upgrade_extensions': 'Cargo'} to test151
[17:10:19] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[17:12:10] <MirahezeLSBot>	 !log [paladox@test151] finished deploy of {'l10n': True, 'versions': ['1.44', '1.45'], 'upgrade_extensions': 'Cargo'} to test151 - SUCCESS in 111s
[17:12:11] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[17:12:21] <MirahezeLSBot>	 !log [paladox@mwtask181] finished deploy of {'l10n': True, 'versions': '1.44', 'upgrade_extensions': 'Cargo'} to all - SUCCESS in 125s
[17:12:22] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[17:34:15] <icinga-miraheze>	 PROBLEM - test151 MediaWiki Rendering on test151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[17:35:26] <icinga-miraheze>	 PROBLEM - cp191 Varnish Backends on cp191 is CRITICAL: 3 backends are down. mw161 mw191 mw203
[17:35:28] <icinga-miraheze>	 PROBLEM - cp161 Varnish Backends on cp161 is CRITICAL: 4 backends are down. mw153 mw183 mw191 mw203
[17:35:54] <icinga-miraheze>	 PROBLEM - mw193 MediaWiki Rendering on mw193 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[17:36:03] <icinga-miraheze>	 PROBLEM - cp201 Varnish Backends on cp201 is CRITICAL: 2 backends are down. mw162 mw202
[17:36:26] <icinga-miraheze>	 PROBLEM - mw173 MediaWiki Rendering on mw173 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.011 second response time
[17:36:42] <icinga-miraheze>	 PROBLEM - cp171 Varnish Backends on cp171 is CRITICAL: 4 backends are down. mw151 mw183 mw191 mw192
[17:36:53] <icinga-miraheze>	 PROBLEM - mw193 HTTPS on mw193 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502
[17:36:56] <icinga-miraheze>	 PROBLEM - mw173 HTTPS on mw173 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10005 milliseconds with 0 bytes received
[17:37:24] <MirahezeRelay>	 <pskyechology> PLEASEEEEEEEEEEEEEEEEEEEEEE
[17:37:48] <MirahezeRelay>	 <pskyechology> @cosmicalpha do you have time to look into QIC later cause I have no idea why it is doing that to beta
[17:37:50] <icinga-miraheze>	 RECOVERY - mw193 MediaWiki Rendering on mw193 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.192 second response time
[17:37:59] <icinga-miraheze>	 PROBLEM - puppet181 Check unit status of listdomains_github_push on puppet181 is CRITICAL: CRITICAL: Status of the systemd unit listdomains_github_push
[17:38:01] <MirahezeRelay>	 <pskyechology> even more concerningly why does it take down half the farm
[17:38:03] <icinga-miraheze>	 RECOVERY - cp201 Varnish Backends on cp201 is OK: All 31 backends are healthy
[17:38:16] <MirahezeRelay>	 <cosmicalpha, replying to pskyechology> Yes I will
[17:38:19] <icinga-miraheze>	 RECOVERY - test151 MediaWiki Rendering on test151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.598 second response time
[17:38:26] <icinga-miraheze>	 RECOVERY - mw173 MediaWiki Rendering on mw173 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.205 second response time
[17:38:42] <icinga-miraheze>	 RECOVERY - cp171 Varnish Backends on cp171 is OK: All 31 backends are healthy
[17:38:51] <icinga-miraheze>	 RECOVERY - mw173 HTTPS on mw173 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4254 bytes in 0.074 second response time
[17:38:53] <icinga-miraheze>	 RECOVERY - mw193 HTTPS on mw193 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4254 bytes in 0.081 second response time
[17:39:26] <icinga-miraheze>	 RECOVERY - cp191 Varnish Backends on cp191 is OK: All 31 backends are healthy
[17:39:28] <icinga-miraheze>	 RECOVERY - cp161 Varnish Backends on cp161 is OK: All 31 backends are healthy
[17:45:59] <icinga-miraheze>	 RECOVERY - puppet181 Check unit status of listdomains_github_push on puppet181 is OK: OK: Status of the systemd unit listdomains_github_push
[18:24:42] <MirahezeLSBot>	 !log [skye@mwtask171] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php MirahezeMagic:RenameDatabase --wiki=loginwiki --rename --old=nintendocinematicserieswiki --new=nintendocinematicuniversewiki --user=Skye (END - exit=0)
[18:24:44] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[18:43:21] <MirahezeLSBot>	 !log [skye@mwtask171] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php CreateWiki:SetContainersAccess --wiki=nintendocinematicuniversewiki (END - exit=0)
[18:43:22] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:00:15] <icinga-miraheze>	 PROBLEM - test151 MediaWiki Rendering on test151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[19:01:50] <icinga-miraheze>	 PROBLEM - cp191 Disk Space on cp191 is WARNING: DISK WARNING - free space: / 45317MiB (10.0% inode=100%);
[19:03:23] <icinga-miraheze>	 PROBLEM - mw193 HTTPS on mw193 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received
[19:03:30] <icinga-miraheze>	 PROBLEM - mw163 HTTPS on mw163 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10006 milliseconds with 0 bytes received
[19:03:57] <icinga-miraheze>	 PROBLEM - mw193 MediaWiki Rendering on mw193 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[19:04:19] <icinga-miraheze>	 RECOVERY - test151 MediaWiki Rendering on test151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.533 second response time
[19:05:22] <icinga-miraheze>	 RECOVERY - mw193 HTTPS on mw193 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4254 bytes in 0.066 second response time
[19:05:24] <icinga-miraheze>	 RECOVERY - mw163 HTTPS on mw163 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4254 bytes in 0.061 second response time
[19:05:53] <icinga-miraheze>	 RECOVERY - mw193 MediaWiki Rendering on mw193 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.187 second response time
[19:16:30] <miraheze-github>	 miraheze/mw-config - Universal-Omega the build passed.
[19:19:04] <MirahezeLSBot>	 !log [universalomega@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all
[19:19:06] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:19:34] <MirahezeLSBot>	 !log [universalomega@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 29s
[19:19:36] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:19:50] <miraheze-github>	 miraheze/mw-config - Universal-Omega the build passed.
[19:20:32] <MirahezeLSBot>	 !log [universalomega@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all
[19:20:33] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:21:00] <MirahezeLSBot>	 !log [universalomega@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 27s
[19:21:02] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:21:10] <miraheze-github>	 miraheze/mw-config - Universal-Omega the build passed.
[19:21:20] <miraheze-github_>	 miraheze/mw-config - Universal-Omega the build passed.
[19:28:15] <MirahezeLSBot>	 !log [skye@mwtask171] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php MirahezeMagic:RenameDatabase --wiki=loginwiki --rename --old=sancswiki --new=nsaswiki --user=Skye (END - exit=0)
[19:28:17] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:32:16] <MirahezeLSBot>	 !log [skye@mwtask171] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php CreateWiki:SetContainersAccess --wiki=nsaswiki (END - exit=0)
[19:32:18] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:57:10] <MirahezeRelay>	 <nabhan_reborn> Tell me what's this for?
[20:02:40] <MirahezeRelay>	 <zppix> It is one of the tech team channels, this one has all the tech-related bot notifications and such
[20:50:23] <MirahezeLSBot>	 !log [skye@test151] starting deploy of {'folders': '1.45/extension/QuickInstantCommons'} to test151
[20:50:25] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[20:50:34] <MirahezeLSBot>	 !log [skye@test151] starting deploy of {'folders': '1.45/extensions/QuickInstantCommons'} to test151
[20:50:35] <MirahezeLSBot>	 !log [skye@test151] finished deploy of {'folders': '1.45/extensions/QuickInstantCommons'} to test151 - SUCCESS in 0s
[20:50:35] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[20:50:37] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[21:28:34] <icinga-miraheze>	 PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 21.36, 16.74, 10.30
[21:30:34] <icinga-miraheze>	 RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 19.55, 17.84, 11.50
[21:44:58] <icinga-miraheze>	 PROBLEM - mwtask151 HTTPS on mwtask151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10005 milliseconds with 0 bytes received
[21:45:06] <icinga-miraheze>	 PROBLEM - mwtask151 jobrunner-high.svc.fsslc.wtnet HTTP on mwtask151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received
[21:45:56] <icinga-miraheze>	 PROBLEM - mwtask181 jobrunner.svc.fsslc.wtnet HTTP on mwtask181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10005 milliseconds with 0 bytes received
[21:46:02] <icinga-miraheze>	 PROBLEM - mwtask181 MediaWiki Rendering on mwtask181 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[21:46:07] <icinga-miraheze>	 PROBLEM - mwtask151 MediaWiki Rendering on mwtask151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[21:46:08] <icinga-miraheze>	 PROBLEM - mwtask171 HTTPS on mwtask171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received
[21:46:10] <icinga-miraheze>	 PROBLEM - mwtask171 jobrunner.svc.fsslc.wtnet HTTP on mwtask171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10006 milliseconds with 0 bytes received
[21:46:11] <icinga-miraheze>	 PROBLEM - mwtask181 HTTPS on mwtask181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10006 milliseconds with 0 bytes received
[21:46:13] <icinga-miraheze>	 PROBLEM - mwtask151 videoscaler.svc.fsslc.wtnet HTTP on mwtask151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received
[21:46:14] <icinga-miraheze>	 PROBLEM - mwtask181 jobrunner-high.svc.fsslc.wtnet HTTP on mwtask181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received
[21:46:22] <icinga-miraheze>	 PROBLEM - mwtask181 videoscaler.svc.fsslc.wtnet HTTP on mwtask181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10006 milliseconds with 0 bytes received
[21:46:35] <icinga-miraheze>	 PROBLEM - mwtask151 jobrunner.svc.fsslc.wtnet HTTP on mwtask151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10005 milliseconds with 0 bytes received
[21:46:36] <icinga-miraheze>	 PROBLEM - mwtask171 MediaWiki Rendering on mwtask171 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[21:46:38] <icinga-miraheze>	 PROBLEM - mwtask171 jobrunner-high.svc.fsslc.wtnet HTTP on mwtask171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10006 milliseconds with 0 bytes received
[21:46:45] <icinga-miraheze>	 PROBLEM - mwtask161 jobrunner-high.svc.fsslc.wtnet HTTP on mwtask161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10006 milliseconds with 0 bytes received
[21:46:51] <icinga-miraheze>	 PROBLEM - mwtask171 videoscaler.svc.fsslc.wtnet HTTP on mwtask171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10006 milliseconds with 0 bytes received
[21:47:03] <icinga-miraheze>	 PROBLEM - mwtask161 jobrunner.svc.fsslc.wtnet HTTP on mwtask161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received
[21:47:38] <icinga-miraheze>	 PROBLEM - mwtask161 HTTPS on mwtask161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10006 milliseconds with 0 bytes received
[21:47:40] <icinga-miraheze>	 PROBLEM - mwtask161 videoscaler.svc.fsslc.wtnet HTTP on mwtask161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received
[21:47:58] <icinga-miraheze>	 PROBLEM - mwtask161 MediaWiki Rendering on mwtask161 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[21:48:05] <icinga-miraheze>	 RECOVERY - mwtask181 MediaWiki Rendering on mwtask181 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 6.038 second response time
[21:48:06] <icinga-miraheze>	 RECOVERY - mwtask181 HTTPS on mwtask181 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4258 bytes in 0.067 second response time
[21:48:10] <icinga-miraheze>	 RECOVERY - mwtask151 MediaWiki Rendering on mwtask151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 5.377 second response time
[21:48:13] <icinga-miraheze>	 RECOVERY - mwtask151 videoscaler.svc.fsslc.wtnet HTTP on mwtask151 is OK: HTTP OK: HTTP/1.1 204 No Content - 171 bytes in 4.718 second response time
[21:48:15] <icinga-miraheze>	 RECOVERY - mwtask181 jobrunner-high.svc.fsslc.wtnet HTTP on mwtask181 is OK: HTTP OK: HTTP/1.1 204 No Content - 171 bytes in 4.444 second response time
[21:48:36] <icinga-miraheze>	 RECOVERY - mwtask151 jobrunner.svc.fsslc.wtnet HTTP on mwtask151 is OK: HTTP OK: HTTP/1.1 204 No Content - 171 bytes in 5.448 second response time
[21:49:02] <icinga-miraheze>	 RECOVERY - mwtask151 HTTPS on mwtask151 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4258 bytes in 0.061 second response time
[21:49:13] <icinga-miraheze>	 RECOVERY - mwtask151 jobrunner-high.svc.fsslc.wtnet HTTP on mwtask151 is OK: HTTP OK: HTTP/1.1 204 No Content - 170 bytes in 0.991 second response time
[21:50:00] <icinga-miraheze>	 RECOVERY - mwtask181 jobrunner.svc.fsslc.wtnet HTTP on mwtask181 is OK: HTTP OK: HTTP/1.1 204 No Content - 170 bytes in 0.294 second response time
[21:50:27] <icinga-miraheze>	 RECOVERY - mwtask171 jobrunner.svc.fsslc.wtnet HTTP on mwtask171 is OK: HTTP OK: HTTP/1.1 204 No Content - 171 bytes in 9.444 second response time
[21:50:30] <icinga-miraheze>	 RECOVERY - mwtask181 videoscaler.svc.fsslc.wtnet HTTP on mwtask181 is OK: HTTP OK: HTTP/1.1 204 No Content - 170 bytes in 0.744 second response time
[21:50:36] <icinga-miraheze>	 RECOVERY - mwtask171 MediaWiki Rendering on mwtask171 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 1.928 second response time
[21:50:49] <icinga-miraheze>	 RECOVERY - mwtask171 jobrunner-high.svc.fsslc.wtnet HTTP on mwtask171 is OK: HTTP OK: HTTP/1.1 204 No Content - 171 bytes in 2.931 second response time
[21:50:54] <icinga-miraheze>	 RECOVERY - mwtask171 videoscaler.svc.fsslc.wtnet HTTP on mwtask171 is OK: HTTP OK: HTTP/1.1 204 No Content - 168 bytes in 0.003 second response time
[21:54:19] <icinga-miraheze>	 PROBLEM - mwtask181 jobrunner.svc.fsslc.wtnet HTTP on mwtask181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received
[21:54:22] <icinga-miraheze>	 PROBLEM - mwtask181 MediaWiki Rendering on mwtask181 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[21:54:26] <icinga-miraheze>	 RECOVERY - mwtask171 HTTPS on mwtask171 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4258 bytes in 2.372 second response time
[21:54:27] <icinga-miraheze>	 PROBLEM - mwtask181 HTTPS on mwtask181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10005 milliseconds with 0 bytes received
[21:56:16] <icinga-miraheze>	 RECOVERY - mwtask181 jobrunner.svc.fsslc.wtnet HTTP on mwtask181 is OK: HTTP OK: HTTP/1.1 204 No Content - 168 bytes in 0.002 second response time
[21:56:19] <icinga-miraheze>	 RECOVERY - mwtask181 MediaWiki Rendering on mwtask181 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.500 second response time
[21:56:22] <icinga-miraheze>	 RECOVERY - mwtask181 HTTPS on mwtask181 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4258 bytes in 0.070 second response time
[21:59:31] <icinga-miraheze>	 PROBLEM - mwtask171 videoscaler.svc.fsslc.wtnet HTTP on mwtask171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received
[21:59:36] <icinga-miraheze>	 PROBLEM - mwtask171 jobrunner-high.svc.fsslc.wtnet HTTP on mwtask171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10006 milliseconds with 0 bytes received
[22:00:50] <icinga-miraheze>	 PROBLEM - mwtask181 jobrunner-high.svc.fsslc.wtnet HTTP on mwtask181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10006 milliseconds with 0 bytes received
[22:01:17] <icinga-miraheze>	 PROBLEM - mwtask151 HTTPS on mwtask151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10006 milliseconds with 0 bytes received
[22:01:55] <icinga-miraheze>	 PROBLEM - mwtask171 HTTPS on mwtask171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10005 milliseconds with 0 bytes received
[22:02:09] <icinga-miraheze>	 PROBLEM - mwtask171 MediaWiki Rendering on mwtask171 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[22:02:24] <icinga-miraheze>	 PROBLEM - mwtask151 videoscaler.svc.fsslc.wtnet HTTP on mwtask151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received
[22:02:27] <icinga-miraheze>	 PROBLEM - mwtask171 jobrunner.svc.fsslc.wtnet HTTP on mwtask171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10006 milliseconds with 0 bytes received
[22:02:30] <icinga-miraheze>	 PROBLEM - mwtask151 MediaWiki Rendering on mwtask151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[22:02:32] <icinga-miraheze>	 PROBLEM - mwtask181 jobrunner.svc.fsslc.wtnet HTTP on mwtask181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10006 milliseconds with 0 bytes received
[22:02:36] <icinga-miraheze>	 PROBLEM - mwtask181 MediaWiki Rendering on mwtask181 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[22:02:40] <icinga-miraheze>	 PROBLEM - mwtask181 HTTPS on mwtask181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10006 milliseconds with 0 bytes received
[22:02:56] <icinga-miraheze>	 RECOVERY - mwtask181 jobrunner-high.svc.fsslc.wtnet HTTP on mwtask181 is OK: HTTP OK: HTTP/1.1 204 No Content - 171 bytes in 9.150 second response time
[22:02:58] <icinga-miraheze>	 RECOVERY - mwtask161 MediaWiki Rendering on mwtask161 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 3.640 second response time
[22:03:15] <icinga-miraheze>	 RECOVERY - mwtask161 jobrunner-high.svc.fsslc.wtnet HTTP on mwtask161 is OK: HTTP OK: HTTP/1.1 204 No Content - 168 bytes in 0.003 second response time
[22:03:18] <icinga-miraheze>	 RECOVERY - mwtask151 HTTPS on mwtask151 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4258 bytes in 3.262 second response time
[22:03:34] <icinga-miraheze>	 RECOVERY - mwtask171 videoscaler.svc.fsslc.wtnet HTTP on mwtask171 is OK: HTTP OK: HTTP/1.1 204 No Content - 170 bytes in 0.209 second response time
[22:03:45] <icinga-miraheze>	 RECOVERY - mwtask171 jobrunner-high.svc.fsslc.wtnet HTTP on mwtask171 is OK: HTTP OK: HTTP/1.1 204 No Content - 171 bytes in 1.421 second response time
[22:03:54] <icinga-miraheze>	 RECOVERY - mwtask171 HTTPS on mwtask171 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4258 bytes in 2.351 second response time
[22:04:04] <icinga-miraheze>	 RECOVERY - mwtask171 MediaWiki Rendering on mwtask171 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.217 second response time
[22:04:04] <icinga-miraheze>	 RECOVERY - mwtask161 jobrunner.svc.fsslc.wtnet HTTP on mwtask161 is OK: HTTP OK: HTTP/1.1 204 No Content - 168 bytes in 0.003 second response time
[22:04:14] <icinga-miraheze>	 RECOVERY - mwtask161 videoscaler.svc.fsslc.wtnet HTTP on mwtask161 is OK: HTTP OK: HTTP/1.1 204 No Content - 168 bytes in 0.003 second response time
[22:04:20] <icinga-miraheze>	 RECOVERY - mwtask151 videoscaler.svc.fsslc.wtnet HTTP on mwtask151 is OK: HTTP OK: HTTP/1.1 204 No Content - 168 bytes in 0.004 second response time
[22:04:21] <icinga-miraheze>	 RECOVERY - mwtask161 HTTPS on mwtask161 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4258 bytes in 0.067 second response time
[22:04:27] <icinga-miraheze>	 RECOVERY - mwtask151 MediaWiki Rendering on mwtask151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.215 second response time
[22:04:29] <icinga-miraheze>	 RECOVERY - mwtask181 jobrunner.svc.fsslc.wtnet HTTP on mwtask181 is OK: HTTP OK: HTTP/1.1 204 No Content - 168 bytes in 0.002 second response time
[22:04:32] <icinga-miraheze>	 RECOVERY - mwtask181 MediaWiki Rendering on mwtask181 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.241 second response time
[22:04:38] <icinga-miraheze>	 RECOVERY - mwtask181 HTTPS on mwtask181 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4258 bytes in 0.796 second response time
[22:05:14] <MirahezeRelay>	 <pskyechology> what did i miss out on
[22:07:16] <icinga-miraheze>	 PROBLEM - mwtask181 jobrunner-high.svc.fsslc.wtnet HTTP on mwtask181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10005 milliseconds with 0 bytes received
[22:07:39] <icinga-miraheze>	 PROBLEM - mwtask151 HTTPS on mwtask151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10006 milliseconds with 0 bytes received
[22:07:43] <icinga-miraheze>	 PROBLEM - mwtask181 videoscaler.svc.fsslc.wtnet HTTP on mwtask181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received
[22:08:44] <icinga-miraheze>	 PROBLEM - mwtask151 videoscaler.svc.fsslc.wtnet HTTP on mwtask151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received
[22:08:47] <icinga-miraheze>	 RECOVERY - mwtask171 jobrunner.svc.fsslc.wtnet HTTP on mwtask171 is OK: HTTP OK: HTTP/1.1 204 No Content - 171 bytes in 2.802 second response time
[22:08:49] <icinga-miraheze>	 PROBLEM - mwtask181 jobrunner.svc.fsslc.wtnet HTTP on mwtask181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received
[22:08:50] <icinga-miraheze>	 PROBLEM - mwtask151 MediaWiki Rendering on mwtask151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[22:08:51] <icinga-miraheze>	 PROBLEM - mwtask151 jobrunner.svc.fsslc.wtnet HTTP on mwtask151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received
[22:08:54] <icinga-miraheze>	 PROBLEM - mwtask151 jobrunner-high.svc.fsslc.wtnet HTTP on mwtask151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received
[22:08:55] <icinga-miraheze>	 PROBLEM - mwtask181 MediaWiki Rendering on mwtask181 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[22:09:05] <icinga-miraheze>	 PROBLEM - mwtask181 HTTPS on mwtask181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10009 milliseconds with 0 bytes received
[22:09:05] <MirahezeRelay>	 <pskyechology> gup,,,,
[22:10:56] <icinga-miraheze>	 RECOVERY - mwtask181 MediaWiki Rendering on mwtask181 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 4.501 second response time
[22:11:00] <icinga-miraheze>	 RECOVERY - mwtask181 HTTPS on mwtask181 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4258 bytes in 0.046 second response time
[22:11:21] <icinga-miraheze>	 RECOVERY - mwtask181 jobrunner-high.svc.fsslc.wtnet HTTP on mwtask181 is OK: HTTP OK: HTTP/1.1 204 No Content - 171 bytes in 2.159 second response time
[22:12:06] <icinga-miraheze>	 PROBLEM - mwtask171 videoscaler.svc.fsslc.wtnet HTTP on mwtask171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received
[22:12:37] <icinga-miraheze>	 PROBLEM - mwtask171 MediaWiki Rendering on mwtask171 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[22:13:15] <icinga-miraheze>	 PROBLEM - mwtask171 jobrunner.svc.fsslc.wtnet HTTP on mwtask171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10006 milliseconds with 0 bytes received
[22:14:30] <icinga-miraheze>	 PROBLEM - mwtask161 jobrunner-high.svc.fsslc.wtnet HTTP on mwtask161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10006 milliseconds with 0 bytes received
[22:14:33] <icinga-miraheze>	 PROBLEM - mwtask161 HTTPS on mwtask161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received
[22:14:35] <icinga-miraheze>	 PROBLEM - mwtask161 MediaWiki Rendering on mwtask161 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[22:14:36] <icinga-miraheze>	 PROBLEM - mwtask161 jobrunner.svc.fsslc.wtnet HTTP on mwtask161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10006 milliseconds with 0 bytes received
[22:15:13] <icinga-miraheze>	 RECOVERY - mwtask151 jobrunner-high.svc.fsslc.wtnet HTTP on mwtask151 is OK: HTTP OK: HTTP/1.1 204 No Content - 171 bytes in 4.102 second response time
[22:15:16] <icinga-miraheze>	 PROBLEM - mwtask181 MediaWiki Rendering on mwtask181 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[22:15:22] <icinga-miraheze>	 PROBLEM - mwtask181 HTTPS on mwtask181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received
[22:15:26] <icinga-miraheze>	 PROBLEM - mwtask171 HTTPS on mwtask171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10006 milliseconds with 0 bytes received
[22:15:41] <icinga-miraheze>	 PROBLEM - mwtask181 jobrunner-high.svc.fsslc.wtnet HTTP on mwtask181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received
[22:15:42] <icinga-miraheze>	 PROBLEM - mwtask171 jobrunner-high.svc.fsslc.wtnet HTTP on mwtask171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 9005: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received
[22:16:04] <icinga-miraheze>	 RECOVERY - mwtask151 HTTPS on mwtask151 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4258 bytes in 5.820 second response time
[22:16:29] <icinga-miraheze>	 RECOVERY - mwtask161 jobrunner-high.svc.fsslc.wtnet HTTP on mwtask161 is OK: HTTP OK: HTTP/1.1 204 No Content - 171 bytes in 3.856 second response time
[22:16:34] <icinga-miraheze>	 RECOVERY - mwtask161 HTTPS on mwtask161 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4258 bytes in 4.158 second response time
[22:16:39] <icinga-miraheze>	 RECOVERY - mwtask161 MediaWiki Rendering on mwtask161 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 4.672 second response time
[22:16:39] <icinga-miraheze>	 RECOVERY - mwtask161 jobrunner.svc.fsslc.wtnet HTTP on mwtask161 is OK: HTTP OK: HTTP/1.1 204 No Content - 171 bytes in 4.298 second response time
[22:16:59] <icinga-miraheze>	 RECOVERY - mwtask151 videoscaler.svc.fsslc.wtnet HTTP on mwtask151 is OK: HTTP OK: HTTP/1.1 204 No Content - 168 bytes in 0.004 second response time
[22:17:04] <icinga-miraheze>	 RECOVERY - mwtask151 jobrunner.svc.fsslc.wtnet HTTP on mwtask151 is OK: HTTP OK: HTTP/1.1 204 No Content - 168 bytes in 0.006 second response time
[22:17:06] <icinga-miraheze>	 RECOVERY - mwtask181 jobrunner.svc.fsslc.wtnet HTTP on mwtask181 is OK: HTTP OK: HTTP/1.1 204 No Content - 169 bytes in 0.027 second response time
[22:17:09] <icinga-miraheze>	 RECOVERY - mwtask151 MediaWiki Rendering on mwtask151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.190 second response time
[22:17:13] <icinga-miraheze>	 RECOVERY - mwtask181 MediaWiki Rendering on mwtask181 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.695 second response time
[22:17:19] <icinga-miraheze>	 RECOVERY - mwtask181 HTTPS on mwtask181 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4258 bytes in 1.520 second response time
[22:17:23] <icinga-miraheze>	 RECOVERY - mwtask171 jobrunner.svc.fsslc.wtnet HTTP on mwtask171 is OK: HTTP OK: HTTP/1.1 204 No Content - 168 bytes in 0.003 second response time
[22:17:23] <icinga-miraheze>	 RECOVERY - mwtask171 HTTPS on mwtask171 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4258 bytes in 0.066 second response time
[22:17:38] <icinga-miraheze>	 RECOVERY - mwtask181 jobrunner-high.svc.fsslc.wtnet HTTP on mwtask181 is OK: HTTP OK: HTTP/1.1 204 No Content - 168 bytes in 0.002 second response time
[22:17:47] <icinga-miraheze>	 RECOVERY - mwtask171 jobrunner-high.svc.fsslc.wtnet HTTP on mwtask171 is OK: HTTP OK: HTTP/1.1 204 No Content - 171 bytes in 6.478 second response time
[22:18:18] <icinga-miraheze>	 RECOVERY - mwtask181 videoscaler.svc.fsslc.wtnet HTTP on mwtask181 is OK: HTTP OK: HTTP/1.1 204 No Content - 171 bytes in 2.126 second response time
[22:18:23] <icinga-miraheze>	 RECOVERY - mwtask171 videoscaler.svc.fsslc.wtnet HTTP on mwtask171 is OK: HTTP OK: HTTP/1.1 204 No Content - 171 bytes in 6.492 second response time
[22:18:26] <icinga-miraheze>	 [Grafana] FIRING: The mediawiki JobQueue backlog is increasing by more than 100 jobs a minute over an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1768857470000&orgId=1&to=1768861106678
[22:18:48] <icinga-miraheze>	 RECOVERY - mwtask171 MediaWiki Rendering on mwtask171 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 9.184 second response time
[22:26:11] <miraheze-github>	 miraheze/MirahezeMagic - dependabot[bot] the build has errored.
[22:28:32] <miraheze-github>	 miraheze/DataDump - dependabot[bot] the build passed.
[22:35:52] <miraheze-github>	 miraheze/WikiDiscover - dependabot[bot] the build passed.
[22:40:34] <icinga-miraheze>	 PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 21.54, 19.98, 14.06
[22:42:34] <icinga-miraheze>	 PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 25.93, 21.85, 15.44
[22:43:26] <icinga-miraheze>	 [Grafana] FIRING: The mediawiki job queue has more than 500,000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?from=1768858910000&orgId=1&to=1768862606683[Grafana] RESOLVED: MediaWiki Jobs are rapidly increasing https://grafana.wikitide.net/d/GtxbP1Xnk?from=1768858970000&orgId=1&to=1768862570000
[22:44:34] <icinga-miraheze>	 PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 21.45, 21.88, 16.26
[22:52:34] <icinga-miraheze>	 PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 24.21, 22.60, 18.77
[23:06:34] <icinga-miraheze>	 PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 20.72, 24.00, 22.52
[23:10:34] <icinga-miraheze>	 PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 26.97, 24.25, 22.82
[23:10:45] <miraheze-github>	 miraheze/CreateWiki - dependabot[bot] the build passed.
[23:14:34] <icinga-miraheze>	 PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 20.73, 23.61, 22.95
[23:29:25] <miraheze-github>	 miraheze/IncidentReporting - dependabot[bot] the build passed.
[23:32:34] <icinga-miraheze>	 PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 24.08, 21.92, 21.26
[23:52:23] <MirahezeLSBot>	 !log [somerandomdeveloper@test151] starting deploy of {'versions': ['1.44', '1.45'], 'upgrade_extensions': 'TreeAndMenu'} to test151
[23:52:25] <MirahezeLSBot>	 !log [somerandomdeveloper@test151] finished deploy of {'versions': ['1.44', '1.45'], 'upgrade_extensions': 'TreeAndMenu'} to test151 - SUCCESS in 1s
[23:52:25] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[23:52:26] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[23:58:43] <miraheze-github>	 miraheze/DataDump - Universal-Omega the build passed.
[23:59:12] <miraheze-github>	 miraheze/IncidentReporting - Universal-Omega the build passed.
[23:59:13] <miraheze-github_>	 miraheze/WikiDiscover - Universal-Omega the build passed.
[23:59:58] <miraheze-github>	 miraheze/CreateWiki - Universal-Omega the build passed.