[00:01:06] <icinga-miraheze>	 PROBLEM - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is WARNING: WARNING - NGINX Error Rate is 58%
[00:01:26] <MacFanBot>	 [02statichelp] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/statichelp/commit/bd9080763f80e38c7d9e8f321bfee1037eee8eae
[00:01:26] <MacFanBot>	 02statichelp/03main 07WikiTideBot 03bd90807 Bot: Auto-update Tech namespace pages 2026-01-14 00:01:23
[00:03:06] <icinga-miraheze>	 PROBLEM - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is CRITICAL: CRITICAL - NGINX Error Rate is 61%
[00:03:23] <MacFanBot>	 [02MirahezeMagic] 07Universal-Omega created 03remove-dpl4-view (+1 new commit) 13https://github.com/miraheze/MirahezeMagic/commit/c210ad95fb86
[00:03:24] <MacFanBot>	 02MirahezeMagic/03remove-dpl4-view 07CosmicAlpha 03c210ad9 RenameDatabase: remove dpl4 view checks…
[00:03:30] <MacFanBot>	 [02MirahezeMagic] 07Universal-Omega opened pull request #619: RenameDatabase: remove dpl4 view checks (03main...03remove-dpl4-view) 13https://github.com/miraheze/MirahezeMagic/pull/619
[00:03:37] <MacFanBot>	 [02MirahezeMagic] 07coderabbitai[bot] commented on pull request #619: <details> […] 13https://github.com/miraheze/MirahezeMagic/pull/619#issuecomment-3747081479
[00:04:26] <MacFanBot>	 [02MirahezeMagic] 07Universal-Omega pushed 1 new commit to 03remove-dpl4-view 13https://github.com/miraheze/MirahezeMagic/commit/447a81db8d8c742ccc358dacc6792570928e7204
[00:04:26] <MacFanBot>	 02MirahezeMagic/03remove-dpl4-view 07CosmicAlpha 03447a81d Remove
[00:05:06] <icinga-miraheze>	 RECOVERY - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is OK: OK - NGINX Error Rate is 23%
[00:07:48] <MacFanBot>	 [02mw-config] 07Universal-Omega created 03remove-dpl4-view (+1 new commit) 13https://github.com/miraheze/mw-config/commit/0df89083e7d5
[00:07:48] <MacFanBot>	 02mw-config/03remove-dpl4-view 07CosmicAlpha 030df8908 Don't run CreateView for DynamicPageList4…
[00:07:56] <MacFanBot>	 [02mw-config] 07Universal-Omega opened pull request #6230: Don't run CreateView for DynamicPageList4 (03main...03remove-dpl4-view) 13https://github.com/miraheze/mw-config/pull/6230
[00:08:22] <MacFanBot>	 [02mw-config] 07github-actions[bot] pushed 1 new commit to 03remove-dpl4-view 13https://github.com/miraheze/mw-config/commit/4b438f25452f6b8d2fd59a4a05b286eb4a6b0ba5
[00:08:22] <MacFanBot>	 02mw-config/03remove-dpl4-view 07github-actions 034b438f2 CI: lint code to MediaWiki standards…
[00:08:41] <miraheze-github>	 miraheze/MirahezeMagic - Universal-Omega the build has errored.
[00:08:54] <miraheze-github>	 miraheze/mw-config - Universal-Omega the build passed.
[00:21:45] <icinga-miraheze>	 PROBLEM - cp171 Disk Space on cp171 is CRITICAL: DISK CRITICAL - free space: / 21956MiB (4.8% inode=100%);
[00:23:09] <MacFanBot>	 [02mw-config] 07Universal-Omega merged pull request #6230: Don't run CreateView for DynamicPageList4 (03main...03remove-dpl4-view) 13https://github.com/miraheze/mw-config/pull/6230
[00:23:11] <MacFanBot>	 [02mw-config] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/mw-config/commit/a1077221b19ed69b24bcb01eb04cbf2abbbc03c6
[00:23:13] <MacFanBot>	 [02mw-config] 07Universal-Omega 04deleted 03remove-dpl4-view at 034b438f2 13https://api.github.com/repos/miraheze/mw-config/commit/4b438f2
[00:23:15] <MacFanBot>	 02mw-config/03main 07CosmicAlpha 03a107722 Don't run CreateView for DynamicPageList4…
[00:23:18] <MirahezeLSBot>	 !log [universalomega@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all
[00:23:19] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[00:23:48] <MirahezeLSBot>	 !log [universalomega@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 30s
[00:23:49] <MacFanBot>	 [02MirahezeMagic] 07Universal-Omega merged pull request #619: RenameDatabase: remove dpl4 view checks (03main...03remove-dpl4-view) 13https://github.com/miraheze/MirahezeMagic/pull/619
[00:23:49] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[00:23:50] <MacFanBot>	 [02MirahezeMagic] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/MirahezeMagic/commit/7f722b4385cf276d705c9122056307f25baa1863
[00:23:50] <MacFanBot>	 02MirahezeMagic/03main 07CosmicAlpha 037f722b4 RenameDatabase: remove dpl4 view checks…
[00:23:50] <MacFanBot>	 [02MirahezeMagic] 07Universal-Omega 04deleted 03remove-dpl4-view at 03447a81d 13https://api.github.com/repos/miraheze/MirahezeMagic/commit/447a81d
[00:24:03] <miraheze-github>	 miraheze/mw-config - Universal-Omega the build passed.
[00:24:12] <MirahezeLSBot>	 !log [universalomega@mwtask181] starting deploy of {'versions': '1.44', 'upgrade_extensions': 'MirahezeMagic'} to all
[00:24:14] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[00:24:41] <MirahezeLSBot>	 !log [universalomega@mwtask181] finished deploy of {'versions': '1.44', 'upgrade_extensions': 'MirahezeMagic'} to all - SUCCESS in 28s
[00:24:42] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[00:25:02] <MirahezeLSBot>	 !log [universalomega@mwtask181] starting deploy of {'versions': '1.44', 'upgrade_extensions': 'DynamicPageList4'} to all
[00:25:03] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[00:25:30] <MirahezeLSBot>	 !log [universalomega@mwtask181] finished deploy of {'versions': '1.44', 'upgrade_extensions': 'DynamicPageList4'} to all - SUCCESS in 28s
[00:25:32] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[00:25:45] <icinga-miraheze>	 RECOVERY - cp171 Disk Space on cp171 is OK: DISK OK - free space: / 60024MiB (13.2% inode=100%);
[00:27:27] <MirahezeLSBot>	 !log [universalomega@mwtask181] sudo -u www-data /usr/local/bin/foreachwikiindblist /tmp/DynamicPageList4.php /srv/mediawiki/1.44/maintenance/run.php DynamicPageList4:DropView (START)
[00:27:29] <MirahezeLSBot>	 !log [universalomega@mwtask181] sudo -u www-data /usr/local/bin/foreachwikiindblist /tmp/DynamicPageList4.php /srv/mediawiki/1.44/maintenance/run.php DynamicPageList4:DropView (END - exit=256)
[00:27:29] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[00:27:30] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[00:27:48] <MirahezeLSBot>	 !log [universalomega@mwtask181] sudo -u www-data /usr/local/bin/foreachwikiindblist /tmp/dynamicpagelist4.php /srv/mediawiki/1.44/maintenance/run.php DynamicPageList4:DropView (START)
[00:27:49] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[00:28:18] <miraheze-github>	 miraheze/MirahezeMagic - Universal-Omega the build has errored.
[00:35:51] <icinga-miraheze>	 RECOVERY - cp201 Disk Space on cp201 is OK: DISK OK - free space: / 70180MiB (15.5% inode=100%);
[00:45:04] <MirahezeLSBot>	 !log [universalomega@mwtask181] sudo -u www-data /usr/local/bin/foreachwikiindblist /tmp/dynamicpagelist4.php /srv/mediawiki/1.44/maintenance/run.php DynamicPageList4:DropView (END - exit=0)
[00:45:05] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[01:00:29] <icinga-miraheze>	 PROBLEM - db161 Current Load on db161 is CRITICAL: LOAD CRITICAL - total load average: 25.72, 13.37, 7.52
[01:06:06] <icinga-miraheze>	 PROBLEM - cp171 Varnish Backends on cp171 is CRITICAL: 4 backends are down. mw162 mw172 mw173 mw192
[01:06:11] <icinga-miraheze>	 PROBLEM - cp161 Varnish Backends on cp161 is CRITICAL: 8 backends are down. mw152 mw162 mw171 mw172 mw173 mw191 mw192 mw201
[01:06:40] <icinga-miraheze>	 PROBLEM - cp191 Varnish Backends on cp191 is CRITICAL: 10 backends are down. mw152 mw171 mw172 mw182 mw173 mw183 mw191 mw192 mw193 mw201
[01:07:06] <icinga-miraheze>	 PROBLEM - cp201 Varnish Backends on cp201 is CRITICAL: 6 backends are down. mw182 mw153 mw191 mw192 mw193 mw203
[01:07:40] <icinga-miraheze>	 PROBLEM - puppet181 Check unit status of listdomains_github_push on puppet181 is CRITICAL: CRITICAL: Status of the systemd unit listdomains_github_push
[01:07:40] <icinga-miraheze>	 RECOVERY - cp191 Disk Space on cp191 is OK: DISK OK - free space: / 71631MiB (15.8% inode=100%);
[01:08:00] <icinga-miraheze>	 PROBLEM - cp161 HTTPS on cp161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502
[01:09:24] <icinga-miraheze>	 PROBLEM - cp191 HTTPS on cp191 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502
[01:10:12] <icinga-miraheze>	 PROBLEM - mw163 MediaWiki Rendering on mw163 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.011 second response time
[01:11:06] <icinga-miraheze>	 PROBLEM - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is WARNING: WARNING - NGINX Error Rate is 57%
[01:12:07] <icinga-miraheze>	 PROBLEM - mw163 HTTPS on mw163 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502
[01:13:06] <icinga-miraheze>	 PROBLEM - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is CRITICAL: CRITICAL - NGINX Error Rate is 65%
[01:13:26] <icinga-miraheze>	 RECOVERY - cp191 HTTPS on cp191 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4303 bytes in 1.312 second response time
[01:14:22] <icinga-miraheze>	 PROBLEM - cp171 HTTPS on cp171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502
[01:15:18] <icinga-miraheze>	 PROBLEM - mw162 HTTPS on mw162 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502
[01:16:14] <icinga-miraheze>	 PROBLEM - mw193 HTTPS on mw193 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10005 milliseconds with 0 bytes received
[01:16:51] <icinga-miraheze>	 PROBLEM - mw192 HTTPS on mw192 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502
[01:16:59] <icinga-miraheze>	 PROBLEM - mw181 MediaWiki Rendering on mw181 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.010 second response time
[01:17:06] <icinga-miraheze>	 PROBLEM - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is WARNING: WARNING - NGINX Error Rate is 54%
[01:17:14] <icinga-miraheze>	 RECOVERY - mw162 HTTPS on mw162 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4254 bytes in 0.070 second response time
[01:17:30] <icinga-miraheze>	 PROBLEM - cp201 HTTPS on cp201 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10005 milliseconds with 0 bytes received
[01:17:56] <icinga-miraheze>	 RECOVERY - cp161 HTTPS on cp161 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4357 bytes in 3.471 second response time
[01:18:13] <icinga-miraheze>	 RECOVERY - mw193 HTTPS on mw193 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4254 bytes in 0.064 second response time
[01:18:55] <icinga-miraheze>	 RECOVERY - mw192 HTTPS on mw192 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4254 bytes in 4.143 second response time
[01:18:56] <icinga-miraheze>	 RECOVERY - mw181 MediaWiki Rendering on mw181 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.200 second response time
[01:19:35] <icinga-miraheze>	 PROBLEM - mw173 HTTPS on mw173 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received
[01:19:39] <icinga-miraheze>	 PROBLEM - cp191 HTTPS on cp191 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502
[01:21:34] <icinga-miraheze>	 RECOVERY - mw173 HTTPS on mw173 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4254 bytes in 0.062 second response time
[01:22:02] <icinga-miraheze>	 PROBLEM - cp161 HTTPS on cp161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502
[01:22:19] <icinga-miraheze>	 PROBLEM - mw173 MediaWiki Rendering on mw173 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[01:23:06] <icinga-miraheze>	 PROBLEM - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is CRITICAL: CRITICAL - NGINX Error Rate is 60%
[01:23:40] <icinga-miraheze>	 RECOVERY - cp191 HTTPS on cp191 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4303 bytes in 2.304 second response time
[01:24:14] <icinga-miraheze>	 RECOVERY - mw173 MediaWiki Rendering on mw173 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.203 second response time
[01:25:06] <icinga-miraheze>	 PROBLEM - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is WARNING: WARNING - NGINX Error Rate is 53%
[01:26:19] <icinga-miraheze>	 PROBLEM - mw172 HTTPS on mw172 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10006 milliseconds with 0 bytes received
[01:26:44] <icinga-miraheze>	 PROBLEM - mw182 MediaWiki Rendering on mw182 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.010 second response time
[01:27:01] <icinga-miraheze>	 PROBLEM - mw182 HTTPS on mw182 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10006 milliseconds with 0 bytes received
[01:27:25] <icinga-miraheze>	 RECOVERY - cp201 HTTPS on cp201 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4303 bytes in 6.741 second response time
[01:27:28] <icinga-miraheze>	 PROBLEM - cp191 HTTPS on cp191 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502
[01:27:49] <icinga-miraheze>	 PROBLEM - mw192 MediaWiki Rendering on mw192 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.013 second response time
[01:28:17] <icinga-miraheze>	 RECOVERY - mw172 HTTPS on mw172 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4254 bytes in 0.046 second response time
[01:28:39] <icinga-miraheze>	 RECOVERY - mw182 MediaWiki Rendering on mw182 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.210 second response time
[01:28:58] <icinga-miraheze>	 RECOVERY - mw182 HTTPS on mw182 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4254 bytes in 0.062 second response time
[01:29:06] <icinga-miraheze>	 PROBLEM - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is CRITICAL: CRITICAL - NGINX Error Rate is 65%
[01:29:27] <icinga-miraheze>	 RECOVERY - cp191 HTTPS on cp191 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4303 bytes in 2.929 second response time
[01:29:46] <icinga-miraheze>	 RECOVERY - cp171 HTTPS on cp171 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4303 bytes in 3.152 second response time
[01:29:47] <icinga-miraheze>	 PROBLEM - mw173 HTTPS on mw173 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received
[01:31:06] <icinga-miraheze>	 PROBLEM - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is WARNING: WARNING - NGINX Error Rate is 52%
[01:31:12] <icinga-miraheze>	 PROBLEM - cp201 HTTPS on cp201 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502
[01:31:46] <icinga-miraheze>	 RECOVERY - mw173 HTTPS on mw173 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4254 bytes in 0.060 second response time
[01:32:20] <icinga-miraheze>	 [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?from=1768350710000&orgId=1&to=1768354340023
[01:33:06] <icinga-miraheze>	 PROBLEM - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is CRITICAL: CRITICAL - NGINX Error Rate is 62%
[01:33:07] <icinga-miraheze>	 RECOVERY - cp201 HTTPS on cp201 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4303 bytes in 0.733 second response time
[01:33:43] <icinga-miraheze>	 PROBLEM - cp171 HTTPS on cp171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502
[01:33:55] <icinga-miraheze>	 RECOVERY - mw192 MediaWiki Rendering on mw192 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 2.828 second response time
[01:34:00] <icinga-miraheze>	 RECOVERY - cp161 HTTPS on cp161 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4357 bytes in 6.867 second response time
[01:35:06] <icinga-miraheze>	 PROBLEM - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is WARNING: WARNING - NGINX Error Rate is 57%
[01:35:24] <icinga-miraheze>	 PROBLEM - cp191 HTTPS on cp191 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502
[01:35:24] <icinga-miraheze>	 PROBLEM - mw203 HTTPS on mw203 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10005 milliseconds with 0 bytes received
[01:35:43] <icinga-miraheze>	 RECOVERY - cp171 HTTPS on cp171 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4303 bytes in 5.752 second response time
[01:37:06] <icinga-miraheze>	 PROBLEM - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is CRITICAL: CRITICAL - NGINX Error Rate is 61%
[01:37:34] <icinga-miraheze>	 RECOVERY - mw203 HTTPS on mw203 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4254 bytes in 9.374 second response time
[01:39:06] <icinga-miraheze>	 PROBLEM - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is WARNING: WARNING - NGINX Error Rate is 44%
[01:39:26] <icinga-miraheze>	 RECOVERY - cp191 HTTPS on cp191 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4303 bytes in 1.765 second response time
[01:41:06] <icinga-miraheze>	 RECOVERY - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is OK: OK - NGINX Error Rate is 32%
[01:43:03] <icinga-miraheze>	 RECOVERY - mw163 MediaWiki Rendering on mw163 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 9.240 second response time
[01:43:24] <icinga-miraheze>	 RECOVERY - mw163 HTTPS on mw163 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4254 bytes in 6.198 second response time
[01:45:06] <icinga-miraheze>	 RECOVERY - cp201 Varnish Backends on cp201 is OK: All 31 backends are healthy
[01:45:40] <icinga-miraheze>	 RECOVERY - puppet181 Check unit status of listdomains_github_push on puppet181 is OK: OK: Status of the systemd unit listdomains_github_push
[01:46:06] <icinga-miraheze>	 RECOVERY - cp171 Varnish Backends on cp171 is OK: All 31 backends are healthy
[01:46:11] <icinga-miraheze>	 RECOVERY - cp161 Varnish Backends on cp161 is OK: All 31 backends are healthy
[01:46:40] <icinga-miraheze>	 RECOVERY - cp191 Varnish Backends on cp191 is OK: All 31 backends are healthy
[01:47:20] <icinga-miraheze>	 [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?from=1768351550000&orgId=1&to=1768355150000
[02:10:29] <icinga-miraheze>	 PROBLEM - db161 Current Load on db161 is WARNING: LOAD WARNING - total load average: 3.99, 5.01, 11.47
[02:14:29] <icinga-miraheze>	 RECOVERY - db161 Current Load on db161 is OK: LOAD OK - total load average: 5.55, 5.19, 10.07
[02:56:29] <icinga-miraheze>	 PROBLEM - db161 Current Load on db161 is CRITICAL: LOAD CRITICAL - total load average: 21.62, 11.22, 8.37
[03:00:33] <MirahezeLSBot>	 !log [universalomega@mwtask181] starting deploy of {'versions': '1.44', 'upgrade_extensions': 'DynamicPageList4'} to all
[03:00:35] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[03:01:02] <MirahezeLSBot>	 !log [universalomega@mwtask181] finished deploy of {'versions': '1.44', 'upgrade_extensions': 'DynamicPageList4'} to all - SUCCESS in 29s
[03:01:04] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[03:04:29] <icinga-miraheze>	 RECOVERY - db161 Current Load on db161 is OK: LOAD OK - total load average: 5.00, 9.92, 9.62
[03:08:12] <icinga-miraheze>	 PROBLEM - db181 Current Load on db181 is WARNING: LOAD WARNING - total load average: 10.85, 10.23, 7.31
[03:10:12] <icinga-miraheze>	 RECOVERY - db181 Current Load on db181 is OK: LOAD OK - total load average: 8.25, 9.69, 7.49
[03:14:12] <icinga-miraheze>	 PROBLEM - db181 Current Load on db181 is CRITICAL: LOAD CRITICAL - total load average: 19.83, 12.85, 9.11
[04:42:12] <icinga-miraheze>	 PROBLEM - db181 Current Load on db181 is WARNING: LOAD WARNING - total load average: 3.83, 5.91, 11.70
[04:44:43] <MacFanBot>	 [02mw-config] 07Universal-Omega created 03sitenotice (+1 new commit) 13https://github.com/miraheze/mw-config/commit/0a1904a5abf3
[04:44:44] <MacFanBot>	 02mw-config/03sitenotice 07CosmicAlpha 030a1904a Add site notice
[04:44:50] <MacFanBot>	 [02mw-config] 07Universal-Omega opened pull request #6231: Add site notice (03main...03sitenotice) 13https://github.com/miraheze/mw-config/pull/6231
[04:45:40] <miraheze-github>	 miraheze/mw-config - Universal-Omega the build passed.
[04:46:12] <icinga-miraheze>	 RECOVERY - db181 Current Load on db181 is OK: LOAD OK - total load average: 3.46, 4.65, 9.88
[04:46:47] <MacFanBot>	 [02mw-config] 07Universal-Omega pushed 1 new commit to 03sitenotice 13https://github.com/miraheze/mw-config/commit/2cab2f3601ddbcacb254a3b132ab0c24ed2db59c
[04:46:48] <MacFanBot>	 02mw-config/03sitenotice 07CosmicAlpha 032cab2f3 Update
[04:47:24] <MacFanBot>	 [02mw-config] 07Universal-Omega merged pull request #6231: Add site notice (03main...03sitenotice) 13https://github.com/miraheze/mw-config/pull/6231
[04:47:24] <MacFanBot>	 [02mw-config] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/mw-config/commit/8b56d391540fdb2cc494ac64f230538364ad836e
[04:47:24] <MacFanBot>	 02mw-config/03main 07CosmicAlpha 038b56d39 Add site notice (#6231)
[04:47:25] <MacFanBot>	 [02mw-config] 07Universal-Omega 04deleted 03sitenotice at 032cab2f3 13https://api.github.com/repos/miraheze/mw-config/commit/2cab2f3
[04:47:39] <miraheze-github>	 miraheze/mw-config - Universal-Omega the build passed.
[04:47:55] <MirahezeLSBot>	 !log [universalomega@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all
[04:47:56] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[04:48:17] <miraheze-github>	 miraheze/mw-config - Universal-Omega the build passed.
[04:48:24] <MirahezeLSBot>	 !log [universalomega@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 29s
[04:48:26] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[05:04:12] <icinga-miraheze>	 PROBLEM - db181 Current Load on db181 is WARNING: LOAD WARNING - total load average: 11.70, 9.24, 8.00
[05:06:12] <icinga-miraheze>	 PROBLEM - db181 Current Load on db181 is CRITICAL: LOAD CRITICAL - total load average: 14.19, 10.93, 8.77
[05:08:12] <icinga-miraheze>	 PROBLEM - db181 Current Load on db181 is WARNING: LOAD WARNING - total load average: 11.86, 11.42, 9.23
[05:10:12] <icinga-miraheze>	 PROBLEM - db181 Current Load on db181 is CRITICAL: LOAD CRITICAL - total load average: 13.98, 12.47, 9.89
[05:18:12] <icinga-miraheze>	 PROBLEM - db181 Current Load on db181 is WARNING: LOAD WARNING - total load average: 11.49, 11.97, 10.73
[05:20:12] <icinga-miraheze>	 PROBLEM - db181 Current Load on db181 is CRITICAL: LOAD CRITICAL - total load average: 12.89, 12.42, 11.05
[05:22:12] <icinga-miraheze>	 PROBLEM - db181 Current Load on db181 is WARNING: LOAD WARNING - total load average: 10.51, 11.74, 10.98
[05:24:12] <icinga-miraheze>	 PROBLEM - db181 Current Load on db181 is CRITICAL: LOAD CRITICAL - total load average: 12.31, 11.95, 11.14
[05:32:12] <icinga-miraheze>	 PROBLEM - db181 Current Load on db181 is WARNING: LOAD WARNING - total load average: 11.39, 11.92, 11.57
[05:36:13] <icinga-miraheze>	 PROBLEM - db181 Current Load on db181 is CRITICAL: LOAD CRITICAL - total load average: 13.34, 12.31, 11.78
[06:04:12] <icinga-miraheze>	 PROBLEM - db181 Current Load on db181 is WARNING: LOAD WARNING - total load average: 11.70, 11.42, 11.94
[06:06:12] <icinga-miraheze>	 PROBLEM - db181 Current Load on db181 is CRITICAL: LOAD CRITICAL - total load average: 13.05, 11.99, 12.08
[06:12:12] <icinga-miraheze>	 PROBLEM - db181 Current Load on db181 is WARNING: LOAD WARNING - total load average: 11.46, 11.78, 11.97
[06:22:12] <icinga-miraheze>	 PROBLEM - db181 Current Load on db181 is CRITICAL: LOAD CRITICAL - total load average: 16.14, 13.01, 12.30
[06:51:39] <MirahezeRelay>	 <pskyechology> ^ caused by dpl4, CA is on it
[07:05:17] <MacFanBot>	 [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/54d70a3e1c63eaf9244efea27952c1f9d4bef9ba
[07:05:17] <MacFanBot>	 02ssl/03main 07WikiTideBot 0354d70a3 Bot: Auto-update domain lists
[07:58:58] <icinga-miraheze>	 PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 24.00, 18.15, 11.15
[08:00:58] <icinga-miraheze>	 PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 25.34, 20.50, 12.86
[08:22:24] <MirahezeLSBot>	 !log [universalomega@mwtask181] starting deploy of {'force_upgrade': True, 'versions': '1.44', 'upgrade_extensions': 'DynamicPageList4'} to all
[08:22:25] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[08:22:55] <MirahezeLSBot>	 !log [universalomega@mwtask181] finished deploy of {'force_upgrade': True, 'versions': '1.44', 'upgrade_extensions': 'DynamicPageList4'} to all - SUCCESS in 31s
[08:22:56] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[08:23:29] <MirahezeLSBot>	 !log [universalomega@mwtask181] starting deploy of {'force_upgrade': True, 'versions': '1.44', 'upgrade_extensions': 'DynamicPageList4'} to all
[08:23:30] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[08:23:59] <MirahezeLSBot>	 !log [universalomega@mwtask181] finished deploy of {'force_upgrade': True, 'versions': '1.44', 'upgrade_extensions': 'DynamicPageList4'} to all - SUCCESS in 30s
[08:24:03] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[08:24:58] <icinga-miraheze>	 PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 15.63, 23.41, 22.58
[08:25:57] <MirahezeLSBot>	 !log [universalomega@mwtask181] starting deploy of {'folders': '1.44/extensions/DynamicPageList4'} to all
[08:25:59] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[08:26:12] <MirahezeRelay>	 <cosmicalpha> !log checkout 4.0.0 tag for DPL4 and revert to it for now.
[08:26:13] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[08:26:25] <MirahezeLSBot>	 !log [universalomega@mwtask181] finished deploy of {'folders': '1.44/extensions/DynamicPageList4'} to all - SUCCESS in 28s
[08:26:27] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[08:28:58] <icinga-miraheze>	 RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 7.00, 14.29, 19.02
[08:33:01] <icinga-miraheze>	 PROBLEM - db181 Current Load on db181 is WARNING: LOAD WARNING - total load average: 0.96, 4.77, 11.96
[08:35:45] <icinga-miraheze>	 RECOVERY - db181 Current Load on db181 is OK: LOAD OK - total load average: 0.96, 3.16, 10.17
[11:55:19] <MacFanBot>	 [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/7f9eb65af6741738f9b9cfb68ac92ae220e013de
[11:55:19] <MacFanBot>	 02ssl/03main 07WikiTideBot 037f9eb65 Bot: Auto-update domain lists
[12:05:17] <MacFanBot>	 [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/21f648284318c73d72f18f9a6dbe4399eba99aba
[12:05:17] <MacFanBot>	 02ssl/03main 07WikiTideBot 0321f6482 Bot: Auto-update domain lists
[13:21:30] <MacFanBot>	 [02puppet] 07paladox created 03paladox-patch-1 (+1 new commit) 13https://github.com/miraheze/puppet/commit/628c4876b456
[13:21:30] <MacFanBot>	 02puppet/03paladox-patch-1 07paladox 03628c487 Install changeprop202
[13:21:34] <MacFanBot>	 [02puppet] 07paladox opened pull request #4675: Install changeprop202 (03main...03paladox-patch-1) 13https://github.com/miraheze/puppet/pull/4675
[13:21:44] <MacFanBot>	 [02puppet] 07coderabbitai[bot] commented on pull request #4675: <details> […] 13https://github.com/miraheze/puppet/pull/4675#issuecomment-3749544498
[13:22:09] <MacFanBot>	 [02dns] 07paladox created 03paladox-patch-2 (+1 new commit) 13https://github.com/miraheze/dns/commit/6922e5fd9c88
[13:22:09] <MacFanBot>	 02dns/03paladox-patch-2 07paladox 036922e5f Add changeprop202 entry with IP 10.0.20.147
[13:22:13] <MacFanBot>	 [02dns] 07paladox opened pull request #704: Add changeprop202 entry with IP 10.0.20.147 (03main...03paladox-patch-2) 13https://github.com/miraheze/dns/pull/704
[13:22:36] <MacFanBot>	 [02dns] 07paladox pushed 1 new commit to 03paladox-patch-2 13https://github.com/miraheze/dns/commit/996dfe14ea33a8778c08fde5927678ebbc9c47df
[13:22:37] <MacFanBot>	 02dns/03paladox-patch-2 07paladox 03996dfe1 Update 10.in-addr.arpa
[13:22:50] <MacFanBot>	 [02dns] 07paladox merged pull request #704: Add changeprop202 entry with IP 10.0.20.147 (03main...03paladox-patch-2) 13https://github.com/miraheze/dns/pull/704
[13:22:51] <MacFanBot>	 [02dns] 07paladox pushed 1 new commit to 03main 13https://github.com/miraheze/dns/commit/b338c005b710d8749c337cb9ddcdf3cb938c5081
[13:22:51] <MacFanBot>	 02dns/03main 07paladox 03b338c00 Add changeprop202 entry with IP 10.0.20.147 (#704)
[13:22:52] <MacFanBot>	 [02dns] 07paladox 04deleted 03paladox-patch-2 at 03996dfe1 13https://api.github.com/repos/miraheze/dns/commit/996dfe1
[13:23:11] <miraheze-github>	 miraheze/dns - paladox the build passed.
[13:23:16] <MacFanBot>	 [02puppet] 07paladox pushed 1 new commit to 03paladox-patch-1 13https://github.com/miraheze/puppet/commit/5a0e9650c090bfb9f8afc57b0ab3e02153d1a398
[13:23:16] <MacFanBot>	 02puppet/03paladox-patch-1 07paladox 035a0e965 Update site.pp
[13:23:36] <miraheze-github>	 miraheze/dns - paladox the build passed.
[13:23:44] <miraheze-github_>	 miraheze/dns - paladox the build passed.
[13:24:09] <MacFanBot>	 [02puppet] 07paladox merged pull request #4675: Install changeprop202 (03main...03paladox-patch-1) 13https://github.com/miraheze/puppet/pull/4675
[13:24:09] <MacFanBot>	 [02puppet] 07paladox pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/b3d86616a5063e99414d8297c083ab8e15c8c27b
[13:24:10] <MacFanBot>	 02puppet/03main 07paladox 03b3d8661 Install changeprop202 (#4675)
[13:24:12] <MacFanBot>	 [02puppet] 07paladox 04deleted 03paladox-patch-1 at 035a0e965 13https://api.github.com/repos/miraheze/puppet/commit/5a0e965
[13:28:26] <MacFanBot>	 [02puppet] 07paladox created 03paladox-patch-1 (+1 new commit) 13https://github.com/miraheze/puppet/commit/fae9bd181ae9
[13:28:26] <MacFanBot>	 02puppet/03paladox-patch-1 07paladox 03fae9bd1 changeprop202: Forward use internal…
[13:28:31] <MacFanBot>	 [02puppet] 07paladox opened pull request #4676: changeprop202: Forward use internal (03main...03paladox-patch-1) 13https://github.com/miraheze/puppet/pull/4676
[13:28:38] <MacFanBot>	 [02puppet] 07coderabbitai[bot] commented on pull request #4676: <details> […] 13https://github.com/miraheze/puppet/pull/4676#issuecomment-3749570978
[13:28:40] <MacFanBot>	 [02puppet] 07paladox merged pull request #4676: changeprop202: Forward use internal (03main...03paladox-patch-1) 13https://github.com/miraheze/puppet/pull/4676
[13:28:42] <MacFanBot>	 [02puppet] 07paladox pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/f5c916a39a99365272a5ee019b867e3cfe8a668a
[13:28:44] <MacFanBot>	 [02puppet] 07paladox 04deleted 03paladox-patch-1 at 03fae9bd1 13https://api.github.com/repos/miraheze/puppet/commit/fae9bd1
[13:28:45] <MacFanBot>	 02puppet/03main 07paladox 03f5c916a changeprop202: Forward use internal (#4676)…
[13:32:02] <MacFanBot>	 [02changeprop-deploy] 07paladox created 03paladox-patch-1 (+1 new commit) 13https://github.com/miraheze/changeprop-deploy/commit/64f66252e3b4
[13:32:02] <MacFanBot>	 02changeprop-deploy/03paladox-patch-1 07paladox 0364f6625 Update submodule
[13:32:07] <MacFanBot>	 [02changeprop-deploy] 07paladox opened pull request #2: Update submodule (03main...03paladox-patch-1) 13https://github.com/miraheze/changeprop-deploy/pull/2
[13:32:15] <MacFanBot>	 [02changeprop-deploy] 07coderabbitai[bot] commented on pull request #2: <details> […] 13https://github.com/miraheze/changeprop-deploy/pull/2#issuecomment-3749584541
[13:32:35] <MacFanBot>	 [02changeprop-deploy] 07paladox pushed 1 new commit to 03paladox-patch-1 13https://github.com/miraheze/changeprop-deploy/commit/7ac3cdf20b4c98333b627e0de1364fdfaf9f0f1c
[13:32:35] <MacFanBot>	 02changeprop-deploy/03paladox-patch-1 07paladox 037ac3cdf Create package-lock.json
[13:34:43] <MacFanBot>	 [02changeprop-deploy] 07paladox pushed 1 new commit to 03paladox-patch-1 13https://github.com/miraheze/changeprop-deploy/commit/58eb0976aaee8bf71aea21aacf2797d534686728
[13:34:44] <MacFanBot>	 02changeprop-deploy/03paladox-patch-1 07paladox 0358eb097 Update
[13:36:15] <icinga-miraheze>	 PROBLEM - changeprop202 PowerDNS Recursor on changeprop202 is CRITICAL: Domain 'changeprop202.fsslc.wtnet' was not found by the server
[13:36:25] <icinga-miraheze>	 PROBLEM - changeprop202 Haproxy backend for localhost:9007 on changeprop202 is CRITICAL: connect to address localhost and port 9007: Connection refused
[13:36:38] <icinga-miraheze>	 PROBLEM - changeprop202 changeprop on changeprop202 is CRITICAL: connect to address 10.0.20.147 and port 7200: Connection refused
[13:36:45] <icinga-miraheze>	 PROBLEM - changeprop202 Puppet on changeprop202 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 7 minutes ago with 0 failures
[13:36:53] <icinga-miraheze>	 PROBLEM - changeprop202 Haproxy backend for localhost:9008 on changeprop202 is CRITICAL: connect to address localhost and port 9008: Connection refused
[13:36:54] <icinga-miraheze>	 PROBLEM - changeprop202 Haproxy backend for localhost:9009 on changeprop202 is CRITICAL: connect to address localhost and port 9009: Connection refused
[13:48:10] <icinga-miraheze>	 RECOVERY - changeprop202 PowerDNS Recursor on changeprop202 is OK: DNS OK: 0.032 seconds response time. changeprop202.fsslc.wtnet returns 10.0.20.147
[13:48:33] <MacFanBot>	 [02changeprop-deploy] 07paladox 04deleted 03paladox-patch-1 at 03d737658 13https://api.github.com/repos/miraheze/changeprop-deploy/commit/d737658
[13:48:34] <icinga-miraheze>	 PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 24.47, 18.08, 12.10
[13:48:35] <MacFanBot>	 [02changeprop-deploy] 07paladox merged pull request #2: Update submodule (03main...03paladox-patch-1) 13https://github.com/miraheze/changeprop-deploy/pull/2
[13:50:24] <icinga-miraheze>	 PROBLEM - changeprop201 Puppet on changeprop201 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 23 minutes ago with 0 failures
[13:50:31] <icinga-miraheze>	 PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 23.69, 19.72, 13.39
[13:52:08] <icinga-miraheze>	 RECOVERY - changeprop202 Puppet on changeprop202 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[13:54:02] <icinga-miraheze>	 RECOVERY - changeprop202 Haproxy backend for localhost:9007 on changeprop202 is OK: TCP OK - 0.000 second response time on localhost port 9007
[13:54:08] <icinga-miraheze>	 RECOVERY - changeprop202 changeprop on changeprop202 is OK: TCP OK - 0.000 second response time on 10.0.20.147 port 7200
[13:54:25] <icinga-miraheze>	 RECOVERY - changeprop202 Haproxy backend for localhost:9009 on changeprop202 is OK: TCP OK - 0.000 second response time on localhost port 9009
[13:54:42] <MirahezeRelay>	 <paladox> !log switching from changeprop201 to 202 by shutting 201 down
[13:54:43] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[13:54:45] <icinga-miraheze>	 RECOVERY - changeprop202 Haproxy backend for localhost:9008 on changeprop202 is OK: TCP OK - 0.000 second response time on localhost port 9008
[13:56:21] <icinga-miraheze>	 RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 8.34, 16.73, 14.46
[14:00:11] <icinga-miraheze>	 PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 22.00, 19.50, 16.05
[14:02:08] <icinga-miraheze>	 PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 24.86, 21.30, 17.13
[14:06:01] <icinga-miraheze>	 PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 15.16, 20.81, 18.04
[14:07:28] <icinga-miraheze>	 PROBLEM - puppet181 Check unit status of listdomains_github_push on puppet181 is CRITICAL: CRITICAL: Status of the systemd unit listdomains_github_push
[14:07:58] <icinga-miraheze>	 RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 6.66, 15.83, 16.55
[14:15:22] <icinga-miraheze>	 RECOVERY - puppet181 Check unit status of listdomains_github_push on puppet181 is OK: OK: Status of the systemd unit listdomains_github_push
[14:18:35] <icinga-miraheze>	 PROBLEM - mw203 Current Load on mw203 is WARNING: LOAD WARNING - total load average: 22.64, 22.17, 14.26
[14:20:32] <icinga-miraheze>	 PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 19.73, 22.70, 14.06
[14:20:33] <icinga-miraheze>	 RECOVERY - mw203 Current Load on mw203 is OK: LOAD OK - total load average: 9.09, 17.16, 13.38
[14:22:30] <icinga-miraheze>	 RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 8.49, 17.46, 13.15
[14:23:02] <MirahezeRelay>	 <paladox> !log delete changeprop201
[14:23:04] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[14:32:33] <MacFanBot>	 [02dns] 07paladox created 03paladox-patch-2 (+1 new commit) 13https://github.com/miraheze/dns/commit/2f7037e78c0e
[14:32:33] <MacFanBot>	 02dns/03paladox-patch-2 07paladox 032f7037e Remove changeprop201
[14:32:38] <MacFanBot>	 [02dns] 07paladox opened pull request #705: Remove changeprop201 (03main...03paladox-patch-2) 13https://github.com/miraheze/dns/pull/705
[14:32:55] <MacFanBot>	 [02dns] 07paladox pushed 1 new commit to 03paladox-patch-2 13https://github.com/miraheze/dns/commit/cb3342a62639948b08337ebce87dfbe545776475
[14:32:55] <MacFanBot>	 02dns/03paladox-patch-2 07paladox 03cb3342a Update 10.in-addr.arpa
[14:33:03] <MacFanBot>	 [02dns] 07paladox pushed 1 new commit to 03main 13https://github.com/miraheze/dns/commit/34c4973bfc3b8d348075c7b77b2602ae8a40d50f
[14:33:04] <MacFanBot>	 02dns/03main 07paladox 0334c4973 Remove changeprop201 (#705)
[14:33:06] <MacFanBot>	 [02dns] 07paladox merged pull request #705: Remove changeprop201 (03main...03paladox-patch-2) 13https://github.com/miraheze/dns/pull/705
[14:33:08] <MacFanBot>	 [02dns] 07paladox 04deleted 03paladox-patch-2 at 03cb3342a 13https://api.github.com/repos/miraheze/dns/commit/cb3342a
[14:33:15] <MacFanBot>	 [02dns] 07paladox pushed 1 new commit to 03main 13https://github.com/miraheze/dns/commit/ed22cd4c3e30c2e25b11525c819f8f6d2ce6ae78
[14:33:15] <MacFanBot>	 02dns/03main 07paladox 03ed22cd4 Remove eventgate181 (#703)…
[14:33:15] <MacFanBot>	 [02dns] 07paladox merged pull request #703: Remove eventgate181 (03main...03paladox-patch-1) 13https://github.com/miraheze/dns/pull/703
[14:33:16] <MacFanBot>	 [02dns] 07paladox 04deleted 03paladox-patch-1 at 03615cb06 13https://api.github.com/repos/miraheze/dns/commit/615cb06
[14:33:35] <miraheze-github>	 miraheze/dns - paladox the build passed.
[14:33:50] <miraheze-github>	 miraheze/dns - paladox the build passed.
[14:33:56] <miraheze-github_>	 miraheze/dns - paladox the build passed.
[14:34:09] <miraheze-github>	 miraheze/dns - paladox the build passed.
[14:42:09] <MacFanBot>	 [02dns] 07paladox created 03paladox-patch-1 (+1 new commit) 13https://github.com/miraheze/dns/commit/b961b2e422f1
[14:42:09] <MacFanBot>	 02dns/03paladox-patch-1 07paladox 03b961b2e Add rdb191 with ip 10.0.19.149
[14:42:20] <MacFanBot>	 [02dns] 07paladox opened pull request #706: Add rdb191 with ip 10.0.19.149 (03main...03paladox-patch-1) 13https://github.com/miraheze/dns/pull/706
[14:42:46] <MacFanBot>	 [02dns] 07paladox pushed 1 new commit to 03paladox-patch-1 13https://github.com/miraheze/dns/commit/72a2cd6be4cd1781b81f22a48db459d32f1b710d
[14:42:47] <MacFanBot>	 02dns/03paladox-patch-1 07paladox 0372a2cd6 Update 10.in-addr.arpa
[14:42:54] <MacFanBot>	 [02dns] 07paladox merged pull request #706: Add rdb191 with ip 10.0.19.149 (03main...03paladox-patch-1) 13https://github.com/miraheze/dns/pull/706
[14:42:55] <MacFanBot>	 [02dns] 07paladox pushed 1 new commit to 03main 13https://github.com/miraheze/dns/commit/11c073c9d0562e97efc391eafe17dc4fea2ee0ef
[14:42:55] <MacFanBot>	 02dns/03main 07paladox 0311c073c Add rdb191 with ip 10.0.19.149 (#706)
[14:42:55] <MacFanBot>	 [02dns] 07paladox 04deleted 03paladox-patch-1 at 0372a2cd6 13https://api.github.com/repos/miraheze/dns/commit/72a2cd6
[14:43:13] <miraheze-github>	 miraheze/dns - paladox the build passed.
[14:43:14] <MacFanBot>	 [02dns] 07paladox pushed 1 new commit to 03main 13https://github.com/miraheze/dns/commit/8aed4532fbffc7479598fbf1fbb86a5ccbb665d2
[14:43:14] <MacFanBot>	 02dns/03main 07paladox 038aed453 Update wtnet
[14:43:27] <MacFanBot>	 [02dns] 07paladox pushed 1 new commit to 03main 13https://github.com/miraheze/dns/commit/53ed0ce9a1f6834e759cb416980b1186d76ffe7c
[14:43:27] <MacFanBot>	 02dns/03main 07paladox 0353ed0ce Update 10.in-addr.arpa
[14:43:39] <miraheze-github>	 miraheze/dns - paladox the build passed.
[14:43:52] <miraheze-github>	 miraheze/dns - paladox the build passed.
[14:44:17] <miraheze-github>	 miraheze/dns - paladox the build passed.
[14:44:45] <miraheze-github>	 miraheze/dns - paladox the build passed.
[14:52:41] <icinga-miraheze>	 PROBLEM - mwtask181 Disk Space on mwtask181 is WARNING: DISK WARNING - free space: / 13429MiB (6% inode=95%);
[15:06:04] <miraheze-github>	 miraheze/mw-config - paladox the build passed.
[15:06:22] <MirahezeLSBot>	 !log [paladox@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all
[15:06:24] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[15:06:25] <MirahezeLSBot>	 !log [paladox@test151] starting deploy of {'pull': 'config', 'config': True} to test151
[15:06:26] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[15:06:27] <MirahezeLSBot>	 !log [paladox@test151] finished deploy of {'pull': 'config', 'config': True} to test151 - SUCCESS in 2s
[15:06:29] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[15:06:31] <MirahezeLSBot>	 !log [paladox@test151] starting deploy of {'pull': 'config', 'config': True} to test151
[15:06:32] <MirahezeLSBot>	 !log [paladox@test151] finished deploy of {'pull': 'config', 'config': True} to test151 - SUCCESS in 0s
[15:06:33] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[15:06:50] <miraheze-github>	 miraheze/mw-config - paladox the build passed.
[15:06:52] <MirahezeLSBot>	 !log [paladox@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 29s
[15:06:53] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[15:07:05] <icinga-miraheze>	 PROBLEM - mwtask181 Disk Space on mwtask181 is CRITICAL: DISK CRITICAL - free space: / 13291MiB (5% inode=95%);
[15:07:35] <MirahezeRelay>	 <paladox> !log switched from rdb151 to rdb191
[15:07:37] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[15:07:54] <icinga-miraheze>	 PROBLEM - rdb191 PowerDNS Recursor on rdb191 is CRITICAL: Domain 'rdb191.fsslc.wtnet' was not found by the server
[15:07:54] <icinga-miraheze>	 PROBLEM - rdb191 NTP time on rdb191 is UNKNOWN: check_ntp_time: Invalid hostname/address - ntp.fsslc.wtnetUsage: check_ntp_time -H <host> [-4|-6] [-w <warn>] [-c <crit>] [-v verbose] [-o <time offset>]
[15:09:02] <icinga-miraheze>	 PROBLEM - mwtask181 Disk Space on mwtask181 is WARNING: DISK WARNING - free space: / 13447MiB (6% inode=95%);
[15:09:51] <MirahezeRelay>	 <paladox> huh
[15:13:10] <icinga-miraheze>	 RECOVERY - rdb191 NTP time on rdb191 is OK: NTP OK: Offset 0.001221001148 secs
[15:13:22] <icinga-miraheze>	 RECOVERY - rdb191 PowerDNS Recursor on rdb191 is OK: DNS OK: 0.031 seconds response time. rdb191.fsslc.wtnet returns 10.0.19.149
[15:14:45] <MirahezeRelay>	 <paladox> !log shutting down rdb151
[15:14:47] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[15:21:33] <icinga-miraheze>	 PROBLEM - cp201 Disk Space on cp201 is WARNING: DISK WARNING - free space: / 45244MiB (10.0% inode=100%);
[15:23:38] <icinga-miraheze>	 PROBLEM - cp191 Disk Space on cp191 is WARNING: DISK WARNING - free space: / 45350MiB (10.0% inode=100%);
[15:45:34] <MirahezeRelay>	 <paladox> !log delete rdb151
[15:45:36] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[15:47:29] <miraheze-github>	 miraheze/dns - paladox the build passed.
[15:47:41] <miraheze-github>	 miraheze/dns - paladox the build passed.
[15:47:51] <miraheze-github_>	 miraheze/dns - paladox the build passed.
[15:48:24] <icinga-miraheze>	 RECOVERY - mwtask181 Disk Space on mwtask181 is OK: DISK OK - free space: / 36135MiB (16% inode=95%);
[15:53:35] <MacFanBot>	 [02puppet] 07paladox created 03paladox-patch-1 (+1 new commit) 13https://github.com/miraheze/puppet/commit/7fde4f3b193d
[15:53:35] <MacFanBot>	 02puppet/03paladox-patch-1 07paladox 037fde4f3 Install rdb191
[15:53:35] <MacFanBot>	 [02puppet] 07paladox opened pull request #4677: Install rdb191 (03main...03paladox-patch-1) 13https://github.com/miraheze/puppet/pull/4677
[15:53:35] <MacFanBot>	 [02puppet] 07coderabbitai[bot] commented on pull request #4677: <details> […] 13https://github.com/miraheze/puppet/pull/4677#issuecomment-3749934158
[15:53:37] <MacFanBot>	 [02puppet] 07paladox pushed 1 new commit to 03paladox-patch-1 13https://github.com/miraheze/puppet/commit/da329cdf3e90f7adb1156d13a6563209c04f77c3
[15:53:38] <MacFanBot>	 02puppet/03paladox-patch-1 07paladox 03da329cd Update site.pp
[15:55:57] <MacFanBot>	 [02puppet] 07paladox created 03paladox-patch-1 (+1 new commit) 13https://github.com/miraheze/puppet/commit/ba13df195613
[15:55:57] <MacFanBot>	 02puppet/03paladox-patch-1 07paladox 03ba13df1 Remove changeprop201
[15:56:00] <MacFanBot>	 [02puppet] 07paladox opened pull request #4679: Remove changeprop201 (03main...03paladox-patch-1) 13https://github.com/miraheze/puppet/pull/4679
[15:56:10] <MacFanBot>	 [02puppet] 07coderabbitai[bot] commented on pull request #4679: <details> […] 13https://github.com/miraheze/puppet/pull/4679#issuecomment-3750249251
[15:56:24] <MacFanBot>	 [02puppet] 07paladox pushed 1 new commit to 03paladox-patch-1 13https://github.com/miraheze/puppet/commit/ae97a48291583e25a84fabd5c90b90c654bbab80
[15:56:24] <MacFanBot>	 02puppet/03paladox-patch-1 07paladox 03ae97a48 Update site.pp
[15:56:38] <MacFanBot>	 [02puppet] 07paladox merged pull request #4679: Remove changeprop201 (03main...03paladox-patch-1) 13https://github.com/miraheze/puppet/pull/4679
[15:56:38] <MacFanBot>	 [02puppet] 07paladox pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/77067ed88f68275338b5072d9761335807a8f5c9
[15:56:39] <MacFanBot>	 02puppet/03main 07paladox 0377067ed Remove changeprop201 (#4679)
[15:56:41] <MacFanBot>	 [02puppet] 07paladox 04deleted 03paladox-patch-1 at 03ae97a48 13https://api.github.com/repos/miraheze/puppet/commit/ae97a48
[15:56:59] <MacFanBot>	 [02puppet] 07paladox created 03paladox-patch-1 (+1 new commit) 13https://github.com/miraheze/puppet/commit/5ee71d72dc54
[15:56:59] <MacFanBot>	 02puppet/03paladox-patch-1 07paladox 035ee71d7 Remove eventgate181
[15:57:02] <MacFanBot>	 [02puppet] 07paladox opened pull request #4680: Remove eventgate181 (03main...03paladox-patch-1) 13https://github.com/miraheze/puppet/pull/4680
[15:57:14] <MacFanBot>	 [02puppet] 07coderabbitai[bot] commented on pull request #4680: <details> […] 13https://github.com/miraheze/puppet/pull/4680#issuecomment-3750254297
[15:57:21] <MacFanBot>	 [02puppet] 07paladox pushed 1 new commit to 03paladox-patch-1 13https://github.com/miraheze/puppet/commit/729a553a9d336e27b82956f91dd20720cc73d3cf
[15:57:21] <MacFanBot>	 02puppet/03paladox-patch-1 07paladox 03729a553 Update site.pp
[15:57:38] <MacFanBot>	 [02puppet] 07paladox merged pull request #4680: Remove eventgate181 (03main...03paladox-patch-1) 13https://github.com/miraheze/puppet/pull/4680
[15:57:38] <MacFanBot>	 [02puppet] 07paladox pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/0f50da047426c285f481c700e244dc3a3dac0392
[15:57:39] <MacFanBot>	 02puppet/03main 07paladox 030f50da0 Remove eventgate181 (#4680)
[15:57:41] <MacFanBot>	 [02puppet] 07paladox 04deleted 03paladox-patch-1 at 03729a553 13https://api.github.com/repos/miraheze/puppet/commit/729a553
[15:57:53] <MacFanBot>	 [02puppet] 07paladox created 03paladox-patch-1 (+1 new commit) 13https://github.com/miraheze/puppet/commit/ae1011794ce5
[15:57:53] <MacFanBot>	 02puppet/03paladox-patch-1 07paladox 03ae10117 Remove rdb151
[15:57:59] <MacFanBot>	 [02puppet] 07paladox opened pull request #4681: Remove rdb151 (03main...03paladox-patch-1) 13https://github.com/miraheze/puppet/pull/4681
[15:58:09] <MacFanBot>	 [02puppet] 07coderabbitai[bot] commented on pull request #4681: <details> […] 13https://github.com/miraheze/puppet/pull/4681#issuecomment-3750258364
[15:58:19] <MacFanBot>	 [02puppet] 07paladox pushed 1 new commit to 03paladox-patch-1 13https://github.com/miraheze/puppet/commit/b75501b455f628c25918a8c401411eaef334407f
[15:58:19] <MacFanBot>	 02puppet/03paladox-patch-1 07paladox 03b75501b Update site.pp
[15:58:27] <MacFanBot>	 [02puppet] 07paladox merged pull request #4681: Remove rdb151 (03main...03paladox-patch-1) 13https://github.com/miraheze/puppet/pull/4681
[15:58:27] <MacFanBot>	 [02puppet] 07paladox pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/6bb9439e8861e274e4e991cfd2ee928af79122ef
[15:58:28] <MacFanBot>	 02puppet/03main 07paladox 036bb9439 Remove rdb151 (#4681)
[15:58:30] <MacFanBot>	 [02puppet] 07paladox 04deleted 03paladox-patch-1 at 03b75501b 13https://api.github.com/repos/miraheze/puppet/commit/b75501b
[15:58:47] <MirahezeLSBot>	 !log [paladox@mwtask181] starting deploy of {'l10n': True, 'versions': '1.44', 'upgrade_extensions': 'EventBus'} to all
[15:58:49] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[16:03:34] <MirahezeLSBot>	 !log [paladox@mwtask181] finished deploy of {'l10n': True, 'versions': '1.44', 'upgrade_extensions': 'EventBus'} to all - SUCCESS in 286s
[16:03:36] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[16:10:23] <MirahezeLSBot>	 !log [paladox@mwtask181] starting deploy of {'l10n': True, 'versions': '1.44', 'upgrade_extensions': 'EventBus'} to all
[16:10:25] <MirahezeLSBot>	 !log [paladox@test151] starting deploy of {'pull': 'config', 'config': True} to test151
[16:10:26] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[16:10:26] <MirahezeLSBot>	 !log [paladox@test151] finished deploy of {'pull': 'config', 'config': True} to test151 - SUCCESS in 0s
[16:10:28] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[16:10:29] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[16:12:01] <MirahezeLSBot>	 !log [paladox@mwtask181] finished deploy of {'l10n': True, 'versions': '1.44', 'upgrade_extensions': 'EventBus'} to all - SUCCESS in 97s
[16:12:02] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[16:24:57] <MirahezeLSBot>	 !log [skye@mwtask181] starting deploy of {'folders': '1.44/extensions/QuickInstantCommons'} to all
[16:25:00] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[16:25:27] <MirahezeLSBot>	 !log [skye@mwtask181] finished deploy of {'folders': '1.44/extensions/QuickInstantCommons'} to all - SUCCESS in 30s
[16:25:29] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[16:36:57] <MacFanBot>	 [02dns] 07paladox created 03paladox-patch-1 (+1 new commit) 13https://github.com/miraheze/dns/commit/3b3c6ea49aaa
[16:36:57] <MacFanBot>	 02dns/03paladox-patch-1 07paladox 033b3c6ea Add kafka182
[16:37:01] <MacFanBot>	 [02dns] 07paladox opened pull request #708: Add kafka182 (03main...03paladox-patch-1) 13https://github.com/miraheze/dns/pull/708
[16:37:11] <icinga-miraheze>	 PROBLEM - cp171 Disk Space on cp171 is WARNING: DISK WARNING - free space: / 45272MiB (10.0% inode=100%);
[16:37:26] <MacFanBot>	 [02dns] 07paladox pushed 1 new commit to 03paladox-patch-1 13https://github.com/miraheze/dns/commit/19ebdf466a10a630a1ae63d2678c910ef4a910bc
[16:37:27] <MacFanBot>	 02dns/03paladox-patch-1 07paladox 0319ebdf4 Update 10.in-addr.arpa
[16:37:42] <MacFanBot>	 [02dns] 07paladox merged pull request #708: Add kafka182 (03main...03paladox-patch-1) 13https://github.com/miraheze/dns/pull/708
[16:37:42] <MacFanBot>	 [02dns] 07paladox pushed 1 new commit to 03main 13https://github.com/miraheze/dns/commit/ce8f0c888f8f920d526b189a34a9a33df62e74e2
[16:37:42] <MacFanBot>	 02dns/03main 07paladox 03ce8f0c8 Add kafka182 (#708)
[16:37:43] <MacFanBot>	 [02dns] 07paladox 04deleted 03paladox-patch-1 at 0319ebdf4 13https://api.github.com/repos/miraheze/dns/commit/19ebdf4
[16:37:59] <miraheze-github>	 miraheze/dns - paladox the build passed.
[16:38:25] <miraheze-github>	 miraheze/dns - paladox the build passed.
[16:38:43] <miraheze-github>	 miraheze/dns - paladox the build passed.
[16:42:13] <MacFanBot>	 [02puppet] 07paladox created 03paladox-patch-1 (+1 new commit) 13https://github.com/miraheze/puppet/commit/7b98730ca2f1
[16:42:13] <MacFanBot>	 02puppet/03paladox-patch-1 07paladox 037b98730 Install kafka182
[16:42:17] <MacFanBot>	 [02puppet] 07paladox opened pull request #4682: Install kafka182 (03main...03paladox-patch-1) 13https://github.com/miraheze/puppet/pull/4682
[16:42:28] <MacFanBot>	 [02puppet] 07coderabbitai[bot] commented on pull request #4682: <details> […] 13https://github.com/miraheze/puppet/pull/4682#issuecomment-3750472651
[16:42:44] <MacFanBot>	 [02puppet] 07paladox pushed 1 new commit to 03paladox-patch-1 13https://github.com/miraheze/puppet/commit/d13ebf273660cff2097790eba759dcddd25c5312
[16:42:45] <MacFanBot>	 02puppet/03paladox-patch-1 07paladox 03d13ebf2 Update kafka182.yaml
[16:43:04] <MacFanBot>	 [02puppet] 07paladox pushed 1 new commit to 03paladox-patch-1 13https://github.com/miraheze/puppet/commit/acfb707217a77793c1bd1fe2ae617066dfa3d8dd
[16:43:04] <MacFanBot>	 02puppet/03paladox-patch-1 07paladox 03acfb707 Update site.pp
[16:48:04] <MacFanBot>	 [02puppet] 07paladox merged pull request #4682: Install kafka182 (03main...03paladox-patch-1) 13https://github.com/miraheze/puppet/pull/4682
[16:48:04] <MacFanBot>	 [02puppet] 07paladox pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/d2cd1a5bad51f8d50cf406a99f87d3970d492f64
[16:48:04] <MacFanBot>	 02puppet/03main 07paladox 03d2cd1a5 Install kafka182 (#4682)
[16:48:05] <MacFanBot>	 [02puppet] 07paladox 04deleted 03paladox-patch-1 at 03acfb707 13https://api.github.com/repos/miraheze/puppet/commit/acfb707
[16:50:00] <icinga-miraheze>	 PROBLEM - bast161 Current Load on bast161 is CRITICAL: LOAD CRITICAL - total load average: 3.49, 2.46, 1.06
[16:51:50] <icinga-miraheze>	 PROBLEM - puppet181 Check unit status of git_pull_puppet on puppet181 is CRITICAL: CRITICAL: Status of the systemd unit git_pull_puppet
[16:51:56] <icinga-miraheze>	 RECOVERY - bast161 Current Load on bast161 is OK: LOAD OK - total load average: 0.57, 1.68, 0.94
[16:52:30] <icinga-miraheze>	 PROBLEM - puppet181 Check unit status of git_pull_ssl on puppet181 is CRITICAL: CRITICAL: Status of the systemd unit git_pull_ssl
[16:55:43] <MacFanBot>	 [02puppet] 07paladox created 03paladox-patch-1 (+1 new commit) 13https://github.com/miraheze/puppet/commit/3d5e383db29a
[16:55:43] <MacFanBot>	 02puppet/03paladox-patch-1 07paladox 033d5e383 Kafka: Use proxy server when installing via archive class
[16:55:48] <MacFanBot>	 [02puppet] 07paladox opened pull request #4683: Kafka: Use proxy server when installing via archive class (03main...03paladox-patch-1) 13https://github.com/miraheze/puppet/pull/4683
[16:56:00] <MacFanBot>	 [02puppet] 07coderabbitai[bot] commented on pull request #4683: <details> […] 13https://github.com/miraheze/puppet/pull/4683#issuecomment-3750548331
[16:58:05] <icinga-miraheze>	 PROBLEM - matomo151 Check unit status of matomo-archiver-3 on matomo151 is CRITICAL: CRITICAL: Status of the systemd unit matomo-archiver-3
[16:58:21] <icinga-miraheze>	 PROBLEM - matomo151 Check unit status of matomo-archiver-2 on matomo151 is CRITICAL: CRITICAL: Status of the systemd unit matomo-archiver-2
[16:58:26] <MacFanBot>	 [02puppet] 07paladox pushed 1 new commit to 03paladox-patch-1 13https://github.com/miraheze/puppet/commit/35a7241c9ca13a91e37c45885682cd3d74545b68
[16:58:27] <MacFanBot>	 02puppet/03paladox-patch-1 07paladox 0335a7241 Update kafka.pp
[16:58:43] <icinga-miraheze>	 PROBLEM - matomo151 Check unit status of matomo-archiver-1 on matomo151 is CRITICAL: CRITICAL: Status of the systemd unit matomo-archiver-1
[16:59:31] <MacFanBot>	 [02puppet] 07paladox merged pull request #4683: Kafka: Use proxy server when installing via archive class (03main...03paladox-patch-1) 13https://github.com/miraheze/puppet/pull/4683
[16:59:31] <MacFanBot>	 [02puppet] 07paladox pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/bc5aca33416f89b84e2971e79072a7fb32bf4764
[16:59:32] <MacFanBot>	 02puppet/03main 07paladox 03bc5aca3 Kafka: Use proxy server when installing via archive class (#4683)
[16:59:34] <MacFanBot>	 [02puppet] 07paladox 04deleted 03paladox-patch-1 at 0335a7241 13https://api.github.com/repos/miraheze/puppet/commit/35a7241
[16:59:50] <icinga-miraheze>	 RECOVERY - puppet181 Check unit status of git_pull_puppet on puppet181 is OK: OK: Status of the systemd unit git_pull_puppet
[17:00:27] <MacFanBot>	 [02puppet] 07paladox pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/850c04d219e0a7ff89447b047c9dabb42eb028c3
[17:00:27] <MacFanBot>	 02puppet/03main 07paladox 03850c04d kafka: switch to java 21 for trixie
[17:00:30] <icinga-miraheze>	 RECOVERY - puppet181 Check unit status of git_pull_ssl on puppet181 is OK: OK: Status of the systemd unit git_pull_ssl
[17:03:03] <icinga-miraheze>	 PROBLEM - kafka181 Puppet on kafka181 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 15 minutes ago with 0 failures
[17:06:27] <icinga-miraheze>	 PROBLEM - kafka182 Kafka Broker Server on kafka182 is CRITICAL: PROCS CRITICAL: 0 processes with command name 'java', args 'Kafka /opt/kafka/config/server.properties'
[17:07:35] <MacFanBot>	 [02puppet] 07paladox created 03paladox-patch-1 (+1 new commit) 13https://github.com/miraheze/puppet/commit/5a3bc00d3296
[17:07:35] <MacFanBot>	 02puppet/03paladox-patch-1 07paladox 035a3bc00 Kafka: Allow configuring zookeeper server…
[17:07:39] <MacFanBot>	 [02puppet] 07paladox opened pull request #4684: Kafka: Allow configuring zookeeper server (03main...03paladox-patch-1) 13https://github.com/miraheze/puppet/pull/4684
[17:07:46] <MacFanBot>	 [02puppet] 07coderabbitai[bot] commented on pull request #4684: <details> […] 13https://github.com/miraheze/puppet/pull/4684#issuecomment-3750618471
[17:08:03] <MacFanBot>	 [02puppet] 07paladox pushed 1 new commit to 03paladox-patch-1 13https://github.com/miraheze/puppet/commit/11298d5c63a8db757823b18ae299c4b4962dd015
[17:08:03] <MacFanBot>	 02puppet/03paladox-patch-1 07paladox 0311298d5 Update kafka181.yaml
[17:08:34] <MacFanBot>	 [02puppet] 07paladox pushed 1 new commit to 03paladox-patch-1 13https://github.com/miraheze/puppet/commit/04591466401aa5ab58aaa81933f866c13a1ea05b
[17:08:35] <MacFanBot>	 02puppet/03paladox-patch-1 07paladox 030459146 Update kafka182.yaml
[17:08:53] <MacFanBot>	 [02puppet] 07paladox pushed 1 new commit to 03paladox-patch-1 13https://github.com/miraheze/puppet/commit/e698ed276258b9d05317a7f67348d40b1e3b7395
[17:08:53] <MacFanBot>	 02puppet/03paladox-patch-1 07paladox 03e698ed2 Update kafka181.yaml
[17:09:13] <MacFanBot>	 [02puppet] 07paladox pushed 1 new commit to 03paladox-patch-1 13https://github.com/miraheze/puppet/commit/ba3ab4f232125baa8ed8c4fd9b64d70ef4edd840
[17:09:14] <MacFanBot>	 02puppet/03paladox-patch-1 07paladox 03ba3ab4f Update kafka.pp
[17:09:25] <MacFanBot>	 [02puppet] 07paladox pushed 1 new commit to 03paladox-patch-1 13https://github.com/miraheze/puppet/commit/5a5cd170e5debf2b7df1b99cd840e8bdc0e6afe8
[17:09:25] <MacFanBot>	 02puppet/03paladox-patch-1 07paladox 035a5cd17 Update kafka.pp
[17:09:39] <MacFanBot>	 [02puppet] 07paladox pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/535eca4908d1b49070bdac5efa738d1b92219c83
[17:09:39] <MacFanBot>	 02puppet/03main 07paladox 03535eca4 Kafka: Allow configuring zookeeper server (#4684)…
[17:09:39] <MacFanBot>	 [02puppet] 07paladox merged pull request #4684: Kafka: Allow configuring zookeeper server (03main...03paladox-patch-1) 13https://github.com/miraheze/puppet/pull/4684
[17:09:40] <MacFanBot>	 [02puppet] 07paladox 04deleted 03paladox-patch-1 at 035a5cd17 13https://api.github.com/repos/miraheze/puppet/commit/5a5cd17
[17:12:25] <icinga-miraheze>	 RECOVERY - kafka182 Kafka Broker Server on kafka182 is OK: PROCS OK: 1 process with command name 'java', args 'Kafka /opt/kafka/config/server.properties'
[17:15:03] <MacFanBot>	 [02puppet] 07paladox created 03paladox-patch-1 (+1 new commit) 13https://github.com/miraheze/puppet/commit/860040b271a3
[17:15:03] <MacFanBot>	 02puppet/03paladox-patch-1 07paladox 03860040b Switch to kafka182
[17:15:07] <MacFanBot>	 [02puppet] 07paladox opened pull request #4685: Switch to kafka182 (03main...03paladox-patch-1) 13https://github.com/miraheze/puppet/pull/4685
[17:15:16] <MacFanBot>	 [02puppet] 07coderabbitai[bot] commented on pull request #4685: <details> […] 13https://github.com/miraheze/puppet/pull/4685#issuecomment-3750648185
[17:15:31] <MacFanBot>	 [02puppet] 07paladox pushed 1 new commit to 03paladox-patch-1 13https://github.com/miraheze/puppet/commit/0f1f656d093357fdd706ffed38e3dc6e50a2c371
[17:15:31] <MacFanBot>	 02puppet/03paladox-patch-1 07paladox 030f1f656 Update default.config.yaml
[17:15:46] <MacFanBot>	 [02puppet] 07paladox pushed 1 new commit to 03paladox-patch-1 13https://github.com/miraheze/puppet/commit/606e50bc5eff7d049866f08dbb433f362510e32d
[17:15:46] <MacFanBot>	 02puppet/03paladox-patch-1 07paladox 03606e50b Update config.yaml
[17:17:25] <MacFanBot>	 [02puppet] 07paladox merged pull request #4685: Switch to kafka182 (03main...03paladox-patch-1) 13https://github.com/miraheze/puppet/pull/4685
[17:17:25] <MacFanBot>	 [02puppet] 07paladox pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/5c22a703f6f2123704e8821f3fc39e5e264a9307
[17:17:25] <MacFanBot>	 02puppet/03main 07paladox 035c22a70 Switch to kafka182 (#4685)
[17:17:27] <MacFanBot>	 [02puppet] 07paladox 04deleted 03paladox-patch-1 at 03606e50b 13https://api.github.com/repos/miraheze/puppet/commit/606e50b
[17:17:51] <MirahezeRelay>	 <paladox> !log switch kafka181 to kafka182
[17:17:53] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[17:22:03] <MirahezeRelay>	 <paladox> !log shutdown kafka181
[17:22:05] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[17:26:42] <MacFanBot>	 [02puppet] 07paladox created 03paladox-patch-1 (+1 new commit) 13https://github.com/miraheze/puppet/commit/d672b79e1aa0
[17:26:42] <MacFanBot>	 02puppet/03paladox-patch-1 07paladox 03d672b79 burrow: Ensure /run/burrow is created
[17:26:46] <MacFanBot>	 [02puppet] 07paladox opened pull request #4686: burrow: Ensure /run/burrow is created (03main...03paladox-patch-1) 13https://github.com/miraheze/puppet/pull/4686
[17:26:58] <MacFanBot>	 [02puppet] 07coderabbitai[bot] commented on pull request #4686: <details> […] 13https://github.com/miraheze/puppet/pull/4686#issuecomment-3750699413
[17:27:11] <MacFanBot>	 [02puppet] 07paladox merged pull request #4686: burrow: Ensure /run/burrow is created (03main...03paladox-patch-1) 13https://github.com/miraheze/puppet/pull/4686
[17:27:11] <MacFanBot>	 [02puppet] 07paladox pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/69e31f1358e38f49affb98a4f374b85bfa82e42e
[17:27:11] <MacFanBot>	 02puppet/03main 07paladox 0369e31f1 burrow: Ensure /run/burrow is created (#4686)
[17:27:12] <MacFanBot>	 [02puppet] 07paladox 04deleted 03paladox-patch-1 at 03d672b79 13https://api.github.com/repos/miraheze/puppet/commit/d672b79
[17:28:05] <miraheze-github>	 miraheze/puppet - paladox the build has errored.
[17:29:35] <miraheze-github>	 miraheze/puppet - paladox the build has errored.
[17:30:57] <MacFanBot>	 [02puppet] 07paladox pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/4f350c85c150fc0a89c4df8145062c76d95f1b9b
[17:30:58] <MacFanBot>	 02puppet/03main 07paladox 034f350c8 Revert "burrow: Ensure /run/burrow is created (#4686)"…
[17:31:34] <icinga-miraheze>	 PROBLEM - cp201 Varnish Backends on cp201 is CRITICAL: 4 backends are down. mw162 mw182 mw201 mw203
[17:31:50] <icinga-miraheze>	 PROBLEM - cp161 Varnish Backends on cp161 is CRITICAL: 3 backends are down. mw161 mw182 mw173
[17:31:59] <icinga-miraheze>	 PROBLEM - cp171 Varnish Backends on cp171 is CRITICAL: 4 backends are down. mw161 mw153 mw173 mw192
[17:32:26] <icinga-miraheze>	 [Grafana] FIRING: The mediawiki JobQueue backlog is increasing by more than 100 jobs a minute over an extended time period https://grafana.wikitide.net/d/GtxbP1Xnk?from=1768408310000&orgId=1&to=1768411946677
[17:32:31] <icinga-miraheze>	 PROBLEM - kafka182 Kafka Broker Server on kafka182 is CRITICAL: PROCS CRITICAL: 0 processes with command name 'java', args 'Kafka /opt/kafka/config/server.properties'
[17:32:44] <icinga-miraheze>	 PROBLEM - kafka182 Puppet on kafka182 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[17:33:02] <icinga-miraheze>	 PROBLEM - cp191 Varnish Backends on cp191 is CRITICAL: 7 backends are down. mw151 mw152 mw172 mw181 mw183 mw202 mw203
[17:33:56] <icinga-miraheze>	 RECOVERY - cp171 Varnish Backends on cp171 is OK: All 31 backends are healthy
[17:34:31] <icinga-miraheze>	 RECOVERY - kafka182 Kafka Broker Server on kafka182 is OK: PROCS OK: 1 process with command name 'java', args 'Kafka /opt/kafka/config/server.properties'
[17:34:40] <icinga-miraheze>	 RECOVERY - kafka182 Puppet on kafka182 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[17:35:02] <icinga-miraheze>	 RECOVERY - cp191 Varnish Backends on cp191 is OK: All 31 backends are healthy
[17:35:34] <icinga-miraheze>	 RECOVERY - cp201 Varnish Backends on cp201 is OK: All 31 backends are healthy
[17:35:50] <icinga-miraheze>	 RECOVERY - cp161 Varnish Backends on cp161 is OK: All 31 backends are healthy
[17:37:26] <icinga-miraheze>	 [Grafana] RESOLVED: DatasourceNoData https://grafana.wikitide.net/d/GtxbP1Xnk?from=1768408310000&orgId=1&to=1768412150000
[17:59:10] <MirahezeRelay>	 <paladox> !log delete kafka181
[17:59:12] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[18:00:15] <MacFanBot>	 [02dns] 07paladox created 03paladox-patch-1 (+1 new commit) 13https://github.com/miraheze/dns/commit/b4f34ba4f2b5
[18:00:15] <MacFanBot>	 02dns/03paladox-patch-1 07paladox 03b4f34ba Remove kafka181
[18:00:25] <MacFanBot>	 [02dns] 07paladox opened pull request #709: Remove kafka181 (03main...03paladox-patch-1) 13https://github.com/miraheze/dns/pull/709
[18:00:53] <MacFanBot>	 [02dns] 07paladox pushed 1 new commit to 03paladox-patch-1 13https://github.com/miraheze/dns/commit/74a6034509b665cf5cebc01b95de6754f7f68aa2
[18:00:54] <MacFanBot>	 02dns/03paladox-patch-1 07paladox 0374a6034 Update 10.in-addr.arpa
[18:01:02] <MacFanBot>	 [02dns] 07paladox pushed 1 new commit to 03paladox-patch-1 13https://github.com/miraheze/dns/commit/6bce30edce2ffb9a880bf8b43c8bf565d517082f
[18:01:03] <MacFanBot>	 02dns/03paladox-patch-1 07paladox 036bce30e Update wtnet
[18:01:10] <MacFanBot>	 [02dns] 07paladox pushed 1 new commit to 03main 13https://github.com/miraheze/dns/commit/afca9682c999dbba31eef829345af849fc58e05f
[18:01:10] <MacFanBot>	 02dns/03main 07paladox 03afca968 Remove kafka181 (#709)
[18:01:10] <MacFanBot>	 [02dns] 07paladox merged pull request #709: Remove kafka181 (03main...03paladox-patch-1) 13https://github.com/miraheze/dns/pull/709
[18:01:12] <MacFanBot>	 [02dns] 07paladox 04deleted 03paladox-patch-1 at 036bce30e 13https://api.github.com/repos/miraheze/dns/commit/6bce30e
[18:01:20] <miraheze-github>	 miraheze/dns - paladox the build passed.
[18:01:53] <miraheze-github>	 miraheze/dns - paladox the build passed.
[18:02:00] <miraheze-github_>	 miraheze/dns - paladox the build passed.
[18:02:21] <miraheze-github>	 miraheze/dns - paladox the build passed.
[18:58:10] <icinga-miraheze>	 PROBLEM - puppet181 Check unit status of listdomains_github_push on puppet181 is CRITICAL: CRITICAL: Status of the systemd unit listdomains_github_push
[19:06:00] <icinga-miraheze>	 RECOVERY - puppet181 Check unit status of listdomains_github_push on puppet181 is OK: OK: Status of the systemd unit listdomains_github_push
[19:35:20] <MacFanBot>	 [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/f8b10e7ed0bb97314705e13d54ad3a1f6c055987
[19:35:20] <MacFanBot>	 02ssl/03main 07WikiTideBot 03f8b10e7 Bot: Auto-update domain lists
[20:26:10] <MacFanBot>	 [02puppet] 07MacFan4000 opened pull request #4687: relaybot: remove ballmedia channels (07miraheze:03main...07MacFan4000:03patch-71) 13https://github.com/miraheze/puppet/pull/4687
[20:26:22] <MacFanBot>	 [02puppet] 07coderabbitai[bot] commented on pull request #4687: <details> […] 13https://github.com/miraheze/puppet/pull/4687#issuecomment-3751582902
[21:14:02] <MirahezeLSBot>	 !log [macfan@test151] starting deploy of {'versions': '1.45', 'upgrade_extensions': 'CheckUser'} to test151
[21:14:03] <MirahezeLSBot>	 !log [macfan@test151] finished deploy of {'versions': '1.45', 'upgrade_extensions': 'CheckUser'} to test151 - SUCCESS in 1s
[21:14:04] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[21:14:06] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[21:21:47] <MirahezeLSBot>	 !log [skye@mwtask181] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php CirrusSearch:UpdateSearchIndexConfig --wiki=ruanimalroyalewiki --startOver (END - exit=0)
[21:21:49] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[21:23:27] <MirahezeLSBot>	 !log [skye@mwtask181] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php CirrusSearch:ForceSearchIndex --wiki=ruanimalroyalewiki (END - exit=0)
[21:23:28] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[21:24:58] <MirahezeLSBot>	 !log [paladox@mwtask181] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php /srv/mediawiki/1.44/extensions/CirrusSearch/maintenance/UpdateSearchIndexConfig.php --startOver --wiki ruanimalroyalewiki
[21:25:00] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[21:25:06] <MirahezeLSBot>	 !log [paladox@mwtask181] sudo -u www-data php /srv/mediawiki/1.44/maintenance/run.php /srv/mediawiki/1.44/extensions/CirrusSearch/maintenance/ForceSearchIndex.php --wiki ruanimalroyalewiki
[21:25:08] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[21:26:30] <MirahezeRelay>	 <pskyechology> double kill
[21:31:14] <MirahezeRelay>	 <paladox> @pskyechology iirc you had to delete the index, script was useless
[21:31:39] <MirahezeRelay>	 <pskyechology, replying to paladox> yeah I did that first and then ran the scripts, as described in [[Tech:CirrusSearch]]
[21:31:40] <MirahezeRelay>	 <Wiki-Bot#2998> <https://meta.miraheze.org/wiki/Tech:CirrusSearch>
[21:31:41] <MirahezeRelay>	 <paladox> kept erroring with `Primary index was expected to be an alias: ruanimalroyalewiki_content`
[21:32:03] <MirahezeRelay>	 <pskyechology> if you delete them, what's it gonna query after if you dont index them again?
[21:32:05] <MirahezeRelay>	 <paladox> oh you directly did a DELETE on the os servers?
[21:32:25] <MirahezeRelay>	 <paladox> UpdateSearchIndexConfig didn't delete it iirc (i ran it, which is what gave me the error multiple times)
[21:32:50] <MirahezeRelay>	 <paladox> had to do `curl -X DELETE 'localhost:9200/ruanimalroyalewiki_content`
[21:33:59] <MirahezeRelay>	 <pskyechology, replying to paladox> yeah that's why the instructions on the page I just linked ask you to send that DELETE request first
[21:34:15] <MirahezeRelay>	 <pskyechology> doesnt necessarily have to be done locally
[21:34:18] <MirahezeRelay>	 <abaddriverlol, replying to paladox> you can just do https://discord.com/channels/407504499280707585/808001911868489748/1414625406270701701 on any server
[21:34:20] <MirahezeRelay>	 <paladox> oh i see
[21:38:31] <icinga-miraheze>	 PROBLEM - mw201 MediaWiki Rendering on mw201 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[21:38:40] <icinga-miraheze>	 PROBLEM - cp171 Varnish Backends on cp171 is CRITICAL: 9 backends are down. mw151 mw152 mw161 mw162 mw172 mw181 mw153 mw173 mw191
[21:38:49] <icinga-miraheze>	 PROBLEM - cp161 Varnish Backends on cp161 is CRITICAL: 10 backends are down. mw151 mw152 mw161 mw162 mw172 mw181 mw153 mw173 mw191 mw193
[21:39:15] <icinga-miraheze>	 PROBLEM - cp191 HTTPS on cp191 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502
[21:39:18] <icinga-miraheze>	 PROBLEM - test151 MediaWiki Rendering on test151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[21:39:31] <icinga-miraheze>	 PROBLEM - cp201 Varnish Backends on cp201 is CRITICAL: 8 backends are down. mw161 mw171 mw182 mw183 mw192 mw193 mw202 mw203
[21:39:42] <icinga-miraheze>	 PROBLEM - mw201 HTTPS on mw201 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10006 milliseconds with 0 bytes received
[21:39:42] <icinga-miraheze>	 PROBLEM - mw163 MediaWiki Rendering on mw163 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 8191 bytes in 0.010 second response time
[21:39:48] <MirahezeRelay>	 <pskyechology> wtf
[21:40:14] <icinga-miraheze>	 PROBLEM - cp191 Varnish Backends on cp191 is CRITICAL: 10 backends are down. mw151 mw152 mw162 mw172 mw181 mw153 mw173 mw191 mw193 mw202
[21:40:20] <icinga-miraheze>	 PROBLEM - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is WARNING: WARNING - NGINX Error Rate is 58%
[21:40:31] <MirahezeRelay>	 <paladox> confused why things went down
[21:41:03] <icinga-miraheze>	 PROBLEM - mw163 HTTPS on mw163 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502
[21:41:40] <icinga-miraheze>	 RECOVERY - mw201 HTTPS on mw201 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4254 bytes in 4.352 second response time
[21:41:46] <icinga-miraheze>	 RECOVERY - mw163 MediaWiki Rendering on mw163 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 4.464 second response time
[21:42:14] <icinga-miraheze>	 RECOVERY - cp191 Varnish Backends on cp191 is OK: All 31 backends are healthy
[21:42:19] <icinga-miraheze>	 RECOVERY - mw201 MediaWiki Rendering on mw201 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.172 second response time
[21:42:40] <icinga-miraheze>	 RECOVERY - cp171 Varnish Backends on cp171 is OK: All 31 backends are healthy
[21:42:49] <icinga-miraheze>	 RECOVERY - cp161 Varnish Backends on cp161 is OK: All 31 backends are healthy
[21:43:03] <icinga-miraheze>	 RECOVERY - mw163 HTTPS on mw163 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4254 bytes in 0.060 second response time
[21:43:07] <icinga-miraheze>	 RECOVERY - cp191 HTTPS on cp191 is OK: HTTP OK: HTTP/2 410 - Status line output matched "HTTP/2 410" - 4325 bytes in 0.059 second response time
[21:43:17] <icinga-miraheze>	 RECOVERY - test151 MediaWiki Rendering on test151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.563 second response time
[21:43:31] <icinga-miraheze>	 RECOVERY - cp201 Varnish Backends on cp201 is OK: All 31 backends are healthy
[21:44:20] <icinga-miraheze>	 RECOVERY - cp161 HTTP 4xx/5xx ERROR Rate on cp161 is OK: OK - NGINX Error Rate is 15%
[22:15:24] <MacFanBot>	 [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/fc362a4710a4a72db986d512955d3b642f5684a1
[22:15:24] <MacFanBot>	 02ssl/03main 07WikiTideBot 03fc362a4 Bot: Auto-update domain lists
[22:35:16] <MacFanBot>	 [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/e7886e10706412d29b527f38fb3c9fd5bafe620b
[22:35:16] <MacFanBot>	 02ssl/03main 07WikiTideBot 03e7886e1 Bot: Auto-update domain lists