[00:00:10] <icinga-miraheze>	 PROBLEM - cp27 HTTP 4xx/5xx ERROR Rate on cp27 is CRITICAL: CRITICAL - NGINX Error Rate is 100%
[00:04:04] <icinga-miraheze>	 PROBLEM - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is CRITICAL: CRITICAL - NGINX Error Rate is 67%
[00:05:11] <icinga-miraheze>	 RECOVERY - cp36 Disk Space on cp36 is OK: DISK OK - free space: / 20385MiB (23% inode=98%);
[00:05:51] <icinga-miraheze>	 RECOVERY - cp37 Disk Space on cp37 is OK: DISK OK - free space: / 20501MiB (23% inode=98%);
[00:06:06] <icinga-miraheze>	 PROBLEM - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is WARNING: WARNING - NGINX Error Rate is 43%
[00:10:05] <icinga-miraheze>	 RECOVERY - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is OK: OK - NGINX Error Rate is 38%
[00:14:07] <icinga-miraheze>	 PROBLEM - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is WARNING: WARNING - NGINX Error Rate is 47%
[00:14:31] <botifico>	 [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03article-paths [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/05b1dd78295a
[00:14:34] <botifico>	 [02miraheze/mw-config] 07Universal-Omega 0305b1dd7 - Support changing article path and using $wgMainPageIsDomainRoot
[00:14:36] <botifico>	 [02mw-config] 07Universal-Omega created branch 03article-paths - 13https://github.com/miraheze/mw-config
[00:14:37] <botifico>	 [02mw-config] 07Universal-Omega opened pull request 03#5478: Support changing article path and using $wgMainPageIsDomainRoot - 13https://github.com/miraheze/mw-config/pull/5478
[00:15:06] <botifico>	 [02miraheze/mw-config] 07github-actions[bot] pushed 031 commit to 03article-paths [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/05b1dd78295a...8f1ffea7e9bf
[00:15:09] <botifico>	 [02miraheze/mw-config] 07github-actions 038f1ffea - CI: lint code to MediaWiki standards
[00:15:11] <botifico>	 [02mw-config] 07github-actions[bot] synchronize pull request 03#5478: Support changing article path and using $wgMainPageIsDomainRoot - 13https://github.com/miraheze/mw-config/pull/5478
[00:15:43] <miraheze-github>	 miraheze/mw-config - Universal-Omega the build passed.
[00:15:59] <botifico>	 [02mw-config] 07Universal-Omega synchronize pull request 03#5478: Support changing article path and using $wgMainPageIsDomainRoot - 13https://github.com/miraheze/mw-config/pull/5478
[00:16:02] <botifico>	 [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03article-paths [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/8f1ffea7e9bf...d408115eb8be
[00:16:03] <botifico>	 [02miraheze/mw-config] 07Universal-Omega 03d408115 - Just disable if not managewiki-restricted
[00:16:05] <icinga-miraheze>	 RECOVERY - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is OK: OK - NGINX Error Rate is 36%
[00:17:06] <miraheze-github>	 miraheze/mw-config - Universal-Omega the build passed.
[00:28:05] <icinga-miraheze>	 PROBLEM - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is WARNING: WARNING - NGINX Error Rate is 42%
[01:02:09] <icinga-miraheze>	 PROBLEM - cp27 HTTP 4xx/5xx ERROR Rate on cp27 is WARNING: WARNING - NGINX Error Rate is 59%
[01:57:26] <botifico>	 [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03cache [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/ca3764bacdd1
[01:57:28] <botifico>	 [02miraheze/mw-config] 07Universal-Omega 03ca3764b - Update some caching configs
[01:57:31] <botifico>	 [02mw-config] 07Universal-Omega created branch 03cache - 13https://github.com/miraheze/mw-config
[01:57:32] <botifico>	 [02mw-config] 07Universal-Omega opened pull request 03#5479: Update some caching configs - 13https://github.com/miraheze/mw-config/pull/5479
[01:57:57] <botifico>	 [02miraheze/mw-config] 07github-actions[bot] pushed 031 commit to 03cache [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/ca3764bacdd1...f877fd8e68eb
[01:57:58] <botifico>	 [02miraheze/mw-config] 07github-actions 03f877fd8 - CI: lint code to MediaWiki standards
[01:57:59] <botifico>	 [02mw-config] 07github-actions[bot] synchronize pull request 03#5479: Update some caching configs - 13https://github.com/miraheze/mw-config/pull/5479
[01:58:26] <miraheze-github>	 miraheze/mw-config - Universal-Omega the build passed.
[02:06:58] <botifico>	 [02mw-config] 07Universal-Omega closed pull request 03#5479: Update some caching configs - 13https://github.com/miraheze/mw-config/pull/5479
[02:07:00] <botifico>	 [02miraheze/mw-config] 07Universal-Omega deleted branch 03cache
[02:07:01] <botifico>	 [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/mw-config/compare/77cf2a4dceb4...a489fdf2fcf9
[02:07:03] <botifico>	 [02miraheze/mw-config] 07Universal-Omega 03a489fdf - Update some caching configs (#5479)
[02:07:05] <botifico>	 [02mw-config] 07Universal-Omega deleted branch 03cache - 13https://github.com/miraheze/mw-config
[02:07:17] <botifico>	 [02miraheze/mw-config] 07Universal-Omega pushed 032 commits to 03article-paths [+0/-0/±2] 13https://github.com/miraheze/mw-config/compare/d408115eb8be...5d52ccda9306
[02:07:18] <botifico>	 [02miraheze/mw-config] 07Universal-Omega 0334d1f6c - Support changing article path and using $wgMainPageIsDomainRoot
[02:07:21] <botifico>	 [02miraheze/mw-config] 07Universal-Omega 035d52ccd - Just disable if not managewiki-restricted
[02:07:22] <botifico>	 [02mw-config] 07Universal-Omega synchronize pull request 03#5478: Support changing article path and using $wgMainPageIsDomainRoot - 13https://github.com/miraheze/mw-config/pull/5478
[02:07:42] <botifico>	 [02mw-config] 07Universal-Omega closed pull request 03#5477: Test some changes for performance - 13https://github.com/miraheze/mw-config/pull/5477
[02:07:45] <botifico>	 [02miraheze/mw-config] 07Universal-Omega deleted branch 03performance
[02:07:46] <botifico>	 [02mw-config] 07Universal-Omega deleted branch 03performance - 13https://github.com/miraheze/mw-config
[02:07:59] <miraheze-github>	 miraheze/mw-config - Universal-Omega the build passed.
[02:08:18] <miraheze-github>	 miraheze/mw-config - Universal-Omega the build passed.
[02:08:51] <MirahezeLSBot>	 !log [universalomega@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all
[02:08:54] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[02:09:00] <MirahezeLSBot>	 !log [universalomega@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 9s
[02:09:03] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[03:18:03] <icinga-miraheze>	 RECOVERY - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is OK: OK - NGINX Error Rate is 39%
[04:06:47] <icinga-miraheze>	 PROBLEM - wiki.hspdev.0t0.jp - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.
[04:20:02] <icinga-miraheze>	 PROBLEM - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is WARNING: WARNING - NGINX Error Rate is 40%
[04:35:22] <icinga-miraheze>	 RECOVERY - wiki.hspdev.0t0.jp - reverse DNS on sslhost is OK: SSL OK - wiki.hspdev.0t0.jp reverse DNS resolves to cp37.wikitide.net - CNAME OK
[06:38:25] <icinga-miraheze>	 PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query line.pm. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL
[07:26:02] <icinga-miraheze>	 RECOVERY - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is OK: OK - NGINX Error Rate is 39%
[07:36:03] <icinga-miraheze>	 PROBLEM - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is WARNING: WARNING - NGINX Error Rate is 40%
[07:37:06] <icinga-miraheze>	 RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp37.wikitide.net - CNAME OK
[09:33:48] <icinga-miraheze>	 RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp37.wikitide.net - CNAME OK
[09:40:05] <icinga-miraheze>	 RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp36.wikitide.net - CNAME OK
[12:15:25] <botifico>	 [02miraheze/ErrorPages] 07translatewiki pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ErrorPages/compare/2210250467b8...e6877e9c40b6
[12:15:26] <botifico>	 [02miraheze/ErrorPages] 07translatewiki 03e6877e9 - Localisation updates from https://translatewiki.net.
[12:15:29] <botifico>	 [02miraheze/GlobalNewFiles] 07translatewiki pushed 031 commit to 03master [+1/-0/±0] 13https://github.com/miraheze/GlobalNewFiles/compare/6c9934c84d4a...799bedba08ba
[12:15:30] <botifico>	 [02miraheze/GlobalNewFiles] 07translatewiki 03799bedb - Localisation updates from https://translatewiki.net.
[12:15:33] <botifico>	 [02miraheze/CreateWiki] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/62a4fcb6a67b...1742a9524957
[12:15:36] <botifico>	 [02miraheze/CreateWiki] 07translatewiki 031742a95 - Localisation updates from https://translatewiki.net.
[12:15:39] <botifico>	 [02miraheze/DataDump] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/DataDump/compare/5ae3b8c82577...2bb228a63ee8
[12:15:40] <botifico>	 [02miraheze/DataDump] 07translatewiki 032bb228a - Localisation updates from https://translatewiki.net.
[12:15:43] <botifico>	 [02miraheze/ImportDump] 07translatewiki pushed 031 commit to 03master [+1/-0/±0] 13https://github.com/miraheze/ImportDump/compare/85627784cc56...5cb64d95ee15
[12:15:46] <botifico>	 [02miraheze/ImportDump] 07translatewiki 035cb64d9 - Localisation updates from https://translatewiki.net.
[12:15:48] <botifico>	 [02miraheze/MatomoAnalytics] 07translatewiki pushed 031 commit to 03master [+1/-0/±0] 13https://github.com/miraheze/MatomoAnalytics/compare/00dfeb47902e...9faf6b76cd3f
[12:15:51] <botifico>	 [02miraheze/MatomoAnalytics] 07translatewiki 039faf6b7 - Localisation updates from https://translatewiki.net.
[12:15:54] <botifico>	 [02miraheze/ManageWiki] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ManageWiki/compare/7bc33a6e641a...e35273da0d0f
[12:15:55] <botifico>	 [02miraheze/ManageWiki] 07translatewiki 03e35273d - Localisation updates from https://translatewiki.net.
[12:15:58] <botifico>	 [02miraheze/landing] 07translatewiki pushed 031 commit to 03master [+0/-0/±5] 13https://github.com/miraheze/landing/compare/a4af7acb7440...2efd8dad833a
[12:16:01] <botifico>	 [02miraheze/landing] 07translatewiki 032efd8da - Localisation updates from https://translatewiki.net.
[12:16:04] <botifico>	 [02miraheze/RemovePII] 07translatewiki pushed 031 commit to 03master [+1/-0/±0] 13https://github.com/miraheze/RemovePII/compare/23b40a1138f5...b6240d59a590
[12:16:07] <botifico>	 [02miraheze/RemovePII] 07translatewiki 03b6240d5 - Localisation updates from https://translatewiki.net.
[12:16:09] <botifico>	 [02miraheze/MirahezeMagic] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/MirahezeMagic/compare/16eee6be8a24...ff7c55eec8a4
[12:16:12] <botifico>	 [02miraheze/MirahezeMagic] 07translatewiki 03ff7c55e - Localisation updates from https://translatewiki.net.
[12:16:13] <botifico>	 [02miraheze/WikiDiscover] 07translatewiki pushed 031 commit to 03master [+1/-0/±0] 13https://github.com/miraheze/WikiDiscover/compare/acefe8616d15...ff62a48ca988
[12:16:15] <botifico>	 [02miraheze/WikiDiscover] 07translatewiki 03ff62a48 - Localisation updates from https://translatewiki.net.
[12:17:22] <miraheze-github>	 miraheze/ErrorPages - translatewiki the build passed.
[12:19:24] <miraheze-github>	 miraheze/landing - translatewiki the build passed.
[12:19:52] <miraheze-github>	 miraheze/ManageWiki - translatewiki the build passed.
[12:20:03] <miraheze-github>	 miraheze/GlobalNewFiles - translatewiki the build passed.
[12:20:14] <miraheze-github>	 miraheze/DataDump - translatewiki the build passed.
[12:20:39] <miraheze-github>	 miraheze/MatomoAnalytics - translatewiki the build passed.
[12:22:14] <miraheze-github>	 miraheze/CreateWiki - translatewiki the build passed.
[12:22:31] <miraheze-github>	 miraheze/ImportDump - translatewiki the build passed.
[12:23:26] <miraheze-github>	 miraheze/WikiDiscover - translatewiki the build passed.
[12:23:28] <MirahezeLSBot>	 !log [@test151] starting deploy of {'folders': '1.41/extensions/MirahezeMagic'} to test151
[12:23:29] <MirahezeLSBot>	 !log [@test151] finished deploy of {'folders': '1.41/extensions/MirahezeMagic'} to test151 - SUCCESS in 0s
[12:23:30] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[12:23:33] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[12:23:37] <MirahezeLSBot>	 !log [@test151] starting deploy of {'folders': '1.42/extensions/MirahezeMagic'} to test151
[12:23:38] <MirahezeLSBot>	 !log [@test151] finished deploy of {'folders': '1.42/extensions/MirahezeMagic'} to test151 - SUCCESS in 0s
[12:23:40] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[12:23:42] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[12:25:00] <miraheze-github>	 miraheze/MirahezeMagic - translatewiki the build has errored.
[12:26:04] <miraheze-github>	 miraheze/RemovePII - translatewiki the build passed.
[12:27:30] <MirahezeLSBot>	 !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importDump.php --wiki=bornagainwiki --no-updates --username-prefix=mh:bornagain /home/reception/ImportDump/bornagainwiki-20240211084951.xml (START)
[12:27:33] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[12:33:18] <MirahezeLSBot>	 !log [@mwtask181] starting deploy of {'folders': '1.41/extensions/MirahezeMagic'} to all
[12:33:20] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[12:33:22] <icinga-miraheze>	 PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mahdiruiz.line.pm All nameservers failed to answer the query.
[12:33:26] <MirahezeLSBot>	 !log [@mwtask181] finished deploy of {'folders': '1.41/extensions/MirahezeMagic'} to all - SUCCESS in 8s
[12:33:28] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[12:38:03] <icinga-miraheze>	 PROBLEM - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is CRITICAL: CRITICAL - NGINX Error Rate is 60%
[12:42:02] <icinga-miraheze>	 PROBLEM - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is WARNING: WARNING - NGINX Error Rate is 59%
[12:44:03] <icinga-miraheze>	 PROBLEM - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is CRITICAL: CRITICAL - NGINX Error Rate is 60%
[12:56:07] <icinga-miraheze>	 PROBLEM - cp27 HTTP 4xx/5xx ERROR Rate on cp27 is CRITICAL: CRITICAL - NGINX Error Rate is 60%
[12:59:32] <MirahezeLSBot>	 !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importDump.php --wiki=bornagainwiki --no-updates --username-prefix=mh:bornagain /home/reception/ImportDump/bornagainwiki-20240211084951.xml (END - exit=0)
[12:59:33] <MirahezeLSBot>	 !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/rebuildall.php --wiki=bornagainwiki (START)
[12:59:34] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[12:59:37] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[13:00:03] <icinga-miraheze>	 PROBLEM - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is WARNING: WARNING - NGINX Error Rate is 59%
[13:02:41] <icinga-miraheze>	 RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp37.wikitide.net - CNAME OK
[13:05:18] <MirahezeLSBot>	 !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/rebuildall.php --wiki=bornagainwiki (END - exit=0)
[13:05:19] <MirahezeLSBot>	 !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/initSiteStats.php --wiki=bornagainwiki --active --update (END - exit=0)
[13:05:21] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[13:05:23] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[13:08:02] <icinga-miraheze>	 PROBLEM - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is CRITICAL: CRITICAL - NGINX Error Rate is 60%
[15:27:11] <icinga-miraheze>	 PROBLEM - cp36 Disk Space on cp36 is WARNING: DISK WARNING - free space: / 9701MiB (10% inode=98%);
[15:59:51] <icinga-miraheze>	 PROBLEM - cp37 Disk Space on cp37 is WARNING: DISK WARNING - free space: / 9703MiB (10% inode=98%);
[16:10:05] <MirahezeRelay>	 <songngu.xyz> Waiting for MHM + CW + ManageWiki get deployed
[16:32:52] <icinga-miraheze>	 PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query line.pm. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL
[16:37:29] <icinga-miraheze>	 PROBLEM - cp41 Puppet on cp41 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[17:02:11] <icinga-miraheze>	 PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mahdiruiz.line.pm All nameservers failed to answer the query.
[17:05:28] <icinga-miraheze>	 RECOVERY - cp41 Puppet on cp41 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures
[17:17:27] <botifico>	 [02mw-config] 07Universal-Omega closed pull request 03#5476: Set wgKartographerDfltStyle to . - 13https://github.com/miraheze/mw-config/pull/5476
[17:17:30] <botifico>	 [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/a489fdf2fcf9...cf33f21b8554
[17:17:33] <botifico>	 [02miraheze/mw-config] 07redbluegreenhat 03cf33f21 - Set wgKartographerDfltStyle to . (#5476)
[17:18:20] <miraheze-github>	 miraheze/mw-config - Universal-Omega the build passed.
[17:18:36] <icinga-miraheze>	 PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 142.99 ms
[17:24:33] <icinga-miraheze>	 RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 144.90 ms
[17:26:45] <icinga-miraheze>	 PROBLEM - test151 Puppet on test151 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki config]
[17:31:30] <icinga-miraheze>	 RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp37.wikitide.net - CNAME OK
[17:32:03] <MirahezeLSBot>	 !log [@mwtask181] starting deploy of {'config': True} to all
[17:32:06] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[17:32:12] <MirahezeLSBot>	 !log [@mwtask181] finished deploy of {'config': True} to all - SUCCESS in 8s
[17:32:14] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[17:51:14] <icinga-miraheze>	 PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 144.27 ms
[17:53:12] <icinga-miraheze>	 RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 144.43 ms
[18:54:11] <icinga-miraheze>	 PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 100%
[18:54:38] <icinga-miraheze>	 PROBLEM - cp41 HTTPS on cp41 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Connection timed out after 10003 milliseconds
[18:55:26] <icinga-miraheze>	 PROBLEM - Host cp41 is DOWN: PING CRITICAL - Packet loss = 100%
[19:02:21] <icinga-miraheze>	 RECOVERY - Host cp41 is UP: PING OK - Packet loss = 0%, RTA = 131.07 ms
[19:03:04] <icinga-miraheze>	 RECOVERY - cp41 HTTPS on cp41 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3587 bytes in 0.909 second response time
[19:05:23] <icinga-miraheze>	 RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 136.49 ms
[19:06:02] <icinga-miraheze>	 PROBLEM - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is WARNING: WARNING - NGINX Error Rate is 59%
[19:22:43] <icinga-miraheze>	 PROBLEM - ping6 on cp28 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 158.06 ms
[19:24:44] <icinga-miraheze>	 RECOVERY - ping6 on cp28 is OK: PING OK - Packet loss = 0%, RTA = 148.98 ms
[20:05:35] <icinga-miraheze>	 [Grafana] !sre FIRING: The mediawiki job queue has more than 2500 unclaimed jobs https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1
[20:17:36] <MirahezeLSBot>	 !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/runJobs.php --wiki=rainworldwiki --procs 64 (START)
[20:17:39] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[20:19:44] <MirahezeLSBot>	 !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/runJobs.php --wiki=rainworldwiki --procs 64 (END - exit=0)
[20:19:47] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[20:20:11] <MirahezeLSBot>	 !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/runJobs.php --wiki=battlecatswiki --procs 64 (START)
[20:20:14] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[20:21:28] <icinga-miraheze>	 PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 70.03, 41.52, 22.73
[20:25:35] <icinga-miraheze>	 [Grafana] !sre RESOLVED: High Job Queue Backlog https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1
[20:29:37] <MirahezeLSBot>	 !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/runJobs.php --wiki=battlecatswiki --procs 64 (END - exit=0)
[20:29:40] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[20:39:28] <icinga-miraheze>	 PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 9.80, 15.55, 23.35
[20:43:28] <icinga-miraheze>	 RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 9.62, 12.27, 20.22
[21:10:35] <icinga-miraheze>	 [Grafana] !sre FIRING: The mediawiki job queue has more than 2500 unclaimed jobs https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1
[21:13:00] <MirahezeLSBot>	 !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/extensions/MirahezeMagic/maintenance/restoreManageWikiBackup.php --wiki=trueearthwiki --filename temw.json (END - exit=0)
[21:13:03] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[21:21:51] <botifico>	 [02miraheze/dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/cc3b8ef2baf5...3b660bfc8f29
[21:21:52] <botifico>	 [02miraheze/dns] 07Universal-Omega 033b660bf - Add trueearth.wikitide.org to Miraheze
[21:22:27] <MirahezeLSBot>	 !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importDump.php --wiki=trueearthwiki --no-updates /home/universalomega/ImportDump/trueearthwiki-20240212212035.xml (START)
[21:22:30] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[21:26:07] <icinga-miraheze>	 PROBLEM - cp27 HTTP 4xx/5xx ERROR Rate on cp27 is WARNING: WARNING - NGINX Error Rate is 59%
[21:35:11] <icinga-miraheze>	 PROBLEM - cp36 Disk Space on cp36 is CRITICAL: DISK CRITICAL - free space: / 5308MiB (5% inode=98%);
[21:35:57] <MirahezeLSBot>	 !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importDump.php --wiki=trueearthwiki --no-updates /home/universalomega/ImportDump/trueearthwiki-20240212212035.xml (END - exit=0)
[21:35:58] <MirahezeLSBot>	 !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/rebuildall.php --wiki=trueearthwiki (START)
[21:35:59] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[21:36:03] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[21:36:56] <MirahezeLSBot>	 !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importImages.php --wiki=trueearthwiki --search-recursively /home/universalomega/trueearthwiki (START)
[21:36:58] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[21:39:40] <MirahezeLSBot>	 !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/rebuildall.php --wiki=trueearthwiki (END - exit=0)
[21:39:41] <MirahezeLSBot>	 !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/initSiteStats.php --wiki=trueearthwiki --active --update (END - exit=0)
[21:39:42] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[21:39:45] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[21:44:30] <MirahezeLSBot>	 !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importImages.php --wiki=trueearthwiki --search-recursively /home/universalomega/trueearthwiki (END - exit=0)
[21:44:32] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[21:52:07] <MirahezeLSBot>	 !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importDump.php --wiki=rushpediawiki --no-updates /home/universalomega/ImportDump/rushpediawiki-20240206174114.xml (START)
[21:52:10] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[21:54:12] <MirahezeLSBot>	 !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importDump.php --wiki=timshotwheelswiki --no-updates /home/universalomega/ImportDump/timshotwheelswiki-20240205194808.xml (START)
[21:54:14] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[21:54:16] <MirahezeLSBot>	 !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importDump.php --wiki=timshotwheelswiki --no-updates /home/universalomega/ImportDump/timshotwheelswiki-20240205194808.xml (END - exit=0)
[21:54:16] <MirahezeLSBot>	 !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/rebuildall.php --wiki=timshotwheelswiki (START)
[21:54:18] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[21:54:21] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[21:54:31] <MirahezeLSBot>	 !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/rebuildall.php --wiki=timshotwheelswiki (END - exit=0)
[21:54:32] <MirahezeLSBot>	 !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/initSiteStats.php --wiki=timshotwheelswiki --active --update (END - exit=0)
[21:54:34] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[21:54:36] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[21:55:40] <MirahezeLSBot>	 !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importDump.php --wiki=monstertruckencyclopediawiki --no-updates /home/universalomega/ImportDump/monstertruckencyclopediawiki-20240209223928.xml (START)
[21:55:42] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[21:56:36] <MirahezeLSBot>	 !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importDump.php --wiki=harvestellawiki --no-updates /home/universalomega/ImportDump/harvestellawiki-20240212191333.xml (START)
[21:56:38] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[21:56:43] <MirahezeLSBot>	 !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importDump.php --wiki=monstertruckencyclopediawiki --no-updates /home/universalomega/ImportDump/monstertruckencyclopediawiki-20240209223928.xml (END - exit=0)
[21:56:44] <MirahezeLSBot>	 !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/rebuildall.php --wiki=monstertruckencyclopediawiki (START)
[21:56:46] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[21:56:48] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[21:57:08] <MirahezeLSBot>	 !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/rebuildall.php --wiki=monstertruckencyclopediawiki (END - exit=0)
[21:57:09] <MirahezeLSBot>	 !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/initSiteStats.php --wiki=monstertruckencyclopediawiki --active --update (END - exit=0)
[21:57:10] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[21:57:13] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:00:02] <MirahezeLSBot>	 !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importDump.php --wiki=harvestellawiki --no-updates /home/universalomega/ImportDump/harvestellawiki-20240212191333.xml (END - exit=0)
[22:00:03] <MirahezeLSBot>	 !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/rebuildall.php --wiki=harvestellawiki (START)
[22:00:04] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:00:07] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:00:52] <MirahezeLSBot>	 !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/rebuildall.php --wiki=harvestellawiki (END - exit=0)
[22:00:53] <MirahezeLSBot>	 !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/initSiteStats.php --wiki=harvestellawiki --active --update (END - exit=0)
[22:00:55] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:00:57] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:03:18] <MirahezeLSBot>	 !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importDump.php --wiki=rushpediawiki --no-updates /home/universalomega/ImportDump/rushpediawiki-20240206174114.xml (END - exit=0)
[22:03:19] <MirahezeLSBot>	 !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/rebuildall.php --wiki=rushpediawiki (START)
[22:03:21] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:03:23] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:12:10] <MirahezeLSBot>	 !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/rebuildall.php --wiki=rushpediawiki (END - exit=0)
[22:12:11] <MirahezeLSBot>	 !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/initSiteStats.php --wiki=rushpediawiki --active --update (END - exit=0)
[22:12:12] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:12:15] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:17:51] <icinga-miraheze>	 PROBLEM - cp37 Disk Space on cp37 is CRITICAL: DISK CRITICAL - free space: / 5295MiB (5% inode=98%);
[23:21:23] <icinga-miraheze>	 PROBLEM - wiki.mcsoft.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.mcsoft.org could not be found
[23:32:07] <icinga-miraheze>	 PROBLEM - cp27 HTTP 4xx/5xx ERROR Rate on cp27 is CRITICAL: CRITICAL - NGINX Error Rate is 60%
[23:32:27] <botifico>	 [02miraheze/MirahezeMagic] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/MirahezeMagic/compare/ff7c55eec8a4...ac4c4fadaf0a
[23:32:30] <botifico>	 [02miraheze/MirahezeMagic] 07Universal-Omega 03ac4c4fa - Fix CI
[23:32:38] <MirahezeLSBot>	 !log [@mwtask181] starting deploy of {'folders': '1.41/extensions/MirahezeMagic'} to all
[23:32:40] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[23:32:46] <MirahezeLSBot>	 !log [@mwtask181] finished deploy of {'folders': '1.41/extensions/MirahezeMagic'} to all - SUCCESS in 8s
[23:32:49] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[23:36:07] <icinga-miraheze>	 PROBLEM - cp27 HTTP 4xx/5xx ERROR Rate on cp27 is WARNING: WARNING - NGINX Error Rate is 59%
[23:38:28] <miraheze-github>	 miraheze/MirahezeMagic - Universal-Omega the build passed.
[23:53:52] <MirahezeLSBot>	 !log [@test151] starting deploy of {'folders': '1.41/extensions/MirahezeMagic'} to test151
[23:53:53] <MirahezeLSBot>	 !log [@test151] finished deploy of {'folders': '1.41/extensions/MirahezeMagic'} to test151 - SUCCESS in 0s
[23:53:54] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[23:53:57] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[23:54:02] <MirahezeLSBot>	 !log [@test151] starting deploy of {'folders': '1.42/extensions/MirahezeMagic'} to test151
[23:54:03] <MirahezeLSBot>	 !log [@test151] finished deploy of {'folders': '1.42/extensions/MirahezeMagic'} to test151 - SUCCESS in 0s
[23:54:04] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[23:54:06] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log