[00:00:10] <icinga-miraheze>	 PROBLEM - cp27 HTTP 4xx/5xx ERROR Rate on cp27 is UNKNOWN: UNKNOWN - NGINX Error Rate is UNKNOWN
[00:02:04] <botifico>	 [02MirahezeMagic] 07amire80 opened pull request 03#479: Clarify wgRCLinkDays title and description - 13https://github.com/miraheze/MirahezeMagic/pull/479
[00:02:09] <icinga-miraheze>	 PROBLEM - cp27 HTTP 4xx/5xx ERROR Rate on cp27 is CRITICAL: CRITICAL - NGINX Error Rate is 67%
[00:15:59] <miraheze-github>	 miraheze/MirahezeMagic - amire80 the build passed.
[00:19:46] <icinga-miraheze>	 PROBLEM - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is WARNING: WARNING - NGINX Error Rate is 54%
[00:29:39] <botifico>	 [02MirahezeMagic] 07Universal-Omega closed pull request 03#479: Clarify wgRCLinkDays title and description - 13https://github.com/miraheze/MirahezeMagic/pull/479
[00:29:40] <botifico>	 [02miraheze/MirahezeMagic] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/MirahezeMagic/compare/604a729739fe...ed0ddbc6dc96
[00:29:42] <botifico>	 [02miraheze/MirahezeMagic] 07amire80 03ed0ddbc - Clarify wgRCLinkDays title and description (#479)
[00:29:46] <icinga-miraheze>	 PROBLEM - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is CRITICAL: CRITICAL - NGINX Error Rate is 60%
[00:32:22] <MirahezeLSBot>	 !log [@mwtask181] starting deploy of {'folders': '1.41/extensions/MirahezeMagic'} to all
[00:32:29] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[00:32:31] <MirahezeLSBot>	 !log [@mwtask181] finished deploy of {'folders': '1.41/extensions/MirahezeMagic'} to all - SUCCESS in 8s
[00:32:38] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[00:32:41] <botifico>	 [02MirahezeMagic] 07amire80 opened pull request 03#480: Fix wgVisualEditorEnableWikitext description - 13https://github.com/miraheze/MirahezeMagic/pull/480
[00:37:03] <MirahezeLSBot>	 !log [@mwtask171] starting deploy of {'folders': '1.41/extensions/MirahezeMagic'} to all
[00:37:10] <MirahezeLSBot>	 !log [@mwtask171] finished deploy of {'folders': '1.41/extensions/MirahezeMagic'} to all - SUCCESS in 6s
[00:37:11] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[00:37:29] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[00:42:48] <miraheze-github>	 miraheze/MirahezeMagic - Universal-Omega the build passed.
[00:46:47] <miraheze-github>	 miraheze/MirahezeMagic - amire80 the build passed.
[00:53:26] <MirahezeLSBot>	 !log [@test151] starting deploy of {'folders': '1.41/extensions/MirahezeMagic'} to test151
[00:53:27] <MirahezeLSBot>	 !log [@test151] finished deploy of {'folders': '1.41/extensions/MirahezeMagic'} to test151 - SUCCESS in 0s
[00:53:33] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[00:53:35] <MirahezeLSBot>	 !log [@test151] starting deploy of {'folders': '1.42/extensions/MirahezeMagic'} to test151
[00:53:36] <MirahezeLSBot>	 !log [@test151] finished deploy of {'folders': '1.42/extensions/MirahezeMagic'} to test151 - SUCCESS in 0s
[00:53:40] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[00:53:48] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[00:53:55] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[01:51:19] <botifico>	 [02MirahezeMagic] 07Universal-Omega closed pull request 03#480: Fix wgVisualEditorEnableWikitext description - 13https://github.com/miraheze/MirahezeMagic/pull/480
[01:51:21] <botifico>	 [02miraheze/MirahezeMagic] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/MirahezeMagic/compare/ed0ddbc6dc96...ab6fcaea8cf4
[01:51:22] <botifico>	 [02miraheze/MirahezeMagic] 07amire80 03ab6fcae - Fix wgVisualEditorEnableWikitext description (#480)
[01:53:19] <MirahezeLSBot>	 !log [@test151] starting deploy of {'folders': '1.41/extensions/MirahezeMagic'} to test151
[01:53:20] <MirahezeLSBot>	 !log [@test151] finished deploy of {'folders': '1.41/extensions/MirahezeMagic'} to test151 - SUCCESS in 0s
[01:53:27] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[01:53:29] <MirahezeLSBot>	 !log [@test151] starting deploy of {'folders': '1.42/extensions/MirahezeMagic'} to test151
[01:53:30] <MirahezeLSBot>	 !log [@test151] finished deploy of {'folders': '1.42/extensions/MirahezeMagic'} to test151 - SUCCESS in 0s
[01:53:34] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[01:53:41] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[01:53:48] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[02:02:28] <MirahezeLSBot>	 !log [@mwtask181] starting deploy of {'folders': '1.41/extensions/MirahezeMagic'} to all
[02:02:35] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[02:02:37] <MirahezeLSBot>	 !log [@mwtask181] finished deploy of {'folders': '1.41/extensions/MirahezeMagic'} to all - SUCCESS in 9s
[02:02:44] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[02:05:09] <miraheze-github>	 miraheze/MirahezeMagic - Universal-Omega the build passed.
[02:07:22] <MirahezeLSBot>	 !log [@mwtask171] starting deploy of {'folders': '1.41/extensions/MirahezeMagic'} to all
[02:07:30] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[02:07:30] <MirahezeLSBot>	 !log [@mwtask171] finished deploy of {'folders': '1.41/extensions/MirahezeMagic'} to all - SUCCESS in 7s
[02:07:38] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[03:01:01] <icinga-miraheze>	 PROBLEM - db162 Backups SQL on db162 is CRITICAL: FILE_AGE CRITICAL: /var/log/sql-backup.log is 1209647 seconds old and 1214 bytes
[04:53:46] <icinga-miraheze>	 PROBLEM - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is WARNING: WARNING - NGINX Error Rate is 59%
[05:07:14] <icinga-miraheze>	 PROBLEM - cloud15 Puppet on cloud15 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2]
[05:11:51] <icinga-miraheze>	 PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query line.pm. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL
[05:35:14] <icinga-miraheze>	 RECOVERY - cloud15 Puppet on cloud15 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[06:10:58] <icinga-miraheze>	 RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp36.wikitide.net - CNAME OK
[06:59:17] <icinga-miraheze>	 PROBLEM - mw182 Puppet on mw182 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[libboost-regex1.74.0-icu72]
[07:29:16] <icinga-miraheze>	 RECOVERY - mw182 Puppet on mw182 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[09:12:19] <icinga-miraheze>	 PROBLEM - db151 Backups SQL on db151 is WARNING: FILE_AGE WARNING: /var/log/sql-backup.log is 864173 seconds old and 111087 bytes
[11:42:50] <icinga-miraheze>	 PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mahdiruiz.line.pm All nameservers failed to answer the query.
[12:12:02] <botifico>	 [02miraheze/ImportDump] 07translatewiki pushed 031 commit to 03master [+0/-0/±7] 13https://github.com/miraheze/ImportDump/compare/43c174ae6f41...40b80dd6f9be
[12:12:05] <botifico>	 [02miraheze/ImportDump] 07translatewiki 0340b80dd - Localisation updates from https://translatewiki.net.
[12:12:08] <botifico>	 [02miraheze/IncidentReporting] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/IncidentReporting/compare/955b6f5f3c62...0a8ad443fe5f
[12:12:10] <botifico>	 [02miraheze/IncidentReporting] 07translatewiki 030a8ad44 - Localisation updates from https://translatewiki.net.
[12:12:12] <botifico>	 [02miraheze/landing] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/landing/compare/e2315dc385bb...77ecee64e731
[12:12:15] <botifico>	 [02miraheze/landing] 07translatewiki 0377ecee6 - Localisation updates from https://translatewiki.net.
[12:12:16] <botifico>	 [02miraheze/ManageWiki] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ManageWiki/compare/cc77da0b3810...fd42ad688356
[12:12:17] <botifico>	 [02miraheze/ManageWiki] 07translatewiki 03fd42ad6 - Localisation updates from https://translatewiki.net.
[12:12:19] <botifico>	 [02miraheze/MatomoAnalytics] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/MatomoAnalytics/compare/727ed539f02f...7032b175a768
[12:12:20] <botifico>	 [02miraheze/MatomoAnalytics] 07translatewiki 037032b17 - Localisation updates from https://translatewiki.net.
[12:12:22] <botifico>	 [02miraheze/DataDump] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/DataDump/compare/fbc3c778457d...8921e69327a9
[12:12:23] <icinga-miraheze>	 RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp37.wikitide.net - CNAME OK
[12:12:25] <botifico>	 [02miraheze/DataDump] 07translatewiki 038921e69 - Localisation updates from https://translatewiki.net.
[12:12:26] <botifico>	 [02miraheze/PDFEmbed] 07translatewiki pushed 031 commit to 03master [+1/-0/±0] 13https://github.com/miraheze/PDFEmbed/compare/c1c42d0e2fa4...9cc54cbbc859
[12:12:27] <botifico>	 [02miraheze/PDFEmbed] 07translatewiki 039cc54cb - Localisation updates from https://translatewiki.net.
[12:12:29] <botifico>	 [02miraheze/MirahezeMagic] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/MirahezeMagic/compare/ab6fcaea8cf4...cd18903db02a
[12:12:32] <botifico>	 [02miraheze/MirahezeMagic] 07translatewiki 03cd18903 - Localisation updates from https://translatewiki.net.
[12:12:35] <botifico>	 [02miraheze/SpriteSheet] 07translatewiki pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/SpriteSheet/compare/e4ae3316584a...3e11d46fc57c
[12:12:37] <botifico>	 [02miraheze/SpriteSheet] 07translatewiki 033e11d46 - Localisation updates from https://translatewiki.net.
[12:12:58] <miraheze-github>	 miraheze/landing - translatewiki the build passed.
[12:15:08] <miraheze-github>	 miraheze/SpriteSheet - translatewiki the build has errored.
[12:16:22] <miraheze-github>	 miraheze/PDFEmbed - translatewiki the build passed.
[12:16:38] <miraheze-github>	 miraheze/IncidentReporting - translatewiki the build passed.
[12:16:45] <miraheze-github_>	 miraheze/MatomoAnalytics - translatewiki the build passed.
[12:16:52] <miraheze-github_>	 miraheze/DataDump - translatewiki the build passed.
[12:17:00] <miraheze-github>	 miraheze/ManageWiki - translatewiki the build passed.
[12:20:09] <miraheze-github>	 miraheze/ImportDump - translatewiki the build passed.
[12:23:52] <MirahezeLSBot>	 !log [@test151] starting deploy of {'folders': '1.41/extensions/MirahezeMagic'} to test151
[12:23:53] <MirahezeLSBot>	 !log [@test151] finished deploy of {'folders': '1.41/extensions/MirahezeMagic'} to test151 - SUCCESS in 0s
[12:24:00] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[12:24:02] <MirahezeLSBot>	 !log [@test151] starting deploy of {'folders': '1.42/extensions/MirahezeMagic'} to test151
[12:24:03] <MirahezeLSBot>	 !log [@test151] finished deploy of {'folders': '1.42/extensions/MirahezeMagic'} to test151 - SUCCESS in 0s
[12:24:07] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[12:24:14] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[12:24:21] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[12:26:26] <miraheze-github>	 miraheze/MirahezeMagic - translatewiki the build passed.
[12:32:30] <MirahezeLSBot>	 !log [@mwtask181] starting deploy of {'folders': '1.41/extensions/MirahezeMagic'} to all
[12:32:38] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[12:32:40] <MirahezeLSBot>	 !log [@mwtask181] finished deploy of {'folders': '1.41/extensions/MirahezeMagic'} to all - SUCCESS in 9s
[12:32:47] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[12:37:06] <MirahezeLSBot>	 !log [@mwtask171] starting deploy of {'folders': '1.41/extensions/MirahezeMagic'} to all
[12:37:14] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[12:37:14] <MirahezeLSBot>	 !log [@mwtask171] finished deploy of {'folders': '1.41/extensions/MirahezeMagic'} to all - SUCCESS in 7s
[12:37:21] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[13:27:46] <icinga-miraheze>	 PROBLEM - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is CRITICAL: CRITICAL - NGINX Error Rate is 60%
[14:06:08] <icinga-miraheze>	 PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 176.67 ms
[14:14:20] <icinga-miraheze>	 RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 175.63 ms
[14:42:06] <icinga-miraheze>	 PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 179.95 ms
[14:44:08] <icinga-miraheze>	 RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 179.93 ms
[15:01:33] <icinga-miraheze>	 PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 180.47 ms
[15:07:42] <icinga-miraheze>	 RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 182.01 ms
[16:36:35] <icinga-miraheze>	 PROBLEM - en.religiononfire.mar.in.ua - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'en.religiononfire.mar.in.ua' expires in 15 day(s) (Sat 23 Mar 2024 04:27:38 PM GMT +0000).
[16:36:48] <botifico>	 [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/155412f2ab22...ff28990a01f2
[16:36:51] <botifico>	 [02miraheze/ssl] 07MirahezeSSLBot 03ff28990 - Bot: Update SSL cert for en.religiononfire.mar.in.ua
[16:42:35] <icinga-miraheze>	 PROBLEM - uk.religiononfire.mar.in.ua - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'uk.religiononfire.mar.in.ua' expires in 15 day(s) (Sat 23 Mar 2024 04:20:14 PM GMT +0000).
[16:42:48] <botifico>	 [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/ff28990a01f2...98be470db95c
[16:42:49] <botifico>	 [02miraheze/ssl] 07MirahezeSSLBot 0398be470 - Bot: Update SSL cert for uk.religiononfire.mar.in.ua
[17:06:10] <icinga-miraheze>	 RECOVERY - en.religiononfire.mar.in.ua - LetsEncrypt on sslhost is OK: OK - Certificate 'en.religiononfire.mar.in.ua' will expire on Wed 05 Jun 2024 03:36:41 PM GMT +0000.
[17:40:35] <icinga-miraheze>	 RECOVERY - uk.religiononfire.mar.in.ua - LetsEncrypt on sslhost is OK: OK - Certificate 'uk.religiononfire.mar.in.ua' will expire on Wed 05 Jun 2024 03:42:41 PM GMT +0000.
[20:25:35] <icinga-miraheze>	 [Grafana] !sre FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1
[20:26:05] <icinga-miraheze>	 PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query.
[20:56:07] <icinga-miraheze>	 PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns.ankh.fr.eu.org.', 'ns1.eu.org.', 'ns1.eriomem.net.'], 'CNAME': 'bouncingwiki.miraheze.org.'}
[22:32:52] <MirahezeLSBot>	 !log [agent@mwtask181] starting deploy of {'versions': '1.41', 'upgrade_extensions': 'RequestSSL'} to all
[22:32:54] <MirahezeLSBot>	 !log [agent@mwtask181] DEPLOY ABORTED: Non-Zero Exit Code in prep, see output.
[22:33:00] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:33:07] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:33:16] <MirahezeRelay>	 <pixldev, replying to MirahezeLSBot> oh dear
[22:33:50] <MirahezeLSBot>	 !log [agent@mwtask181] starting deploy of {'versions': '1.41', 'upgrade_extensions': 'RequestSSL'} to all
[22:33:58] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:34:02] <MirahezeLSBot>	 !log [agent@mwtask181] finished deploy of {'versions': '1.41', 'upgrade_extensions': 'RequestSSL'} to all - SUCCESS in 11s
[22:34:09] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log