[00:01:19] <icinga-miraheze>	 RECOVERY - cp27 HTTP 4xx/5xx ERROR Rate on cp27 is OK: OK - NGINX Error Rate is 0%
[00:05:03] <icinga-miraheze>	 RECOVERY - cp36 Disk Space on cp36 is OK: DISK OK - free space: / 21587MiB (24% inode=98%);
[00:05:15] <icinga-miraheze>	 RECOVERY - cp37 Disk Space on cp37 is OK: DISK OK - free space: / 21596MiB (24% inode=98%);
[00:05:19] <icinga-miraheze>	 PROBLEM - cp27 HTTP 4xx/5xx ERROR Rate on cp27 is CRITICAL: CRITICAL - NGINX Error Rate is 67%
[00:31:13] <botifico>	 [02ImportDump] 07Universal-Omega created branch 03renamespace - 13https://github.com/miraheze/ImportDump
[00:31:16] <botifico>	 [02miraheze/ImportDump] 07Universal-Omega created branch 03renamespace 13https://github.com/miraheze/ImportDump/commit/881a922e51b7545c73800a7b6fc32c8599fe1641
[00:33:34] <botifico>	 [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03renamespace [+0/-0/±22] 13https://github.com/miraheze/ImportDump/compare/881a922e51b7...47292fb62f8f
[00:33:37] <botifico>	 [02miraheze/ImportDump] 07Universal-Omega 0347292fb - Renamespace under Miraheze\RequestImport
[00:33:48] <botifico>	 [02ImportDump] 07Universal-Omega opened pull request 03#85: Renamespace under Miraheze\RequestImport - 13https://github.com/miraheze/ImportDump/pull/85
[00:36:17] <botifico>	 [02mw-config] 07Disgustedorite opened pull request 03#5498: Adding semantic drilldown - 13https://github.com/miraheze/mw-config/pull/5498
[00:42:08] <miraheze-github>	 miraheze/ImportDump - Universal-Omega the build passed.
[00:47:16] <botifico>	 [02mw-config] 07Universal-Omega reviewed pull request 03#5498 commit - 13https://github.com/miraheze/mw-config/pull/5498#discussion_r1519032956
[00:51:10] <botifico>	 [02mw-config] 07Disgustedorite synchronize pull request 03#5498: Adding semantic drilldown - 13https://github.com/miraheze/mw-config/pull/5498
[01:05:35] <icinga-miraheze>	 [Grafana] !sre FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1
[01:41:08] <botifico>	 [02mw-config] 07Disgustedorite commented on pull request 03#5498: Adding semantic drilldown - 13https://github.com/miraheze/mw-config/pull/5498#issuecomment-1987480629
[01:47:34] <miraheze-github>	 miraheze/mw-config - Disgustedorite the build passed.
[02:10:35] <icinga-miraheze>	 [Grafana] !sre RESOLVED: High Job Queue Backlog https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1
[02:32:23] <icinga-miraheze>	 PROBLEM - cloud16 Puppet on cloud16 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2]
[02:45:57] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 171.01 ms
[02:50:06] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 168.73 ms
[03:00:22] <icinga-miraheze>	 RECOVERY - cloud16 Puppet on cloud16 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures
[03:01:38] <icinga-miraheze>	 RECOVERY - mon181 Backups Grafana on mon181 is OK: FILE_AGE OK: /var/log/grafana-backup.log is 79 seconds old and 93 bytes
[03:33:14] <icinga-miraheze>	 PROBLEM - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is WARNING: WARNING - NGINX Error Rate is 58%
[04:34:45] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 169.65 ms
[04:36:49] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 168.80 ms
[04:41:01] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 171.04 ms
[05:05:54] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 168.20 ms
[05:38:40] <icinga-miraheze>	 PROBLEM - pwiki.arkcls.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for pwiki.arkcls.com could not be found
[05:53:14] <icinga-miraheze>	 PROBLEM - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is CRITICAL: CRITICAL - NGINX Error Rate is 60%
[06:11:31] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 169.40 ms
[06:13:34] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 169.54 ms
[06:29:14] <icinga-miraheze>	 PROBLEM - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is WARNING: WARNING - NGINX Error Rate is 59%
[06:38:36] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 170.76 ms
[06:40:40] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 168.59 ms
[06:53:14] <icinga-miraheze>	 PROBLEM - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is CRITICAL: CRITICAL - NGINX Error Rate is 60%
[07:08:21] <MirahezeLSBot>	 !log [reception@mwtask181] starting deploy of {'versions': '1.41', 'upgrade_extensions': 'ImportDump'} to all
[07:08:28] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[07:08:31] <botifico>	 [02mw-config] 07Reception123 closed pull request 03#5497: change ImportDump rights - 13https://github.com/miraheze/mw-config/pull/5497
[07:08:32] <botifico>	 [02miraheze/mw-config] 07Reception123 deleted branch 03Reception123-patch-1
[07:08:32] <MirahezeLSBot>	 !log [reception@mwtask181] finished deploy of {'versions': '1.41', 'upgrade_extensions': 'ImportDump'} to all - SUCCESS in 11s
[07:08:35] <botifico>	 [02mw-config] 07Reception123 deleted branch 03Reception123-patch-1 - 13https://github.com/miraheze/mw-config
[07:08:38] <botifico>	 [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/2cae7e1eb0fe...478eed30cbad
[07:08:39] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[07:08:40] <botifico>	 [02miraheze/mw-config] 07Reception123 03478eed3 - change ImportDump rights (#5497)
[07:08:44] <MirahezeLSBot>	 !log [reception@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all
[07:08:52] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[07:08:54] <MirahezeLSBot>	 !log [reception@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 9s
[07:09:01] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[07:09:14] <icinga-miraheze>	 PROBLEM - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is WARNING: WARNING - NGINX Error Rate is 59%
[07:09:19] <MirahezeLSBot>	 !log [reception@mwtask181] starting deploy of {'l10n': True, 'versions': '1.41', 'upgrade_extensions': 'ImportDump'} to all
[07:09:22] <miraheze-github>	 miraheze/mw-config - Reception123 the build passed.
[07:09:27] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[07:15:42] <MirahezeLSBot>	 !log [reception@mwtask181] finished deploy of {'l10n': True, 'versions': '1.41', 'upgrade_extensions': 'ImportDump'} to all - SUCCESS in 382s
[07:15:49] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[07:23:14] <icinga-miraheze>	 PROBLEM - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is CRITICAL: CRITICAL - NGINX Error Rate is 60%
[07:23:49] <MirahezeLSBot>	 !log [@test151] starting deploy of {'config': True} to test151
[07:23:49] <MirahezeLSBot>	 !log [@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s
[07:23:56] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[07:24:03] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[07:35:48] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 171.49 ms
[07:37:07] <MirahezeLSBot>	 !log [@mwtask171] starting deploy of {'config': True} to all
[07:37:14] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[07:37:15] <MirahezeLSBot>	 !log [@mwtask171] finished deploy of {'config': True} to all - SUCCESS in 8s
[07:37:23] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[07:39:56] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 170.62 ms
[07:58:38] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 167.90 ms
[08:00:42] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 169.11 ms
[08:34:46] <icinga-miraheze>	 PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 37%, RTA = 171.74 ms
[08:36:49] <icinga-miraheze>	 RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 170.08 ms
[09:10:03] <icinga-miraheze>	 PROBLEM - db151 Backups SQL on db151 is CRITICAL: FILE_AGE CRITICAL: /var/log/sql-backup.log is 1209637 seconds old and 111087 bytes
[09:36:41] <icinga-miraheze>	 RECOVERY - pwiki.arkcls.com - reverse DNS on sslhost is OK: SSL OK - pwiki.arkcls.com reverse DNS resolves to cp37.wikitide.net - CNAME OK
[09:37:35] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 169.15 ms
[09:41:42] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 174.80 ms
[09:53:10] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 184.32 ms
[10:01:28] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 166.71 ms
[10:24:41] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 169.33 ms
[10:26:44] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 166.73 ms
[10:33:05] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 165.44 ms
[10:35:08] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 171.92 ms
[10:47:42] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 167.88 ms
[10:49:45] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 167.70 ms
[11:07:29] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 169.37 ms
[11:09:32] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 169.74 ms
[11:42:57] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 100%
[11:45:00] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 168.50 ms
[11:57:35] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 168.84 ms
[12:03:49] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 169.87 ms
[12:12:24] <botifico>	 [02miraheze/CreateWiki] 07translatewiki pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/CreateWiki/compare/20e47488053a...31edd3cbbc2a
[12:12:27] <botifico>	 [02miraheze/CreateWiki] 07translatewiki 0331edd3c - Localisation updates from https://translatewiki.net.
[12:12:30] <botifico>	 [02miraheze/ErrorPages] 07translatewiki pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/ErrorPages/compare/a70d3f2b561f...5e4b620dcfa2
[12:12:31] <botifico>	 [02miraheze/ErrorPages] 07translatewiki 035e4b620 - Localisation updates from https://translatewiki.net.
[12:12:33] <botifico>	 [02miraheze/ImportDump] 07translatewiki pushed 031 commit to 03master [+0/-0/±3] 13https://github.com/miraheze/ImportDump/compare/881a922e51b7...75cf43fa172e
[12:12:34] <botifico>	 [02miraheze/ImportDump] 07translatewiki 0375cf43f - Localisation updates from https://translatewiki.net.
[12:12:36] <botifico>	 [02miraheze/IncidentReporting] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/IncidentReporting/compare/0a8ad443fe5f...92979c919b7f
[12:12:38] <botifico>	 [02miraheze/IncidentReporting] 07translatewiki 0392979c9 - Localisation updates from https://translatewiki.net.
[12:12:41] <botifico>	 [02miraheze/DataDump] 07translatewiki pushed 031 commit to 03master [+0/-0/±4] 13https://github.com/miraheze/DataDump/compare/8921e69327a9...4f36fa7af945
[12:12:43] <botifico>	 [02miraheze/DataDump] 07translatewiki 034f36fa7 - Localisation updates from https://translatewiki.net.
[12:12:45] <botifico>	 [02miraheze/landing] 07translatewiki pushed 031 commit to 03master [+0/-0/±3] 13https://github.com/miraheze/landing/compare/77ecee64e731...d11f511eb581
[12:12:46] <botifico>	 [02miraheze/landing] 07translatewiki 03d11f511 - Localisation updates from https://translatewiki.net.
[12:12:49] <botifico>	 [02miraheze/RemovePII] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/RemovePII/compare/29760e57d2e9...0eb282fc9635
[12:12:50] <botifico>	 [02miraheze/RemovePII] 07translatewiki 030eb282f - Localisation updates from https://translatewiki.net.
[12:12:52] <botifico>	 [02miraheze/WikiDiscover] 07translatewiki pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/WikiDiscover/compare/774366c1fa54...f11df1aceb85
[12:12:53] <botifico>	 [02miraheze/WikiDiscover] 07translatewiki 03f11df1a - Localisation updates from https://translatewiki.net.
[12:12:56] <botifico>	 [02miraheze/MirahezeMagic] 07translatewiki pushed 031 commit to 03master [+1/-0/±5] 13https://github.com/miraheze/MirahezeMagic/compare/020c024188e5...4407e070ca99
[12:12:58] <botifico>	 [02miraheze/MirahezeMagic] 07translatewiki 034407e07 - Localisation updates from https://translatewiki.net.
[12:13:01] <botifico>	 [02miraheze/SpriteSheet] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/SpriteSheet/compare/3e11d46fc57c...bf8dc0b5f17d
[12:13:04] <botifico>	 [02miraheze/SpriteSheet] 07translatewiki 03bf8dc0b - Localisation updates from https://translatewiki.net.
[12:13:15] <miraheze-github>	 miraheze/ErrorPages - translatewiki the build passed.
[12:13:48] <miraheze-github>	 miraheze/landing - translatewiki the build passed.
[12:15:37] <icinga-miraheze>	 PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 189.20 ms
[12:16:28] <miraheze-github>	 miraheze/SpriteSheet - translatewiki the build has errored.
[12:16:42] <miraheze-github>	 miraheze/IncidentReporting - translatewiki the build passed.
[12:16:58] <miraheze-github>	 miraheze/DataDump - translatewiki the build passed.
[12:17:14] <miraheze-github>	 miraheze/WikiDiscover - translatewiki the build passed.
[12:17:40] <icinga-miraheze>	 RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 188.53 ms
[12:20:03] <miraheze-github>	 miraheze/RemovePII - translatewiki the build passed.
[12:20:16] <miraheze-github>	 miraheze/ImportDump - translatewiki the build passed.
[12:20:36] <miraheze-github>	 miraheze/CreateWiki - translatewiki the build passed.
[12:23:19] <MirahezeLSBot>	 !log [@test151] starting deploy of {'folders': '1.41/extensions/MirahezeMagic'} to test151
[12:23:20] <MirahezeLSBot>	 !log [@test151] finished deploy of {'folders': '1.41/extensions/MirahezeMagic'} to test151 - SUCCESS in 0s
[12:23:27] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[12:23:29] <MirahezeLSBot>	 !log [@test151] starting deploy of {'folders': '1.42/extensions/MirahezeMagic'} to test151
[12:23:30] <MirahezeLSBot>	 !log [@test151] finished deploy of {'folders': '1.42/extensions/MirahezeMagic'} to test151 - SUCCESS in 0s
[12:23:34] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[12:23:41] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[12:23:49] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[12:26:39] <miraheze-github>	 miraheze/MirahezeMagic - translatewiki the build passed.
[12:32:57] <MirahezeLSBot>	 !log [@mwtask181] starting deploy of {'folders': '1.41/extensions/MirahezeMagic'} to all
[12:33:04] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[12:33:05] <MirahezeLSBot>	 !log [@mwtask181] finished deploy of {'folders': '1.41/extensions/MirahezeMagic'} to all - SUCCESS in 8s
[12:33:13] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[12:38:07] <MirahezeLSBot>	 !log [@mwtask171] starting deploy of {'folders': '1.41/extensions/MirahezeMagic'} to all
[12:38:15] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[12:38:15] <MirahezeLSBot>	 !log [@mwtask171] finished deploy of {'folders': '1.41/extensions/MirahezeMagic'} to all - SUCCESS in 8s
[12:38:23] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[12:42:17] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 201.41 ms
[12:48:38] <icinga-miraheze>	 PROBLEM - cp51 HTTPS on cp51 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Connection timed out after 10002 milliseconds
[12:49:38] <icinga-miraheze>	 PROBLEM - cp51 SSH on cp51 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[12:51:00] <icinga-miraheze>	 PROBLEM - cp51 Varnish Backends on cp51 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[12:51:02] <icinga-miraheze>	 PROBLEM - cp51 Nginx Backend for puppet181 on cp51 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[12:51:08] <icinga-miraheze>	 PROBLEM - cp51 Nginx Backend for reports171 on cp51 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds.
[12:51:13] <icinga-miraheze>	 PROBLEM - Host cp51 is DOWN: PING CRITICAL - Packet loss = 100%
[13:08:10] <icinga-miraheze>	 RECOVERY - Host cp51 is UP: PING OK - Packet loss = 16%, RTA = 200.12 ms
[13:08:26] <icinga-miraheze>	 RECOVERY - cp51 SSH on cp51 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u2 (protocol 2.0)
[13:08:29] <icinga-miraheze>	 PROBLEM - cp51 Puppet on cp51 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle.
[13:09:34] <icinga-miraheze>	 RECOVERY - cp51 Nginx Backend for puppet181 on cp51 is OK: TCP OK - 0.000 second response time on localhost port 8204
[13:09:35] <icinga-miraheze>	 RECOVERY - cp51 HTTPS on cp51 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3611 bytes in 4.339 second response time
[13:09:39] <icinga-miraheze>	 RECOVERY - cp51 Nginx Backend for reports171 on cp51 is OK: TCP OK - 0.002 second response time on localhost port 8205
[13:09:54] <icinga-miraheze>	 RECOVERY - cp51 Varnish Backends on cp51 is OK: All 19 backends are healthy
[13:12:05] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 180.61 ms
[13:16:18] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 185.88 ms
[13:20:26] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 193.37 ms
[13:30:10] <icinga-miraheze>	 RECOVERY - cp51 Puppet on cp51 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures
[13:41:21] <icinga-miraheze>	 PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 174.79 ms
[13:43:24] <icinga-miraheze>	 RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 176.14 ms
[13:47:36] <icinga-miraheze>	 PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 176.86 ms
[13:51:40] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 168.26 ms
[13:53:43] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 171.10 ms
[13:53:48] <icinga-miraheze>	 RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 182.75 ms
[14:34:17] <icinga-miraheze>	 PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 193.01 ms
[14:38:24] <icinga-miraheze>	 RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 192.53 ms
[14:50:56] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 170.50 ms
[14:55:03] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 165.20 ms
[15:02:19] <icinga-miraheze>	 PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 195.18 ms
[15:04:21] <icinga-miraheze>	 RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 188.45 ms
[15:10:38] <icinga-miraheze>	 PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 194.12 ms
[15:12:40] <icinga-miraheze>	 RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 189.00 ms
[15:27:14] <icinga-miraheze>	 PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 186.89 ms
[15:31:22] <icinga-miraheze>	 RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 186.82 ms
[15:35:38] <icinga-miraheze>	 PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 187.01 ms
[15:37:41] <icinga-miraheze>	 RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 189.40 ms
[16:10:02] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 170.96 ms
[16:10:37] <icinga-miraheze>	 PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 184.67 ms
[16:12:05] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 171.69 ms
[16:12:40] <icinga-miraheze>	 RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 185.49 ms
[16:18:21] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 179.19 ms
[16:24:33] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 170.24 ms
[16:30:50] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 168.63 ms
[16:36:36] <icinga-miraheze>	 PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 37%, RTA = 182.14 ms
[16:37:03] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 170.73 ms
[16:40:43] <icinga-miraheze>	 RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 170.69 ms
[16:50:35] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 167.19 ms
[16:54:42] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 168.75 ms
[17:01:21] <icinga-miraheze>	 PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 50%, RTA = 170.42 ms
[17:03:24] <icinga-miraheze>	 RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 162.89 ms
[17:10:45] <icinga-miraheze>	 PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 175.43 ms
[17:11:32] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 174.42 ms
[17:12:48] <icinga-miraheze>	 RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 175.24 ms
[17:15:41] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 169.89 ms
[17:23:46] <botifico>	 [02miraheze/MirahezeMagic] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±27] 13https://github.com/miraheze/MirahezeMagic/compare/4407e070ca99...8ffb5852e50e
[17:23:49] <botifico>	 [02miraheze/MirahezeMagic] 07Universal-Omega 038ffb585 - Update ImportDump messages
[17:24:01] <MirahezeLSBot>	 !log [@test151] starting deploy of {'folders': '1.41/extensions/MirahezeMagic'} to test151
[17:24:02] <MirahezeLSBot>	 !log [@test151] finished deploy of {'folders': '1.41/extensions/MirahezeMagic'} to test151 - SUCCESS in 0s
[17:24:08] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[17:24:10] <MirahezeLSBot>	 !log [@test151] starting deploy of {'folders': '1.42/extensions/MirahezeMagic'} to test151
[17:24:11] <MirahezeLSBot>	 !log [@test151] finished deploy of {'folders': '1.42/extensions/MirahezeMagic'} to test151 - SUCCESS in 0s
[17:24:16] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[17:24:23] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[17:24:30] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[17:26:51] <botifico>	 [02miraheze/MirahezeMagic] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/MirahezeMagic/compare/8ffb5852e50e...679cf9567036
[17:26:52] <botifico>	 [02miraheze/MirahezeMagic] 07Universal-Omega 03679cf95 - Remove 'dump' from miraheze-importdump-help-target
[17:27:41] <botifico>	 [02miraheze/MirahezeMagic] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/MirahezeMagic/compare/679cf9567036...3c2613ce69da
[17:27:44] <botifico>	 [02miraheze/MirahezeMagic] 07Universal-Omega 033c2613c - System administrators -> Site Reliability Engineering
[17:28:18] <botifico>	 [02miraheze/MirahezeMagic] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/MirahezeMagic/compare/3c2613ce69da...af8143e3c183
[17:28:20] <botifico>	 [02miraheze/MirahezeMagic] 07Universal-Omega 03af8143e - Fix grammer
[17:32:03] <MirahezeLSBot>	 !log [@mwtask181] starting deploy of {'folders': '1.41/extensions/MirahezeMagic'} to all
[17:32:10] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[17:32:13] <MirahezeLSBot>	 !log [@mwtask181] finished deploy of {'folders': '1.41/extensions/MirahezeMagic'} to all - SUCCESS in 9s
[17:32:20] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[17:36:17] <botifico>	 [02miraheze/MirahezeMagic] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±11] 13https://github.com/miraheze/MirahezeMagic/compare/af8143e3c183...e55417181135
[17:36:18] <botifico>	 [02miraheze/MirahezeMagic] 07Universal-Omega 03e554171 - ImportDump -> RequestRequest in requestimport-text
[17:49:12] <miraheze-github>	 miraheze/MirahezeMagic - Universal-Omega the build passed.
[17:53:51] <MirahezeLSBot>	 !log [@test151] starting deploy of {'folders': '1.41/extensions/MirahezeMagic'} to test151
[17:53:52] <MirahezeLSBot>	 !log [@test151] finished deploy of {'folders': '1.41/extensions/MirahezeMagic'} to test151 - SUCCESS in 0s
[17:53:59] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[17:54:00] <MirahezeLSBot>	 !log [@test151] starting deploy of {'folders': '1.42/extensions/MirahezeMagic'} to test151
[17:54:01] <MirahezeLSBot>	 !log [@test151] finished deploy of {'folders': '1.42/extensions/MirahezeMagic'} to test151 - SUCCESS in 0s
[17:54:07] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[17:54:14] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[17:54:21] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[17:56:24] <miraheze-github>	 miraheze/MirahezeMagic - Dringsim the build passed.
[18:02:52] <botifico>	 [02ImportDump] 07Universal-Omega closed pull request 03#77: Rename importdump_request* tables - 13https://github.com/miraheze/ImportDump/pull/77
[18:02:55] <botifico>	 [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±9] 13https://github.com/miraheze/ImportDump/compare/75cf43fa172e...e51bca697b9c
[18:02:56] <MirahezeLSBot>	 !log [@mwtask181] starting deploy of {'folders': '1.41/extensions/MirahezeMagic'} to all
[18:02:57] <botifico>	 [02miraheze/ImportDump] 07Universal-Omega 03e51bca6 - Rename importdump_request* tables (#77)
[18:02:58] <botifico>	 [02miraheze/ImportDump] 07Universal-Omega deleted branch 03rename-tables
[18:03:01] <botifico>	 [02ImportDump] 07Universal-Omega deleted branch 03rename-tables - 13https://github.com/miraheze/ImportDump
[18:03:03] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[18:03:05] <MirahezeLSBot>	 !log [@mwtask181] finished deploy of {'folders': '1.41/extensions/MirahezeMagic'} to all - SUCCESS in 9s
[18:03:13] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[18:06:48] <MirahezeRelay>	 <cosmicalpha> !log MariaDB [metawiki]> RENAME TABLE importdump_requests TO import_requests;
[18:06:56] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[18:07:12] <MirahezeRelay>	 <cosmicalpha> !log MariaDB [metawiki]> RENAME TABLE importdump_request_comments TO import_request_comments;
[18:07:19] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[18:08:16] <MirahezeLSBot>	 !log [universalomega@mwtask181] starting deploy of {'l10n': True, 'versions': '1.41'} to all
[18:08:24] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[18:10:32] <miraheze-github>	 miraheze/ImportDump - Universal-Omega the build passed.
[18:14:22] <MirahezeLSBot>	 !log [universalomega@mwtask181] finished deploy of {'l10n': True, 'versions': '1.41'} to all - SUCCESS in 365s
[18:14:29] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[18:16:05] <MirahezeRelay>	 <bluemoon0332> @cosmicalpha Can you do test151 too?
[18:16:20] <MirahezeRelay>	 <bluemoon0332> if not it seems I could do it myself
[18:19:07] <MirahezeRelay>	 <cosmicalpha, replying to bluemoon0332> You can if you want, I have to go for now.
[18:19:57] <MirahezeRelay>	 <bluemoon0332> I'll try to do it after I finish some things over on RequestSSL
[18:30:00] <botifico>	 [02RequestSSL] 07redbluegreenhat synchronize pull request 03#40: T11699: Create a job to automate CNAME checks - 13https://github.com/miraheze/RequestSSL/pull/40
[18:31:58] <botifico>	 [02RequestSSL] 07redbluegreenhat synchronize pull request 03#40: T11699: Create a job to automate CNAME checks - 13https://github.com/miraheze/RequestSSL/pull/40
[18:33:06] <miraheze-github>	 miraheze/RequestSSL - redbluegreenhat the build has errored.
[18:35:39] <miraheze-github>	 miraheze/RequestSSL - redbluegreenhat the build has errored.
[18:36:59] <MirahezeLSBot>	 !log [alex@test151] starting deploy of {'versions': '1.41', 'upgrade_extensions': 'ImportDump'} to test151
[18:37:06] <MirahezeLSBot>	 !log [alex@test151] finished deploy of {'versions': '1.41', 'upgrade_extensions': 'ImportDump'} to test151 - SUCCESS in 7s
[18:37:07] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[18:37:14] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[18:39:41] <MirahezeRelay>	 <bluemoon0332> !log [alex@test151] RENAME TABLE importdump_request_comments TO import_request_comments;
[18:39:48] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[18:40:41] <MirahezeRelay>	 <bluemoon0332> !log [alex@test151] RENAME TABLE importdump_requests TO import_requests;
[18:40:48] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[18:41:48] <MirahezeRelay>	 <bluemoon0332> https://meta.mirabeta.org/wiki/Special:RequestImportQueue/1 is working, perfect
[18:41:52] <MirahezeLSBot>	 !log [alex@test151] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/sql.php --wiki=metawikibeta (END - exit=2)
[18:41:59] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[18:41:59] <MirahezeRelay>	 <bluemoon0332> ez pz
[18:43:02] <MirahezeRelay>	 <bluemoon0332> oops, guess I should regenerate l10n
[18:43:36] <MirahezeRelay>	 <bluemoon0332> eh, that's a problem for future me
[19:44:49] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 170.05 ms
[19:46:52] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 166.14 ms
[19:50:35] <icinga-miraheze>	 [Grafana] !sre FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1
[20:30:35] <icinga-miraheze>	 [Grafana] !sre RESOLVED: High Job Queue Backlog https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1
[20:57:28] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 168.73 ms
[20:59:31] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 167.75 ms
[21:11:51] <icinga-miraheze>	 PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query uk.eu.org. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL
[21:41:10] <icinga-miraheze>	 RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp36.wikitide.net - CNAME OK
[22:28:04] <botifico>	 [02miraheze/ci-scripts] 07Universal-Omega pushed 031 commit to 03InsertQueryBuilder [+0/-0/±1] 13https://github.com/miraheze/ci-scripts/commit/874dc0d153f9
[22:28:06] <botifico>	 [02miraheze/ci-scripts] 07Universal-Omega 03874dc0d - Migrate to use InsertQueryBuilder
[22:28:07] <botifico>	 [02ci-scripts] 07Universal-Omega created branch 03InsertQueryBuilder - 13https://github.com/miraheze/ci-scripts
[22:28:11] <botifico>	 [02ci-scripts] 07Universal-Omega opened pull request 03#1: Migrate to use InsertQueryBuilder - 13https://github.com/miraheze/ci-scripts/pull/1
[22:28:45] <botifico>	 [02ci-scripts] 07Universal-Omega closed pull request 03#1: Migrate to use InsertQueryBuilder - 13https://github.com/miraheze/ci-scripts/pull/1
[22:28:46] <botifico>	 [02miraheze/ci-scripts] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ci-scripts/compare/6b349c9d0f4a...8eeeb5e04583
[22:28:48] <botifico>	 [02miraheze/ci-scripts] 07Universal-Omega 038eeeb5e - Migrate to use InsertQueryBuilder
[22:28:51] <botifico>	 [02miraheze/ci-scripts] 07Universal-Omega deleted branch 03InsertQueryBuilder
[22:28:53] <botifico>	 [02ci-scripts] 07Universal-Omega deleted branch 03InsertQueryBuilder - 13https://github.com/miraheze/ci-scripts
[22:39:02] <botifico>	 [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03InsertQueryBuilder [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/commit/c090562c8469
[22:39:05] <botifico>	 [02miraheze/CreateWiki] 07Universal-Omega 03c090562 - Migrate to use InsertQueryBuilder
[22:39:06] <botifico>	 [02CreateWiki] 07Universal-Omega created branch 03InsertQueryBuilder - 13https://github.com/miraheze/CreateWiki
[22:39:09] <botifico>	 [02CreateWiki] 07Universal-Omega opened pull request 03#485: Migrate to use InsertQueryBuilder - 13https://github.com/miraheze/CreateWiki/pull/485
[22:42:21] <miraheze-github>	 miraheze/CreateWiki - Universal-Omega the build has errored.
[22:43:42] <botifico>	 [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03InsertQueryBuilder [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/c090562c8469...52d7a4524c9e
[22:43:45] <botifico>	 [02miraheze/CreateWiki] 07Universal-Omega 0352d7a45 - Migrate
[22:43:47] <botifico>	 [02CreateWiki] 07Universal-Omega synchronize pull request 03#485: Migrate to use InsertQueryBuilder - 13https://github.com/miraheze/CreateWiki/pull/485
[22:46:21] <botifico>	 [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03InsertQueryBuilder [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/52d7a4524c9e...955e6b5b23c6
[22:46:24] <botifico>	 [02miraheze/CreateWiki] 07Universal-Omega 03955e6b5 - Migrate
[22:46:26] <botifico>	 [02CreateWiki] 07Universal-Omega synchronize pull request 03#485: Migrate to use InsertQueryBuilder - 13https://github.com/miraheze/CreateWiki/pull/485
[22:46:41] <miraheze-github>	 miraheze/CreateWiki - Universal-Omega the build has errored.
[22:47:02] <botifico>	 [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03InsertQueryBuilder [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/955e6b5b23c6...41c508f4da16
[22:47:05] <botifico>	 [02miraheze/CreateWiki] 07Universal-Omega 0341c508f - Migrate
[22:47:07] <botifico>	 [02CreateWiki] 07Universal-Omega synchronize pull request 03#485: Migrate to use InsertQueryBuilder - 13https://github.com/miraheze/CreateWiki/pull/485
[22:50:48] <miraheze-github>	 miraheze/CreateWiki - Universal-Omega the build has errored.
[23:00:36] <icinga-miraheze>	 [Grafana] !sre FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1
[23:20:36] <icinga-miraheze>	 [Grafana] !sre RESOLVED: High Job Queue Backlog https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1
[23:39:03] <icinga-miraheze>	 PROBLEM - cp36 Disk Space on cp36 is WARNING: DISK WARNING - free space: / 9702MiB (10% inode=98%);
[23:57:15] <icinga-miraheze>	 PROBLEM - cp37 Disk Space on cp37 is WARNING: DISK WARNING - free space: / 9706MiB (10% inode=98%);