[00:04:32] <icinga-miraheze>	 RECOVERY - cp37 Disk Space on cp37 is OK: DISK OK - free space: / 21319MiB (24% inode=98%);
[00:05:04] <icinga-miraheze>	 RECOVERY - cp41 Disk Space on cp41 is OK: DISK OK - free space: / 19365MiB (20% inode=98%);
[00:05:08] <icinga-miraheze>	 RECOVERY - cp36 Disk Space on cp36 is OK: DISK OK - free space: / 21318MiB (24% inode=98%);
[01:09:39] <botifico>	 [02MirahezeMagic] 07waki285 opened pull request 03#489: Add electionadmin - 13https://github.com/miraheze/MirahezeMagic/pull/489
[01:15:27] <botifico>	 [02MirahezeMagic] 07waki285 synchronize pull request 03#489: Add electionadmin - 13https://github.com/miraheze/MirahezeMagic/pull/489
[03:26:28] <icinga-miraheze>	 PROBLEM - wiki.mahdiruiz.line.pm - LetsEncrypt on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket
[03:31:43] <icinga-miraheze>	 PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query line.pm. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL
[03:56:10] <icinga-miraheze>	 RECOVERY - wiki.mahdiruiz.line.pm - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.mahdiruiz.line.pm' will expire on Fri 14 Jun 2024 04:28:50 PM GMT +0000.
[05:01:40] <icinga-miraheze>	 PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mahdiruiz.line.pm All nameservers failed to answer the query.
[05:04:52] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 170.32 ms
[05:06:50] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 170.40 ms
[05:19:58] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 37%, RTA = 171.55 ms
[05:23:56] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 170.77 ms
[05:31:35] <icinga-miraheze>	 RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp36.wikitide.net - CNAME OK
[06:27:01] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 168.88 ms
[06:28:59] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 170.66 ms
[07:03:02] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 178.66 ms
[07:07:00] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 168.47 ms
[07:21:47] <icinga-miraheze>	 PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query uk.eu.org. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL
[07:50:34] <icinga-miraheze>	 RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp36.wikitide.net - CNAME OK
[08:04:41] <icinga-miraheze>	 PROBLEM - db181 Backups SQL on db181 is WARNING: FILE_AGE WARNING: /var/log/sql-backup.log is 864233 seconds old and 107143 bytes
[08:48:06] <icinga-miraheze>	 PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query pt.eu.org. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL
[09:06:28] <icinga-miraheze>	 PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query line.pm. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL
[09:10:27] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 174.95 ms
[09:14:25] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 193.41 ms
[09:17:58] <icinga-miraheze>	 RECOVERY - wiki.gab.pt.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.gab.pt.eu.org reverse DNS resolves to cp36.wikitide.net - CNAME OK
[09:34:38] <icinga-miraheze>	 PROBLEM - db161 Backups SQL on db161 is WARNING: FILE_AGE WARNING: /var/log/sql-backup.log is 864235 seconds old and 99417 bytes
[09:39:28] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 170.69 ms
[09:43:25] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 170.45 ms
[09:52:44] <MirahezeLSBot>	 !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/runJobs.php --wiki=chizunetserieswiki (START)
[09:52:45] <MirahezeLSBot>	 !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/runJobs.php --wiki=chizunetserieswiki (END - exit=0)
[09:52:53] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[09:53:01] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[10:03:17] <icinga-miraheze>	 PROBLEM - alternatewiki.tombricks.com - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.402 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.
[10:03:22] <icinga-miraheze>	 PROBLEM - fc.songcontests.eu - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - fc.songcontests.eu All nameservers failed to answer the query.
[10:03:46] <icinga-miraheze>	 PROBLEM - wiki.col6.de - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.col6.de All nameservers failed to answer the query.
[10:32:23] <icinga-miraheze>	 RECOVERY - wiki.col6.de - reverse DNS on sslhost is OK: SSL OK - wiki.col6.de reverse DNS resolves to cp37.wikitide.net - CNAME OK
[10:32:52] <icinga-miraheze>	 RECOVERY - alternatewiki.tombricks.com - reverse DNS on sslhost is OK: SSL OK - alternatewiki.tombricks.com reverse DNS resolves to cp36.wikitide.net - CNAME OK
[10:33:03] <icinga-miraheze>	 RECOVERY - fc.songcontests.eu - reverse DNS on sslhost is OK: SSL OK - fc.songcontests.eu reverse DNS resolves to cp37.wikitide.net - CNAME OK
[10:33:30] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 174.68 ms
[10:35:28] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 170.77 ms
[10:36:31] <icinga-miraheze>	 RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp37.wikitide.net - CNAME OK
[11:08:35] <icinga-miraheze>	 PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query line.pm. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL
[11:26:23] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 170.57 ms
[11:28:22] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 175.02 ms
[11:38:31] <icinga-miraheze>	 RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp36.wikitide.net - CNAME OK
[12:01:03] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 171.59 ms
[12:05:01] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 168.66 ms
[12:13:21] <botifico>	 [02miraheze/CreateWiki] 07translatewiki pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/CreateWiki/compare/632db0357ecd...3083633da9a0
[12:13:24] <botifico>	 [02miraheze/CreateWiki] 07translatewiki 033083633 - Localisation updates from https://translatewiki.net.
[12:13:25] <botifico>	 [02miraheze/ImportDump] 07translatewiki pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/ImportDump/compare/53553bda3d5d...7bf21da78d8a
[12:13:28] <botifico>	 [02miraheze/ImportDump] 07translatewiki 037bf21da - Localisation updates from https://translatewiki.net.
[12:13:30] <botifico>	 [02miraheze/landing] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/landing/compare/49274190324a...0a8c9b2812e5
[12:13:31] <botifico>	 [02miraheze/landing] 07translatewiki 030a8c9b2 - Localisation updates from https://translatewiki.net.
[12:13:32] <botifico>	 [02miraheze/RequestSSL] 07translatewiki pushed 031 commit to 03master [+1/-0/±2] 13https://github.com/miraheze/RequestSSL/compare/d751f3ac07fe...77c22683a9a8
[12:13:34] <botifico>	 [02miraheze/RequestSSL] 07translatewiki 0377c2268 - Localisation updates from https://translatewiki.net.
[12:13:37] <botifico>	 [02miraheze/RottenLinks] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/RottenLinks/compare/df13f2d0a27d...56b882420168
[12:13:40] <botifico>	 [02miraheze/RottenLinks] 07translatewiki 0356b8824 - Localisation updates from https://translatewiki.net.
[12:13:43] <botifico>	 [02miraheze/MirahezeMagic] 07translatewiki pushed 031 commit to 03master [+0/-0/±6] 13https://github.com/miraheze/MirahezeMagic/compare/74d61e52fe85...0e549248d6ab
[12:13:46] <botifico>	 [02miraheze/MirahezeMagic] 07translatewiki 030e54924 - Localisation updates from https://translatewiki.net.
[12:13:47] <botifico>	 [02miraheze/SpriteSheet] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/SpriteSheet/compare/8ea98b2eb4d7...ec4422e8c098
[12:13:48] <botifico>	 [02miraheze/SpriteSheet] 07translatewiki 03ec4422e - Localisation updates from https://translatewiki.net.
[12:13:50] <botifico>	 [02miraheze/WikiDiscover] 07translatewiki pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/WikiDiscover/compare/bc8cda0e6ecc...0efc32e762a8
[12:13:53] <botifico>	 [02miraheze/WikiDiscover] 07translatewiki 030efc32e - Localisation updates from https://translatewiki.net.
[12:14:14] <miraheze-github>	 miraheze/landing - translatewiki the build passed.
[12:17:10] <miraheze-github>	 miraheze/SpriteSheet - translatewiki the build has errored.
[12:17:50] <miraheze-github>	 miraheze/RottenLinks - translatewiki the build passed.
[12:17:58] <miraheze-github_>	 miraheze/WikiDiscover - translatewiki the build passed.
[12:18:21] <miraheze-github>	 miraheze/MirahezeMagic - translatewiki the build has errored.
[12:21:29] <miraheze-github>	 miraheze/RequestSSL - translatewiki the build passed.
[12:21:38] <miraheze-github>	 miraheze/ImportDump - translatewiki the build passed.
[12:22:23] <miraheze-github>	 miraheze/CreateWiki - translatewiki the build passed.
[12:24:03] <MirahezeLSBot>	 !log [@test151] starting deploy of {'folders': '1.41/extensions/MirahezeMagic'} to test151
[12:24:04] <MirahezeLSBot>	 !log [@test151] finished deploy of {'folders': '1.41/extensions/MirahezeMagic'} to test151 - SUCCESS in 0s
[12:24:12] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[12:24:13] <MirahezeLSBot>	 !log [@test151] starting deploy of {'folders': '1.42/extensions/MirahezeMagic'} to test151
[12:24:14] <MirahezeLSBot>	 !log [@test151] finished deploy of {'folders': '1.42/extensions/MirahezeMagic'} to test151 - SUCCESS in 0s
[12:24:20] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[12:24:24] <MirahezeLSBot>	 !log [@test151] starting deploy of {'folders': '1.43/extensions/MirahezeMagic'} to test151
[12:24:25] <MirahezeLSBot>	 !log [@test151] finished deploy of {'folders': '1.43/extensions/MirahezeMagic'} to test151 - SUCCESS in 0s
[12:24:28] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[12:24:36] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[12:24:44] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[12:24:52] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[12:31:38] <icinga-miraheze>	 PROBLEM - wiki.buryland.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.buryland.net All nameservers failed to answer the query.
[12:32:37] <MirahezeLSBot>	 !log [@mwtask181] starting deploy of {'folders': '1.41/extensions/MirahezeMagic'} to all
[12:32:45] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[12:32:46] <MirahezeLSBot>	 !log [@mwtask181] finished deploy of {'folders': '1.41/extensions/MirahezeMagic'} to all - SUCCESS in 9s
[12:32:55] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[12:32:55] <MirahezeLSBot>	 !log [@mwtask181] starting deploy of {'folders': '1.42/extensions/MirahezeMagic'} to all
[12:33:04] <MirahezeLSBot>	 !log [@mwtask181] finished deploy of {'folders': '1.42/extensions/MirahezeMagic'} to all - SUCCESS in 9s
[12:33:04] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[12:33:12] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[12:37:34] <MirahezeLSBot>	 !log [@mwtask171] starting deploy of {'folders': '1.41/extensions/MirahezeMagic'} to all
[12:37:43] <MirahezeLSBot>	 !log [@mwtask171] finished deploy of {'folders': '1.41/extensions/MirahezeMagic'} to all - SUCCESS in 8s
[12:37:43] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[12:37:51] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[12:37:51] <MirahezeLSBot>	 !log [@mwtask171] starting deploy of {'folders': '1.42/extensions/MirahezeMagic'} to all
[12:38:00] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[12:38:00] <MirahezeLSBot>	 !log [@mwtask171] finished deploy of {'folders': '1.42/extensions/MirahezeMagic'} to all - SUCCESS in 8s
[12:38:08] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[13:00:55] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 37%, RTA = 171.01 ms
[13:01:12] <icinga-miraheze>	 RECOVERY - wiki.buryland.net - reverse DNS on sslhost is OK: SSL OK - wiki.buryland.net reverse DNS resolves to cp37.wikitide.net - CNAME OK
[13:02:53] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 171.64 ms
[13:27:04] <icinga-miraheze>	 PROBLEM - ns2 NTP time on ns2 is CRITICAL: NTP CRITICAL: Offset 1.060155153 secs
[13:39:40] <icinga-miraheze>	 PROBLEM - ping on changeprop151 is CRITICAL: PING CRITICAL - Packet loss = 0%, RTA = 262.31 ms
[13:41:41] <icinga-miraheze>	 RECOVERY - ping on changeprop151 is OK: PING OK - Packet loss = 0%, RTA = 0.29 ms
[13:47:05] <icinga-miraheze>	 RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset -0.002638220787 secs
[13:52:56] <MirahezeRelay>	 <bluemoon0332> @reception123 FYI, runJobs is broken for the same reason as showJobs, MediaWiki itself doesn't even know there are pending jobs anymore.
[14:10:28] <icinga-miraheze>	 PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.mahdiruiz.line.pm. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL
[14:11:06] <MirahezeRelay>	 <reception123, replying to bluemoon0332> oh huh? Kafka sure is making things difficult
[14:40:25] <icinga-miraheze>	 PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mahdiruiz.line.pm All nameservers failed to answer the query.
[14:41:54] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 171.12 ms
[14:49:19] <icinga-miraheze>	 PROBLEM - mw171 Puppet on mw171 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_femiwiki-deploy-1.41]
[14:49:52] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 174.37 ms
[14:53:56] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 169.88 ms
[14:57:53] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 167.01 ms
[15:10:23] <icinga-miraheze>	 PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query line.pm. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL
[15:15:19] <icinga-miraheze>	 RECOVERY - mw171 Puppet on mw171 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures
[15:18:53] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 178.42 ms
[15:20:51] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 173.85 ms
[15:40:22] <icinga-miraheze>	 RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp36.wikitide.net - CNAME OK
[15:43:40] <MirahezeLSBot>	 !log [agent@mwtask181] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/deleted.json /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/sql.php bv2022_tables.sql (START)
[15:43:48] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[16:05:01] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 164.70 ms
[16:06:59] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 163.48 ms
[16:28:59] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 174.37 ms
[16:32:56] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 182.86 ms
[16:37:46] <icinga-miraheze>	 PROBLEM - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is WARNING: WARNING - NGINX Error Rate is 58%
[16:44:18] <botifico>	 [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03database-admins [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/fcf534e7f774
[16:44:20] <botifico>	 [02miraheze/puppet] 07Universal-Omega 03fcf534e - Add a database-admins group
[16:44:22] <botifico>	 [02puppet] 07Universal-Omega created branch 03database-admins - 13https://github.com/miraheze/puppet
[16:44:24] <botifico>	 [02puppet] 07Universal-Omega opened pull request 03#3834: Add a database-admins group - 13https://github.com/miraheze/puppet/pull/3834
[16:45:16] <botifico>	 [02puppet] 07Universal-Omega synchronize pull request 03#3834: Add a database-admins group - 13https://github.com/miraheze/puppet/pull/3834
[16:45:17] <botifico>	 [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03database-admins [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/fcf534e7f774...92d12126cc65
[16:45:20] <botifico>	 [02miraheze/puppet] 07Universal-Omega 0392d1212 - Allow status
[16:55:48] <MirahezeLSBot>	 !log [agent@mwtask181] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/deleted.json /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/sql.php bv2022_tables.sql (END - exit=0)
[16:55:57] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[17:07:41] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 168.26 ms
[17:09:39] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 171.61 ms
[17:15:04] <icinga-miraheze>	 PROBLEM - cp41 Disk Space on cp41 is WARNING: DISK WARNING - free space: / 10232MiB (10% inode=98%);
[18:09:32] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 37%, RTA = 168.48 ms
[18:11:31] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 167.60 ms
[18:50:31] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 170.04 ms
[18:52:29] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 168.24 ms
[19:03:15] <Mannys>	 Trying to add ref tags for text *within* the article. How can I do so?
[19:04:23] <RhinosF1>	 Mannys: please use #miraheze for general help
[19:04:23] <MirahezeRelay>	 <pixldev, replying to Mannys> What do you mean? Also support stuff should go in #miraheze probably
[19:07:05] <Mannys>	 Oh okay, thank you
[19:21:46] <icinga-miraheze>	 PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.andreijiroh.uk.eu.org All nameservers failed to answer the query.
[19:30:47] <icinga-miraheze>	 PROBLEM - cloud18 APT on cloud18 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected.
[19:32:48] <icinga-miraheze>	 PROBLEM - cloud18 APT on cloud18 is CRITICAL: APT CRITICAL: 74 packages available for upgrade (14 critical updates).
[20:19:46] <icinga-miraheze>	 RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp37.wikitide.net - CNAME OK
[20:45:15] <icinga-miraheze>	 PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.mahdiruiz.line.pm. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL
[21:45:15] <icinga-miraheze>	 RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp36.wikitide.net - CNAME OK
[21:46:32] <icinga-miraheze>	 PROBLEM - cp37 Disk Space on cp37 is WARNING: DISK WARNING - free space: / 9698MiB (10% inode=98%);
[21:49:08] <icinga-miraheze>	 PROBLEM - cp36 Disk Space on cp36 is WARNING: DISK WARNING - free space: / 9716MiB (10% inode=98%);
[22:17:15] <icinga-miraheze>	 PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.mahdiruiz.line.pm. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL
[22:47:10] <icinga-miraheze>	 RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp36.wikitide.net - CNAME OK
[23:19:46] <icinga-miraheze>	 PROBLEM - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is CRITICAL: CRITICAL - NGINX Error Rate is 60%