[00:09:08] <icinga-miraheze>	 PROBLEM - wiki.yahyabd.xyz - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.yahyabd.xyz All nameservers failed to answer the query.
[00:13:23] <icinga-miraheze>	 RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp36.wikitide.net - CNAME OK
[00:32:25] <icinga-miraheze>	 [Grafana] !sre FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[01:09:09] <icinga-miraheze>	 PROBLEM - wiki.yahyabd.xyz - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.
[02:07:25] <icinga-miraheze>	 [Grafana] !sre RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[03:01:04] <icinga-miraheze>	 RECOVERY - db171 Backups SQL on db171 is OK: FILE_AGE OK: /var/log/sql-backup.log is 62 seconds old and 0 bytes
[04:04:23] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/97de29391c8e...51d7ae62daff
[04:04:26] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot 0351d7ae6 - Bot: Update SSL cert for wiki.altilunium.my.id
[05:41:16] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 171.07 ms
[05:43:19] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 171.56 ms
[05:44:24] <icinga-miraheze>	 PROBLEM - eros.locksteel.net - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'eros.locksteel.net' expires in 15 day(s) (Mon 20 May 2024 05:14:00 AM GMT +0000).
[05:44:35] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/51d7ae62daff...15d026c63137
[05:44:38] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot 0315d026c - Bot: Update SSL cert for eros.locksteel.net
[05:58:49] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 172.81 ms
[06:00:52] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 169.82 ms
[06:09:08] <icinga-miraheze>	 PROBLEM - wiki.yahyabd.xyz - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.yahyabd.xyz All nameservers failed to answer the query.
[06:16:25] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 171.49 ms
[06:18:28] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 168.49 ms
[06:26:22] <icinga-miraheze>	 PROBLEM - private.yahyabd.xyz - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - private.yahyabd.xyz All nameservers failed to answer the query.
[06:42:24] <icinga-miraheze>	 RECOVERY - eros.locksteel.net - LetsEncrypt on sslhost is OK: OK - Certificate 'eros.locksteel.net' will expire on Fri 02 Aug 2024 04:44:29 AM GMT +0000.
[06:43:22] <botifico>	 [02mw-config] 07princessellie closed pull request 03#5548: $wgTabberNeueParseTabName togglable - 13https://github.com/miraheze/mw-config/pull/5548
[07:09:09] <icinga-miraheze>	 PROBLEM - wiki.yahyabd.xyz - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.
[07:18:43] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 172.38 ms
[07:22:50] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 171.40 ms
[07:26:28] <icinga-miraheze>	 PROBLEM - private.yahyabd.xyz - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.404 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.
[07:56:33] <botifico>	 [02CreateWiki] 07redbluegreenhat synchronize pull request 03#518: Make the `visibility` property private - 13https://github.com/miraheze/CreateWiki/pull/518
[08:00:26] <miraheze-github>	 miraheze/CreateWiki - redbluegreenhat the build has errored.
[08:04:15] <botifico>	 [02CreateWiki] 07redbluegreenhat synchronize pull request 03#518: Make the `visibility` property private - 13https://github.com/miraheze/CreateWiki/pull/518
[08:07:21] <icinga-miraheze>	 PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.403 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.
[08:11:44] <botifico>	 [02CreateWiki] 07redbluegreenhat synchronize pull request 03#518: Make the `visibility` property private - 13https://github.com/miraheze/CreateWiki/pull/518
[08:12:14] <miraheze-github>	 miraheze/CreateWiki - redbluegreenhat the build passed.
[08:16:29] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 171.64 ms
[08:19:51] <miraheze-github>	 miraheze/CreateWiki - redbluegreenhat the build passed.
[08:21:39] <botifico>	 [02CreateWiki] 07redbluegreenhat closed pull request 03#518: Make the `visibility` property private - 13https://github.com/miraheze/CreateWiki/pull/518
[08:21:41] <botifico>	 [02miraheze/CreateWiki] 07redbluegreenhat pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/CreateWiki/compare/8885e7c35c65...66b4874c4d7b
[08:21:42] <botifico>	 [02miraheze/CreateWiki] 07redbluegreenhat 0366b4874 - Make the `visibility` property private (#518)
[08:22:39] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 172.22 ms
[08:26:26] <icinga-miraheze>	 PROBLEM - private.yahyabd.xyz - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - private.yahyabd.xyz All nameservers failed to answer the query.
[08:26:51] <botifico>	 [02CreateWiki] 07redbluegreenhat synchronize pull request 03#516: Add an API endpoint to post comments to wiki requests - 13https://github.com/miraheze/CreateWiki/pull/516
[08:29:41] <miraheze-github>	 miraheze/CreateWiki - redbluegreenhat the build passed.
[08:34:21] <botifico>	 [02CreateWiki] 07redbluegreenhat synchronize pull request 03#516: Add an API endpoint to post comments to wiki requests - 13https://github.com/miraheze/CreateWiki/pull/516
[08:34:27] <miraheze-github>	 miraheze/CreateWiki - redbluegreenhat the build passed.
[08:46:27] <miraheze-github>	 miraheze/CreateWiki - redbluegreenhat the build passed.
[08:52:49] <botifico>	 [02CreateWiki] 07redbluegreenhat opened pull request 03#523: No need to cast to int anymore - 13https://github.com/miraheze/CreateWiki/pull/523
[08:52:57] <botifico>	 [02CreateWiki] 07redbluegreenhat closed pull request 03#523: No need to cast to int anymore - 13https://github.com/miraheze/CreateWiki/pull/523
[08:52:59] <botifico>	 [02miraheze/CreateWiki] 07redbluegreenhat pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/66b4874c4d7b...5cffb94adc29
[08:53:01] <botifico>	 [02miraheze/CreateWiki] 07redbluegreenhat 035cffb94 - No need to cast to int anymore (#523)
[09:00:53] <miraheze-github>	 miraheze/CreateWiki - redbluegreenhat the build passed.
[09:01:02] <miraheze-github>	 miraheze/CreateWiki - redbluegreenhat the build passed.
[09:08:07] <botifico>	 [02CreateWiki] 07redbluegreenhat synchronize pull request 03#516: Add an API endpoint to post comments to wiki requests - 13https://github.com/miraheze/CreateWiki/pull/516
[09:09:05] <icinga-miraheze>	 PROBLEM - wiki.yahyabd.xyz - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.yahyabd.xyz All nameservers failed to answer the query.
[09:16:32] <miraheze-github>	 miraheze/CreateWiki - redbluegreenhat the build passed.
[09:16:35] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 172.21 ms
[09:18:38] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 172.56 ms
[09:22:50] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 169.66 ms
[09:26:27] <icinga-miraheze>	 PROBLEM - private.yahyabd.xyz - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.403 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.
[09:27:16] <botifico>	 [02CreateWiki] 07redbluegreenhat opened pull request 03#524: Return false for empty comments, true if comment was added - 13https://github.com/miraheze/CreateWiki/pull/524
[09:33:09] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 171.68 ms
[09:35:06] <icinga-miraheze>	 RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp37.wikitide.net - CNAME OK
[09:35:19] <miraheze-github>	 miraheze/CreateWiki - redbluegreenhat the build passed.
[09:39:09] <icinga-miraheze>	 PROBLEM - wiki.yahyabd.xyz - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.
[09:39:59] <botifico>	 [02CreateWiki] 07redbluegreenhat closed pull request 03#524: Return false for empty comments, true if comment was added - 13https://github.com/miraheze/CreateWiki/pull/524
[09:40:01] <botifico>	 [02miraheze/CreateWiki] 07redbluegreenhat pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/5cffb94adc29...00f1511a1761
[09:40:03] <botifico>	 [02miraheze/CreateWiki] 07redbluegreenhat 0300f1511 - Return false for empty comments, true if comment was added (#524)
[09:42:32] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 171.65 ms
[09:44:35] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 172.11 ms
[09:48:09] <miraheze-github>	 miraheze/CreateWiki - redbluegreenhat the build passed.
[09:56:24] <icinga-miraheze>	 PROBLEM - private.yahyabd.xyz - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - private.yahyabd.xyz All nameservers failed to answer the query.
[10:09:05] <icinga-miraheze>	 PROBLEM - wiki.yahyabd.xyz - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.yahyabd.xyz All nameservers failed to answer the query.
[11:32:56] <icinga-miraheze>	 PROBLEM - www.johanloopmans.nl - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.johanloopmans.nl' expires in 15 day(s) (Mon 20 May 2024 11:20:22 AM GMT +0000).
[11:33:07] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/15d026c63137...b58955137aab
[11:33:10] <botifico>	 [02miraheze/ssl] 07WikiTideSSLBot 03b589551 - Bot: Update SSL cert for www.johanloopmans.nl
[11:38:30] <icinga-miraheze>	 PROBLEM - johanloopmans.nl - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.johanloopmans.nl' expires in 15 day(s) (Mon 20 May 2024 11:20:22 AM GMT +0000).
[12:02:22] <icinga-miraheze>	 RECOVERY - www.johanloopmans.nl - LetsEncrypt on sslhost is OK: OK - Certificate 'www.johanloopmans.nl' will expire on Fri 02 Aug 2024 10:33:02 AM GMT +0000.
[12:02:51] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 170.08 ms
[12:04:54] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 171.35 ms
[12:07:30] <icinga-miraheze>	 RECOVERY - johanloopmans.nl - LetsEncrypt on sslhost is OK: OK - Certificate 'www.johanloopmans.nl' will expire on Fri 02 Aug 2024 10:33:02 AM GMT +0000.
[12:09:07] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 202.79 ms
[12:13:13] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 170.96 ms
[12:56:27] <icinga-miraheze>	 PROBLEM - private.yahyabd.xyz - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.404 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.
[13:35:44] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 171.58 ms
[13:43:58] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 169.93 ms
[13:56:26] <icinga-miraheze>	 PROBLEM - private.yahyabd.xyz - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - private.yahyabd.xyz All nameservers failed to answer the query.
[13:57:27] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 171.16 ms
[14:03:39] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 170.78 ms
[14:13:02] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 171.49 ms
[14:15:05] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 183.43 ms
[14:19:19] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 169.96 ms
[14:25:30] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 170.37 ms
[14:53:25] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 171.80 ms
[14:55:28] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 174.20 ms
[14:56:27] <icinga-miraheze>	 PROBLEM - private.yahyabd.xyz - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.
[15:08:38] <icinga-miraheze>	 PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.404 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.
[15:09:09] <icinga-miraheze>	 PROBLEM - wiki.yahyabd.xyz - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.404 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.
[15:15:03] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 180.95 ms
[15:25:26] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 171.78 ms
[15:26:24] <icinga-miraheze>	 PROBLEM - private.yahyabd.xyz - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - private.yahyabd.xyz All nameservers failed to answer the query.
[16:06:42] <icinga-miraheze>	 RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp37.wikitide.net - CNAME OK
[16:09:08] <icinga-miraheze>	 PROBLEM - wiki.yahyabd.xyz - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.yahyabd.xyz All nameservers failed to answer the query.
[16:13:03] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 172.46 ms
[16:15:06] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 210.59 ms
[16:39:04] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 168.36 ms
[16:41:07] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 171.04 ms
[16:45:19] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 169.78 ms
[16:49:26] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 168.69 ms
[17:05:07] <MirahezeLSBot>	 !log [alex@test151] Upgraded packages libc-devtools, libc6-dev, libc-dev-bin, libc6, libc-bin, less, libc-l10n, locales, libruby3.1, locales-all, and ruby3.1
[17:05:10] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[17:05:22] <MirahezeRelay>	 <bluemoon0332> !log rebooted test151
[17:05:26] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[17:06:20] <icinga-miraheze>	 RECOVERY - test151 APT on test151 is OK: APT OK: 27 packages available for upgrade (0 critical updates).
[17:16:16] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 169.98 ms
[17:22:27] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 174.68 ms
[17:37:39] <MirahezeLSBot>	 !log [alex@mw151] Upgraded packages libc-devtools, libc6-dev, libc-dev-bin, libc6, libc-bin, less, libc-l10n, locales, libruby3.1, locales-all, pdns-recursor, and ruby3.1
[17:37:43] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[17:37:56] <icinga-miraheze>	 RECOVERY - mw151 APT on mw151 is OK: APT OK: 27 packages available for upgrade (0 critical updates).
[17:38:04] <MirahezeRelay>	 <bluemoon0332> !log rebooted mw151
[17:38:08] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[17:41:40] <MirahezeLSBot>	 !log [alex@mw152] Upgraded packages libc-devtools, libc6-dev, libc-dev-bin, libc6, libc-bin, less, libc-l10n, locales, libruby3.1, locales-all, pdns-recursor, and ruby3.1
[17:41:44] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[17:42:01] <MirahezeRelay>	 <bluemoon0332> !log rebooted mw152
[17:42:05] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[17:43:05] <icinga-miraheze>	 RECOVERY - mw152 APT on mw152 is OK: APT OK: 27 packages available for upgrade (0 critical updates).
[17:43:54] <MirahezeLSBot>	 !log [alex@mw161] Upgraded packages libc-devtools, libc6-dev, libc-dev-bin, libc6, libc-bin, less, libc-l10n, locales, libruby3.1, locales-all, pdns-recursor, and ruby3.1
[17:43:57] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[17:44:27] <MirahezeRelay>	 <bluemoon0332> !log rebooted mw161
[17:44:30] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[17:44:45] <icinga-miraheze>	 RECOVERY - mw161 APT on mw161 is OK: APT OK: 27 packages available for upgrade (0 critical updates).
[17:46:23] <MirahezeLSBot>	 !log [alex@mw162] Upgraded packages libc-devtools, libc6-dev, libc-dev-bin, libc6, libc-bin, less, libc-l10n, locales, libruby3.1, locales-all, pdns-recursor, and ruby3.1
[17:46:26] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[17:46:53] <MirahezeRelay>	 <bluemoon0332> !log rebooted mw162
[17:47:08] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[17:49:05] <icinga-miraheze>	 RECOVERY - mw162 APT on mw162 is OK: APT OK: 27 packages available for upgrade (0 critical updates).
[17:50:06] <MirahezeLSBot>	 !log [alex@mw171] Upgraded packages libc-devtools, libc6-dev, libc-dev-bin, libc6, libc-bin, less, libc-l10n, locales, libruby3.1, locales-all, pdns-recursor, and ruby3.1
[17:50:10] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[17:50:36] <MirahezeRelay>	 <bluemoon0332> !log rebooted mw171
[17:50:40] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[17:52:50] <icinga-miraheze>	 RECOVERY - mw171 APT on mw171 is OK: APT OK: 27 packages available for upgrade (0 critical updates).
[17:53:34] <MirahezeLSBot>	 !log [alex@mw172] Upgraded packages libc-devtools, libc6-dev, libc-dev-bin, libc6, libc-bin, less, libc-l10n, locales, libruby3.1, locales-all, pdns-recursor, and ruby3.1
[17:53:37] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[17:54:02] <MirahezeRelay>	 <bluemoon0332> !log reboot mw172
[17:54:06] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[17:54:27] <icinga-miraheze>	 RECOVERY - mw172 APT on mw172 is OK: APT OK: 27 packages available for upgrade (0 critical updates).
[17:55:32] <MirahezeLSBot>	 !log [alex@mw181] Upgraded packages libc-devtools, libc6-dev, libc-dev-bin, libc6, libc-bin, less, libc-l10n, locales, libruby3.1, locales-all, pdns-recursor, and ruby3.1
[17:55:36] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[17:55:53] <MirahezeRelay>	 <bluemoon0332> !log rebooted mw181
[17:56:16] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[17:56:51] <icinga-miraheze>	 RECOVERY - mw181 APT on mw181 is OK: APT OK: 27 packages available for upgrade (0 critical updates).
[17:57:46] <MirahezeLSBot>	 !log [alex@mw182] Upgraded packages libc-devtools, libc6-dev, libc-dev-bin, libc6, libc-bin, less, libc-l10n, locales, libruby3.1, locales-all, pdns-recursor, and ruby3.1
[17:57:49] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[17:58:06] <MirahezeRelay>	 <bluemoon0332> !log rebooted mw182
[17:58:10] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[17:58:42] <MirahezeRelay>	 <pixldev> Huh
[17:59:41] <MirahezeLSBot>	 !log [alex@mwtask171] Upgraded packages libc-devtools, libc6-dev, libc-dev-bin, libc6, libc-bin, less, libc-l10n, locales, libruby3.1, locales-all, pdns-recursor, and ruby3.1
[17:59:45] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[17:59:56] <icinga-miraheze>	 RECOVERY - mwtask171 APT on mwtask171 is OK: APT OK: 1 packages available for upgrade (0 critical updates).
[17:59:59] <MirahezeRelay>	 <bluemoon0332> !log rebooted mwtask171
[18:00:03] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[18:00:17] <icinga-miraheze>	 RECOVERY - mw182 APT on mw182 is OK: APT OK: 27 packages available for upgrade (0 critical updates).
[18:01:15] <MirahezeLSBot>	 !log [alex@mwtask181] Upgraded packages libc-devtools, libc6-dev, libc-dev-bin, libc6, libc-bin, less, libc-l10n, locales, libruby3.1, locales-all, pdns-recursor, and ruby3.1
[18:01:19] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[18:01:34] <MirahezeRelay>	 <bluemoon0332> !log rebooted mwtask181
[18:01:38] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[18:03:02] <icinga-miraheze>	 PROBLEM - index.pdcommunity.ir - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.404 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.
[18:03:04] <icinga-miraheze>	 RECOVERY - mwtask181 APT on mwtask181 is OK: APT OK: 1 packages available for upgrade (0 critical updates).
[18:04:36] <MirahezeRelay>	 <cosmicalpha> OrangeStar and pixldev "Chatting about log booting issues" is what Discord notified me for lol
[18:06:02] <MirahezeRelay>	 <pixldev> Better then last time lol. I don’t have that feature since my phone don’t wanna update discord
[18:06:08] <MirahezeRelay>	 <pixldev> I’m still on the old yo
[18:06:21] <MirahezeRelay>	 <cosmicalpha> I have Discord Canary mobile lol
[18:06:29] <MirahezeRelay>	 <pixldev> ANDROID?
[18:06:30] <MirahezeRelay>	 <pixldev> Caps
[18:06:39] <MirahezeRelay>	 <cosmicalpha> hmm?
[18:07:01] <MirahezeRelay>	 <pixldev> You use android? Or does that exist on iOS as well
[18:08:33] <MirahezeRelay>	 <cosmicalpha, replying to pixldev> I use Android... Alpha is not available on iOS but Beta is I believe.
[18:18:50] <icinga-miraheze>	 PROBLEM - index.pdcommunity.ir - LetsEncrypt on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket
[18:26:27] <icinga-miraheze>	 PROBLEM - private.yahyabd.xyz - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.403 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.
[18:32:26] <icinga-miraheze>	 RECOVERY - index.pdcommunity.ir - reverse DNS on sslhost is OK: SSL OK - index.pdcommunity.ir reverse DNS resolves to cp37.wikitide.net - CNAME OK
[18:39:09] <icinga-miraheze>	 PROBLEM - wiki.yahyabd.xyz - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.
[18:39:54] <icinga-miraheze>	 PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.andreijiroh.uk.eu.org. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL
[18:44:09] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 168.54 ms
[18:46:12] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 172.99 ms
[18:47:22] <icinga-miraheze>	 RECOVERY - index.pdcommunity.ir - LetsEncrypt on sslhost is OK: OK - Certificate 'index.pdcommunity.ir' will expire on Sat 27 Jul 2024 05:29:12 PM GMT +0000.
[19:08:06] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 168.71 ms
[19:09:43] <icinga-miraheze>	 RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp36.wikitide.net - CNAME OK
[19:10:09] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 171.93 ms
[19:22:33] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 178.83 ms
[19:28:44] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 170.19 ms
[20:09:08] <icinga-miraheze>	 PROBLEM - wiki.yahyabd.xyz - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.yahyabd.xyz All nameservers failed to answer the query.
[20:09:40] <MirahezeLSBot>	 !log [universalomega@mwtask181] starting deploy of {'versions': ['1.41', '1.42'], 'show_tags': True, 'upgrade_extensions': 'DiscordNotifications'} to all
[20:09:44] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[20:09:59] <MirahezeLSBot>	 !log [universalomega@mwtask181] finished deploy of {'versions': ['1.41', '1.42'], 'show_tags': True, 'upgrade_extensions': 'DiscordNotifications'} to all - SUCCESS in 19s
[20:10:03] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[20:13:16] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 169.12 ms
[20:15:19] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 170.63 ms
[20:25:39] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 168.66 ms
[20:33:54] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 171.32 ms
[20:53:36] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 169.89 ms
[20:55:39] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 171.40 ms
[20:56:26] <icinga-miraheze>	 PROBLEM - private.yahyabd.xyz - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - private.yahyabd.xyz All nameservers failed to answer the query.
[21:07:21] <icinga-miraheze>	 PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.mahdiruiz.line.pm. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL
[21:26:34] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 171.45 ms
[21:28:37] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 170.35 ms
[21:32:25] <icinga-miraheze>	 [Grafana] !sre FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[21:36:21] <icinga-miraheze>	 RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp37.wikitide.net - CNAME OK
[22:07:42] <MirahezeLSBot>	 !log [@mwtask171] starting deploy of {'folders': '1.41/extensions/MirahezeMagic'} to all
[22:07:46] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[22:07:51] <MirahezeLSBot>	 !log [@mwtask171] finished deploy of {'folders': '1.41/extensions/MirahezeMagic'} to all - SUCCESS in 8s
[22:07:54] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[23:17:25] <icinga-miraheze>	 [Grafana] !sre RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[23:26:27] <icinga-miraheze>	 PROBLEM - private.yahyabd.xyz - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.
[23:56:26] <icinga-miraheze>	 PROBLEM - private.yahyabd.xyz - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - private.yahyabd.xyz All nameservers failed to answer the query.