[00:00:09] PROBLEM - cp27 HTTP 4xx/5xx ERROR Rate on cp27 is UNKNOWN: UNKNOWN - NGINX Error Rate is UNKNOWN [00:02:09] PROBLEM - cp27 HTTP 4xx/5xx ERROR Rate on cp27 is CRITICAL: CRITICAL - NGINX Error Rate is 100% [00:07:46] PROBLEM - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is WARNING: WARNING - NGINX Error Rate is 57% [00:09:46] PROBLEM - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is CRITICAL: CRITICAL - NGINX Error Rate is 63% [00:10:09] PROBLEM - cp27 HTTP 4xx/5xx ERROR Rate on cp27 is WARNING: WARNING - NGINX Error Rate is 57% [00:16:09] PROBLEM - cp27 HTTP 4xx/5xx ERROR Rate on cp27 is CRITICAL: CRITICAL - NGINX Error Rate is 61% [00:18:09] PROBLEM - cp27 HTTP 4xx/5xx ERROR Rate on cp27 is WARNING: WARNING - NGINX Error Rate is 58% [00:22:08] PROBLEM - kinnikupedia.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for kinnikupedia.org could not be found [00:22:09] PROBLEM - cp27 HTTP 4xx/5xx ERROR Rate on cp27 is CRITICAL: CRITICAL - NGINX Error Rate is 61% [00:25:35] [Grafana] !sre FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [00:43:29] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query uk.eu.org. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [01:12:56] RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp37.wikitide.net - CNAME OK [01:19:05] PROBLEM - mw172 Puppet on mw172 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[libboost-regex1.74.0-icu72] [01:20:35] [Grafana] !sre RESOLVED: High Job Queue Backlog https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [01:47:05] RECOVERY - mw172 Puppet on mw172 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:56:05] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [02:26:06] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['NS.ANKH.FR.eu.org.', 'NS1.eu.org.', 'NS1.ERIOMEM.NET.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [02:56:05] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [03:01:01] RECOVERY - db162 Backups SQL on db162 is OK: FILE_AGE OK: /var/log/sql-backup.log is 47 seconds old and 1218 bytes [03:01:36] PROBLEM - cloud.neptune.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'cloud.neptune.wiki' expires in 15 day(s) (Sun 24 Mar 2024 02:48:46 AM GMT +0000). [03:01:49] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/98be470db95c...8de053e002ec [03:01:51] [02miraheze/ssl] 07MirahezeSSLBot 038de053e - Bot: Update SSL cert for cloud.neptune.wiki [03:31:12] RECOVERY - cloud.neptune.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'cloud.neptune.wiki' will expire on Thu 06 Jun 2024 02:01:42 AM GMT +0000. [03:56:07] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.gab.pt.eu.org. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [04:56:05] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [05:26:06] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['NS.ANKH.FR.eu.org.', 'NS1.eu.org.', 'NS1.ERIOMEM.NET.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [07:56:05] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [08:26:06] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query pt.eu.org. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [08:31:46] PROBLEM - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is WARNING: WARNING - NGINX Error Rate is 50% [11:04:26] PROBLEM - removededm.com - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': None, 'CNAME': None} [12:26:05] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [12:49:46] PROBLEM - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is CRITICAL: CRITICAL - NGINX Error Rate is 60% [12:56:07] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns.ankh.fr.eu.org.', 'ns1.eu.org.', 'ns1.eriomem.net.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [12:59:27] PROBLEM - dutki.kro.kr - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'dutki.kro.kr' expires in 15 day(s) (Sun 24 Mar 2024 12:28:15 PM GMT +0000). [12:59:42] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/8de053e002ec...8661d298434f [12:59:44] [02miraheze/ssl] 07MirahezeSSLBot 038661d29 - Bot: Update SSL cert for dutki.kro.kr [13:26:45] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 167.24 ms [13:28:24] RECOVERY - dutki.kro.kr - LetsEncrypt on sslhost is OK: OK - Certificate 'dutki.kro.kr' will expire on Thu 06 Jun 2024 11:59:36 AM GMT +0000. [13:28:47] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 170.25 ms [13:32:26] RECOVERY - removededm.com - reverse DNS on sslhost is OK: SSL OK - removededm.com reverse DNS resolves to cp37.wikitide.net - NS RECORDS OK [13:43:44] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query line.pm. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [14:13:17] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mahdiruiz.line.pm All nameservers failed to answer the query. [14:18:40] PROBLEM - mwtask181 APT on mwtask181 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [14:20:40] PROBLEM - mwtask181 APT on mwtask181 is CRITICAL: APT CRITICAL: 43 packages available for upgrade (15 critical updates). [14:24:54] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 182.90 ms [14:26:56] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 178.74 ms [14:35:21] [02MirahezeMagic] 07Dringsim opened pull request 03#481: Remove 'forcesafemode' option - 13https://github.com/miraheze/MirahezeMagic/pull/481 [14:42:52] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.404 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out. [14:51:56] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 165.48 ms [14:56:01] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 169.81 ms [15:07:17] PROBLEM - elephants.ggreig.com - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out. [15:12:28] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp37.wikitide.net - CNAME OK [15:39:28] PROBLEM - emmytherobot.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'emmytherobot.wiki' expires in 15 day(s) (Sun 24 Mar 2024 03:17:14 PM GMT +0000). [15:39:40] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/8661d298434f...b16b85e4c853 [15:39:42] [02miraheze/ssl] 07MirahezeSSLBot 03b16b85e - Bot: Update SSL cert for emmytherobot.wiki [15:45:04] PROBLEM - wiki.ooer.ooo - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.ooer.ooo' expires in 15 day(s) (Sun 24 Mar 2024 03:34:42 PM GMT +0000). [15:45:15] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/b16b85e4c853...f7bd3dd5ffaf [15:45:18] [02miraheze/ssl] 07MirahezeSSLBot 03f7bd3dd - Bot: Update SSL cert for wiki.ooer.ooo [16:05:26] RECOVERY - elephants.ggreig.com - reverse DNS on sslhost is OK: SSL OK - elephants.ggreig.com reverse DNS resolves to cp37.wikitide.net - CNAME OK [16:39:23] RECOVERY - emmytherobot.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'emmytherobot.wiki' will expire on Thu 06 Jun 2024 02:39:34 PM GMT +0000. [16:44:11] RECOVERY - wiki.ooer.ooo - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.ooer.ooo' will expire on Thu 06 Jun 2024 02:45:10 PM GMT +0000. [17:31:50] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 158.30 ms [17:33:52] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 144.35 ms [20:15:56] PROBLEM - wiki.altilunium.my.id - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.altilunium.my.id. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [20:44:51] RECOVERY - wiki.altilunium.my.id - reverse DNS on sslhost is OK: SSL OK - wiki.altilunium.my.id reverse DNS resolves to cp36.wikitide.net - CNAME OK [21:01:02] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/f7bd3dd5ffaf...ca5d07dc3d7c [21:01:03] [02miraheze/ssl] 07MirahezeSSLBot 03ca5d07d - Bot: Add SSL cert for www.ngnl.wiki [21:03:53] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/ca5d07dc3d7c...a140356f1c2a [21:03:56] [02miraheze/ssl] 07MirahezeSSLBot 03a140356 - Bot: Add SSL cert for lpfteam.nl [21:13:56] [02dns] 07MacFan4000 opened pull request 03#495: Add phighting.wiki zone - 13https://github.com/miraheze/dns/pull/495 [21:44:26] [02dns] 07Universal-Omega closed pull request 03#495: Add phighting.wiki zone - 13https://github.com/miraheze/dns/pull/495 [21:44:28] [02miraheze/dns] 07Universal-Omega pushed 031 commit to 03master [+1/-0/±0] 13https://github.com/miraheze/dns/compare/d41fcca0d1fb...a81fe1fe3922 [21:44:30] [02miraheze/dns] 07MacFan4000 03a81fe1f - Add phighting.wiki zone (#495) [22:33:27] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/a140356f1c2a...13ce4d906502 [22:33:30] [02miraheze/ssl] 07MirahezeSSLBot 0313ce4d9 - Bot: Add SSL cert for phighting.wiki [22:50:23] [02miraheze/MirahezeMagic] 07OAuthority pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/MirahezeMagic/compare/cd18903db02a...020c024188e5 [22:50:24] [02miraheze/MirahezeMagic] 07OAuthority 03020c024 - remove defunct visual editor i18n [22:53:14] !log [@test151] starting deploy of {'folders': '1.41/extensions/MirahezeMagic'} to test151 [22:53:15] !log [@test151] finished deploy of {'folders': '1.41/extensions/MirahezeMagic'} to test151 - SUCCESS in 0s [22:53:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:53:23] !log [@test151] starting deploy of {'folders': '1.42/extensions/MirahezeMagic'} to test151 [22:53:24] !log [@test151] finished deploy of {'folders': '1.42/extensions/MirahezeMagic'} to test151 - SUCCESS in 0s [22:53:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:53:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:53:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:03:05] !log [@mwtask181] starting deploy of {'folders': '1.41/extensions/MirahezeMagic'} to all [23:03:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:03:15] !log [@mwtask181] finished deploy of {'folders': '1.41/extensions/MirahezeMagic'} to all - SUCCESS in 10s [23:03:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:04:12] miraheze/MirahezeMagic - OAuthority the build passed. [23:07:35] !log [@mwtask171] starting deploy of {'folders': '1.41/extensions/MirahezeMagic'} to all [23:07:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:07:43] !log [@mwtask171] finished deploy of {'folders': '1.41/extensions/MirahezeMagic'} to all - SUCCESS in 8s [23:07:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log