[00:01:27] PROBLEM - cp27 HTTP 4xx/5xx ERROR Rate on cp27 is UNKNOWN: UNKNOWN - NGINX Error Rate is UNKNOWN [00:03:27] PROBLEM - cp27 HTTP 4xx/5xx ERROR Rate on cp27 is CRITICAL: CRITICAL - NGINX Error Rate is 100% [02:04:58] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query line.pm. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [03:02:16] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp36.wikitide.net - CNAME OK [03:59:06] !log [agent@mwtask181] starting deploy of {'extension_list': True, 'versions': '1.41'} to all [03:59:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:59:17] !log [agent@mwtask181] finished deploy of {'extension_list': True, 'versions': '1.41'} to all - SUCCESS in 10s [03:59:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:53:28] !log [@test151] starting deploy of {'folders': '1.41/extensions/MirahezeMagic'} to test151 [04:53:29] !log [@test151] finished deploy of {'folders': '1.41/extensions/MirahezeMagic'} to test151 - SUCCESS in 0s [04:53:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:53:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:43:20] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 150.85 ms [06:45:21] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 153.73 ms [06:57:33] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 150.46 ms [06:59:34] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 166.27 ms [07:12:44] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 174.74 ms [07:14:45] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 173.07 ms [07:41:06] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 168.48 ms [07:43:06] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 153.79 ms [07:59:06] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/invalidateUserSessions.php --wiki=telepediawiki Fersteax_Pasique (END - exit=256) [07:59:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:56:23] PROBLEM - test151 Puppet on test151 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[libboost-regex1.74.0-icu72] [09:24:22] RECOVERY - test151 Puppet on test151 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [10:35:35] [Grafana] !sre FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [10:50:35] [Grafana] !sre RESOLVED: High Job Queue Backlog https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [11:21:26] PROBLEM - ping6 on cp51 is WARNING: PING WARNING - Packet loss = 0%, RTA = 333.31 ms [11:21:49] PROBLEM - ping6 on cp41 is WARNING: PING WARNING - Packet loss = 0%, RTA = 307.14 ms [11:23:30] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 167.54 ms [11:23:50] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 150.23 ms [12:07:56] [02RequestSSL] 07redbluegreenhat synchronize pull request 03#40: T11699: Create a job to automate CNAME and Reverse DNS checks - 13https://github.com/miraheze/RequestSSL/pull/40 [12:09:43] [02RequestSSL] 07redbluegreenhat synchronize pull request 03#40: T11699: Create a job to automate CNAME and Reverse DNS checks - 13https://github.com/miraheze/RequestSSL/pull/40 [12:10:51] miraheze/RequestSSL - redbluegreenhat the build has errored. [12:12:44] miraheze/RequestSSL - redbluegreenhat the build has errored. [13:00:39] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 184.26 ms [13:02:40] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 178.52 ms [13:07:19] [02miraheze/dns] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/a81fe1fe3922...52ae5956b54c [13:07:20] [02miraheze/dns] 07AgentIsai 0352ae595 - Add status.wikitide.net [13:13:00] [02miraheze/dns] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/52ae5956b54c...1914cc3906b4 [13:13:03] [02miraheze/dns] 07AgentIsai 031914cc3 - Add trailing point [13:13:15] PROBLEM - ns1 GDNSD Datacenters on ns1 is UNKNOWN: NRPE: Unable to read output [13:15:15] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 159.255.146.183/cpweb, 2001:470:25:715::2/cpweb, 63.141.240.6/cpweb2 [13:24:55] [02RequestSSL] 07redbluegreenhat synchronize pull request 03#40: T11699: Create a job to automate CNAME and Reverse DNS checks - 13https://github.com/miraheze/RequestSSL/pull/40 [13:26:22] [02RequestSSL] 07redbluegreenhat synchronize pull request 03#40: T11699: Create a job to automate CNAME and Reverse DNS checks - 13https://github.com/miraheze/RequestSSL/pull/40 [13:28:34] miraheze/RequestSSL - redbluegreenhat the build has errored. [13:29:32] [02RequestSSL] 07redbluegreenhat edited pull request 03#40: T11699: Create a job to automate CNAME checks - 13https://github.com/miraheze/RequestSSL/pull/40 [13:29:36] miraheze/RequestSSL - redbluegreenhat the build has errored. [13:29:46] [02RequestSSL] 07redbluegreenhat edited pull request 03#40: T11699: Create a job to automate CNAME checks - 13https://github.com/miraheze/RequestSSL/pull/40 [13:30:50] [02RequestSSL] 07redbluegreenhat synchronize pull request 03#40: T11699: Create a job to automate CNAME checks - 13https://github.com/miraheze/RequestSSL/pull/40 [13:33:49] miraheze/RequestSSL - redbluegreenhat the build has errored. [13:34:21] [02RequestSSL] 07redbluegreenhat commented on pull request 03#40: T11699: Create a job to automate CNAME checks - 13https://github.com/miraheze/RequestSSL/pull/40#issuecomment-1986858750 [13:56:55] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 171.48 ms [13:58:58] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 168.22 ms [14:06:22] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 177.50 ms [14:10:26] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 186.79 ms [14:20:40] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 181.60 ms [14:24:43] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 178.55 ms [14:43:59] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 178.88 ms [14:52:05] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 183.06 ms [15:16:19] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 165.46 ms [15:18:20] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 175.18 ms [16:11:58] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 170.47 ms [16:12:26] PROBLEM - gil.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'gil.wiki' expires in 15 day(s) (Mon 25 Mar 2024 04:02:27 PM GMT +0000). [16:12:38] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/13ce4d906502...bf526b77b49e [16:12:39] [02miraheze/ssl] 07MirahezeSSLBot 03bf526b7 - Bot: Update SSL cert for gil.wiki [16:14:01] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 170.97 ms [16:24:31] PROBLEM - fanonpedia.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'fanonpedia.com' expires in 15 day(s) (Mon 25 Mar 2024 04:05:42 PM GMT +0000). [16:24:42] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/bf526b77b49e...396b5233dd08 [16:24:45] [02miraheze/ssl] 07MirahezeSSLBot 03396b523 - Bot: Update SSL cert for fanonpedia.com [16:26:26] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 171.19 ms [16:28:29] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 170.19 ms [16:34:18] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query uk.eu.org. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [16:53:01] RECOVERY - fanonpedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'fanonpedia.com' will expire on Fri 07 Jun 2024 03:24:36 PM GMT +0000. [16:57:24] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 168.96 ms [16:59:27] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 169.50 ms [17:04:02] RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp36.wikitide.net - CNAME OK [17:12:16] RECOVERY - gil.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'gil.wiki' will expire on Fri 07 Jun 2024 03:12:31 PM GMT +0000. [17:31:29] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 168.25 ms [17:33:32] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 178.49 ms [17:56:04] [02RequestSSL] 07Universal-Omega reviewed pull request 03#40 commit - 13https://github.com/miraheze/RequestSSL/pull/40#discussion_r1518629801 [17:56:06] [02RequestSSL] 07Universal-Omega reviewed pull request 03#40 commit - 13https://github.com/miraheze/RequestSSL/pull/40#discussion_r1518632670 [17:56:07] [02RequestSSL] 07Universal-Omega reviewed pull request 03#40 commit - 13https://github.com/miraheze/RequestSSL/pull/40#discussion_r1518629925 [17:56:09] [02RequestSSL] 07Universal-Omega reviewed pull request 03#40 commit - 13https://github.com/miraheze/RequestSSL/pull/40#discussion_r1518630798 [17:56:12] [02RequestSSL] 07Universal-Omega reviewed pull request 03#40 commit - 13https://github.com/miraheze/RequestSSL/pull/40#discussion_r1518631379 [17:56:15] [02RequestSSL] 07Universal-Omega reviewed pull request 03#40 commit - 13https://github.com/miraheze/RequestSSL/pull/40#discussion_r1518631053 [18:23:25] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 171.60 ms [18:25:28] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 166.36 ms [19:01:18] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/396b5233dd08...2d68ef6f01cd [19:01:20] [02miraheze/ssl] 07MirahezeSSLBot 032d68ef6 - Bot: Add SSL cert for wiki.greatphermesia.com [19:38:21] PROBLEM - wiki.cityrp.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.cityrp.org could not be found [19:54:43] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/2d68ef6f01cd...2d1190f58bfc [19:54:45] [02miraheze/ssl] 07MirahezeSSLBot 032d1190f - Bot: Add SSL cert for wiki.cogsci.live [22:06:54] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 169.00 ms [22:08:54] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 160.80 ms [23:18:32] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.mahdiruiz.line.pm. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [23:38:17] PROBLEM - cp36 Disk Space on cp36 is WARNING: DISK WARNING - free space: / 9713MiB (10% inode=98%); [23:40:51] PROBLEM - cp37 Disk Space on cp37 is WARNING: DISK WARNING - free space: / 9697MiB (10% inode=98%); [23:48:23] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp36.wikitide.net - CNAME OK