[00:02:27] PROBLEM - thesimswiki.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:02:41] PROBLEM - cp51 Current Load on cp51 is CRITICAL: LOAD CRITICAL - total load average: 12.15, 9.02, 4.43 [00:03:01] PROBLEM - cp36 HTTP 4xx/5xx ERROR Rate on cp36 is CRITICAL: CRITICAL - NGINX Error Rate is 100% [00:04:51] RECOVERY - cp41 Disk Space on cp41 is OK: DISK OK - free space: / 19151MiB (20% inode=98%); [00:04:56] RECOVERY - cp36 Disk Space on cp36 is OK: DISK OK - free space: / 20413MiB (23% inode=98%); [00:06:28] RECOVERY - Host phorge171 is UP: PING OK - Packet loss = 0%, RTA = 0.24 ms [00:06:30] RECOVERY - Host ldap171 is UP: PING OK - Packet loss = 0%, RTA = 0.78 ms [00:06:48] RECOVERY - Host mw171 is UP: PING OK - Packet loss = 0%, RTA = 0.25 ms [00:06:49] RECOVERY - Host mw172 is UP: PING OK - Packet loss = 0%, RTA = 0.84 ms [00:06:54] RECOVERY - cp36 NTP time on cp36 is OK: NTP OK: Offset -0.002576172352 secs [00:07:01] PROBLEM - cp36 HTTP 4xx/5xx ERROR Rate on cp36 is WARNING: WARNING - NGINX Error Rate is 56% [00:07:18] RECOVERY - Host cp37 is UP: PING OK - Packet loss = 0%, RTA = 0.29 ms [00:07:28] RECOVERY - cp37 Nginx Backend for mwtask181 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8160 [00:07:28] RECOVERY - cp37 Nginx Backend for swiftproxy161 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8206 [00:07:28] RECOVERY - cp37 Current Load on cp37 is OK: LOAD OK - total load average: 0.67, 0.20, 0.07 [00:07:29] RECOVERY - cp37 Nginx Backend for mw172 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8118 [00:07:29] RECOVERY - cp37 Nginx Backend for mon181 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8201 [00:07:29] RECOVERY - cp36 Varnish Backends on cp36 is OK: All 19 backends are healthy [00:07:30] RECOVERY - phorge171 phd on phorge171 is OK: PROCS OK: 1 process with args 'phd' [00:07:31] RECOVERY - mw172 Disk Space on mw172 is OK: DISK OK - free space: / 29876MiB (56% inode=87%); [00:07:31] RECOVERY - cp37 Nginx Backend for matomo151 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8203 [00:07:31] RECOVERY - cp37 Nginx Backend for mw171 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8117 [00:07:33] RECOVERY - phorge171 conntrack_table_size on phorge171 is OK: OK: nf_conntrack is 0 % full [00:07:33] RECOVERY - cp37 Nginx Backend for mw161 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8115 [00:07:34] RECOVERY - mw172 Puppet on mw172 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [00:07:34] RECOVERY - cp37 Nginx Backend for mwtask171 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8161 [00:07:35] RECOVERY - cp37 Nginx Backend for mw181 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8119 [00:07:35] RECOVERY - cp37 NTP time on cp37 is OK: NTP OK: Offset -0.007249951363 secs [00:07:43] RECOVERY - ldap171 ferm_active on ldap171 is OK: OK ferm input default policy is set [00:07:43] RECOVERY - ldap171 conntrack_table_size on ldap171 is OK: OK: nf_conntrack is 0 % full [00:07:44] RECOVERY - cp27 Varnish Backends on cp27 is OK: All 19 backends are healthy [00:07:50] RECOVERY - cp26 Varnish Backends on cp26 is OK: All 19 backends are healthy [00:07:50] RECOVERY - mw172 HTTPS on mw172 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 285 bytes in 0.065 second response time [00:07:56] RECOVERY - mw171 MediaWiki Rendering on mw171 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.311 second response time [00:07:56] RECOVERY - cp37 Nginx Backend for mw151 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8113 [00:07:58] RECOVERY - mw171 Puppet on mw171 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [00:08:00] RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 2.41, 0.77, 0.27 [00:08:02] RECOVERY - cp37 Varnish Backends on cp37 is OK: All 19 backends are healthy [00:08:02] RECOVERY - cp36 PowerDNS Recursor on cp36 is OK: DNS OK: 0.323 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206 [00:08:03] RECOVERY - mw172 ferm_active on mw172 is OK: OK ferm input default policy is set [00:08:03] RECOVERY - mw171 SSH on mw171 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u2 (protocol 2.0) [00:08:04] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [00:08:05] RECOVERY - phorge171 Disk Space on phorge171 is OK: DISK OK - free space: / 25191MiB (56% inode=93%); [00:08:05] RECOVERY - phorge171 Current Load on phorge171 is OK: LOAD OK - total load average: 0.47, 0.18, 0.06 [00:08:06] RECOVERY - ldap171 LDAP on ldap171 is OK: LDAP OK - 0.006 seconds response time [00:08:07] RECOVERY - mw171 conntrack_table_size on mw171 is OK: OK: nf_conntrack is 2 % full [00:08:09] RECOVERY - cp36 HTTPS on cp36 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3761 bytes in 0.065 second response time [00:08:09] RECOVERY - phorge171 NTP time on phorge171 is OK: NTP OK: Offset -0.002682179213 secs [00:08:12] RECOVERY - cp51 Varnish Backends on cp51 is OK: All 19 backends are healthy [00:08:12] RECOVERY - phorge171 phorge-static.wikitide.net HTTPS on phorge171 is OK: HTTP OK: Status line output matched "HTTP/1.1 200" - 17717 bytes in 0.044 second response time [00:08:12] RECOVERY - cp37 Nginx Backend for reports171 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8205 [00:08:15] RECOVERY - cp37 Nginx Backend for phorge171 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8202 [00:08:16] RECOVERY - Host bots171 is UP: PING OK - Packet loss = 0%, RTA = 0.70 ms [00:08:18] RECOVERY - ldap171 SSH on ldap171 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u2 (protocol 2.0) [00:08:18] RECOVERY - ldap171 Current Load on ldap171 is OK: LOAD OK - total load average: 0.43, 0.33, 0.13 [00:08:21] RECOVERY - ldap171 Puppet on ldap171 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [00:08:22] RECOVERY - cp37 Nginx Backend for mw182 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8120 [00:08:22] RECOVERY - cp37 Nginx Backend for mw162 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8116 [00:08:22] RECOVERY - cp37 Nginx Backend for puppet181 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8204 [00:08:27] RECOVERY - mw171 PowerDNS Recursor on mw171 is OK: DNS OK: 0.032 seconds response time. wikitide.net returns 2407:3641:2161:9774::1,46.250.240.167 [00:08:32] RECOVERY - cp37 Nginx Backend for test151 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8181 [00:08:32] RECOVERY - ping on mw171 is OK: PING OK - Packet loss = 0%, RTA = 0.24 ms [00:08:35] PROBLEM - cp51 Current Load on cp51 is WARNING: LOAD WARNING - total load average: 3.99, 7.24, 5.30 [00:08:37] PROBLEM - cp37 Disk Space on cp37 is WARNING: DISK WARNING - free space: / 9059MiB (10% inode=98%); [00:08:47] RECOVERY - cp37 Nginx Backend for swiftproxy171 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8207 [00:08:47] RECOVERY - cp37 PowerDNS Recursor on cp37 is OK: DNS OK: 0.042 seconds response time. wikitide.net returns 2602:294:0:b23::112,38.46.223.206 [00:08:47] RECOVERY - cp37 HTTPS on cp37 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3759 bytes in 0.067 second response time [00:09:01] RECOVERY - cp36 HTTP 4xx/5xx ERROR Rate on cp36 is OK: OK - NGINX Error Rate is 4% [00:09:01] RECOVERY - cp41 Varnish Backends on cp41 is OK: All 19 backends are healthy [00:09:02] RECOVERY - cp37 Nginx Backend for mw152 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8114 [00:09:27] RECOVERY - ldap171 NTP time on ldap171 is OK: NTP OK: Offset -0.0002628862858 secs [00:09:32] RECOVERY - bots171 IRC Log Server Bot on bots171 is OK: PROCS OK: 1 process with args 'irclogserverbot.py' [00:09:44] RECOVERY - cp37 Puppet on cp37 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:09:57] RECOVERY - bots171 Current Load on bots171 is OK: LOAD OK - total load average: 0.30, 0.38, 0.18 [00:09:57] RECOVERY - bots171 IRC-Discord Relay Bot on bots171 is OK: PROCS OK: 4 processes with args 'relaybot' [00:10:02] RECOVERY - bots171 PowerDNS Recursor on bots171 is OK: DNS OK: 0.040 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206 [00:10:02] RECOVERY - bots171 IRC RC Bot on bots171 is OK: PROCS OK: 1 process with args 'ircrcbot.py' [00:10:07] RECOVERY - bots171 Disk Space on bots171 is OK: DISK OK - free space: / 13029MiB (72% inode=91%); [00:10:33] RECOVERY - cp51 Current Load on cp51 is OK: LOAD OK - total load average: 1.52, 5.36, 4.84 [00:12:56] PROBLEM - cp36 Disk Space on cp36 is WARNING: DISK WARNING - free space: / 7934MiB (8% inode=98%); [00:16:56] RECOVERY - cp36 Disk Space on cp36 is OK: DISK OK - free space: / 22807MiB (25% inode=98%); [00:23:21] PROBLEM - wiki.aclevo.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.aclevo.com' expires in 11 day(s) (Tue 30 Apr 2024 06:30:41 PM GMT +0000). [00:23:35] [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/8e0422307f38...050b32ffae0f [00:23:37] [02miraheze/ssl] 07WikiTideSSLBot 03050b32f - Bot: Update SSL cert for wiki.aclevo.com [00:24:41] RECOVERY - smashbroswiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'smashbroswiki.com' will expire on Wed 15 May 2024 04:10:10 PM GMT +0000. [00:24:53] RECOVERY - wiki.teessidehackspace.org.uk - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.teessidehackspace.org.uk' will expire on Wed 15 May 2024 05:07:21 PM GMT +0000. [00:32:25] RECOVERY - thesimswiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.thesimswiki.com' will expire on Tue 16 Jul 2024 12:27:30 AM GMT +0000. [00:41:35] PROBLEM - mwtask171 Puppet on mwtask171 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [00:52:25] [Grafana] !sre FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [00:52:26] RECOVERY - wiki.aclevo.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.aclevo.com' will expire on Wed 17 Jul 2024 11:23:29 PM GMT +0000. [00:57:25] [Grafana] !sre RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [01:03:02] PROBLEM - www.pyramidgames.wiki - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query pyramidgames.wiki. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [01:09:35] RECOVERY - mwtask171 Puppet on mwtask171 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:27:06] PROBLEM - www.burnout.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - www.burnout.wiki All nameservers failed to answer the query. [01:32:21] RECOVERY - www.pyramidgames.wiki - reverse DNS on sslhost is OK: SSL OK - www.pyramidgames.wiki reverse DNS resolves to cp36.wikitide.net - NS RECORDS OK [01:38:52] miraheze/mw-config - anpang54 the build passed. [01:41:03] PROBLEM - tssm.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - tssm.wiki All nameservers failed to answer the query. [01:46:36] PROBLEM - miraheze.com - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query miraheze.com. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [01:50:40] PROBLEM - mystiverse.wiki - LetsEncrypt on sslhost is CRITICAL: Name or service not knownHTTP CRITICAL - Unable to open TCP socket [01:54:08] !log [@test151] starting deploy of {'config': True} to test151 [01:54:09] !log [@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [01:54:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:54:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:56:48] RECOVERY - www.burnout.wiki - reverse DNS on sslhost is OK: SSL OK - www.burnout.wiki reverse DNS resolves to cp36.wikitide.net - NS RECORDS OK [02:10:07] RECOVERY - tssm.wiki - reverse DNS on sslhost is OK: SSL OK - tssm.wiki reverse DNS resolves to cp37.wikitide.net - NS RECORDS OK [02:11:58] PROBLEM - removededm.com - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query removededm.com. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [02:15:21] RECOVERY - miraheze.com - reverse DNS on sslhost is OK: SSL OK - miraheze.com reverse DNS resolves to cp37.wikitide.net - NS RECORDS OK [02:19:31] PROBLEM - mystiverse.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for mystiverse.wiki could not be found [02:32:43] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 9.61, 8.00, 6.45 [02:34:43] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 7.07, 7.47, 6.44 [02:38:43] RECOVERY - cp41 Current Load on cp41 is OK: LOAD OK - total load average: 5.60, 6.57, 6.31 [02:40:27] !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/rebuildall.php --wiki=epicduelwikiwiki (START) [02:40:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:41:31] RECOVERY - removededm.com - reverse DNS on sslhost is OK: SSL OK - removededm.com reverse DNS resolves to cp37.wikitide.net - NS RECORDS OK [02:48:43] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 8.46, 6.79, 6.28 [02:50:43] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 6.88, 6.80, 6.34 [02:54:43] RECOVERY - cp41 Current Load on cp41 is OK: LOAD OK - total load average: 5.69, 6.46, 6.32 [02:58:43] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 7.33, 7.06, 6.59 [03:00:43] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 11.76, 7.83, 6.87 [03:01:07] RECOVERY - db181 Backups SQL on db181 is OK: FILE_AGE OK: /var/log/sql-backup.log is 66 seconds old and 0 bytes [03:01:18] RECOVERY - db161 Backups SQL on db161 is OK: FILE_AGE OK: /var/log/sql-backup.log is 77 seconds old and 0 bytes [03:02:43] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 7.12, 7.40, 6.83 [03:06:46] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query line.pm. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [03:08:43] RECOVERY - cp41 Current Load on cp41 is OK: LOAD OK - total load average: 5.46, 6.43, 6.55 [03:11:59] PROBLEM - smashbroswiki.com - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query smashbroswiki.com. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [03:15:34] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 6.97, 6.69, 6.59 [03:15:54] PROBLEM - looneypyramids.wiki - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query looneypyramids.wiki. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [03:17:20] [Grafana] !sre FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [03:17:30] RECOVERY - cp41 Current Load on cp41 is OK: LOAD OK - total load average: 5.04, 6.27, 6.46 [03:33:58] PROBLEM - lebork.info.pl - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - lebork.info.pl All nameservers failed to answer the query. [03:36:01] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp37.wikitide.net - CNAME OK [03:41:01] RECOVERY - smashbroswiki.com - reverse DNS on sslhost is OK: SSL OK - smashbroswiki.com reverse DNS resolves to cp36.wikitide.net - NS RECORDS OK [03:41:34] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 10.67, 7.56, 6.60 [03:43:29] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 7.57, 7.72, 6.79 [03:44:37] PROBLEM - looneypyramids.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - looneypyramids.wiki All nameservers failed to answer the query. [03:47:43] PROBLEM - yoshipedia.com - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query yoshipedia.com. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [03:49:14] RECOVERY - cp41 Current Load on cp41 is OK: LOAD OK - total load average: 5.13, 6.08, 6.40 [03:49:21] !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/rebuildall.php --wiki=epicduelwikiwiki (END - exit=0) [03:49:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:54:05] !log [@test151] starting deploy of {'config': True} to test151 [03:54:06] !log [@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [03:55:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:59:14] PROBLEM - wikitide.net - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wikitide.net. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [04:02:48] RECOVERY - lebork.info.pl - reverse DNS on sslhost is OK: SSL OK - lebork.info.pl reverse DNS resolves to cp37.wikitide.net - NS RECORDS OK [04:13:41] RECOVERY - looneypyramids.wiki - reverse DNS on sslhost is OK: SSL OK - looneypyramids.wiki reverse DNS resolves to cp37.wikitide.net - NS RECORDS OK [04:16:46] RECOVERY - yoshipedia.com - reverse DNS on sslhost is OK: SSL OK - yoshipedia.com reverse DNS resolves to cp37.wikitide.net - NS RECORDS OK [04:27:52] RECOVERY - wikitide.net - reverse DNS on sslhost is OK: SSL OK - wikitide.net reverse DNS resolves to cp37.wikitide.net - NS RECORDS OK [04:33:39] PROBLEM - cp51 Puppet on cp51 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [04:41:25] PROBLEM - richterian.com - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.407 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out. [04:49:55] PROBLEM - es.countryhumans.polandball.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'es.countryhumans.polandball.wiki' expires in 15 day(s) (Sun 05 May 2024 04:22:05 AM GMT +0000). [04:50:09] [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/050b32ffae0f...e66d0dd31312 [04:50:10] [02miraheze/ssl] 07WikiTideSSLBot 03e66d0dd - Bot: Update SSL cert for es.countryhumans.polandball.wiki [04:54:10] !log [@test151] starting deploy of {'config': True} to test151 [04:54:11] !log [@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [04:55:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:02:15] RECOVERY - cp51 Puppet on cp51 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [05:10:05] RECOVERY - richterian.com - reverse DNS on sslhost is OK: SSL OK - richterian.com reverse DNS resolves to cp36.wikitide.net - NS RECORDS OK [05:19:49] RECOVERY - es.countryhumans.polandball.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'es.countryhumans.polandball.wiki' will expire on Thu 18 Jul 2024 03:50:03 AM GMT +0000. [05:29:10] PROBLEM - thelonsdalebattalion.co.uk - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.406 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out. [05:34:45] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 8.65, 7.35, 6.54 [05:36:43] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 6.05, 6.87, 6.46 [05:37:14] PROBLEM - ru.countryhumans.polandball.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'ru.countryhumans.polandball.wiki' expires in 15 day(s) (Sun 05 May 2024 05:06:43 AM GMT +0000). [05:37:29] [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/e66d0dd31312...00ef94cb80d6 [05:37:30] [02miraheze/ssl] 07WikiTideSSLBot 0300ef94c - Bot: Update SSL cert for ru.countryhumans.polandball.wiki [05:38:44] PROBLEM - atlas.starworld.zone - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'atlas.starworld.zone' expires in 15 day(s) (Sun 05 May 2024 05:15:36 AM GMT +0000). [05:38:54] [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/00ef94cb80d6...5232eec7556b [05:38:55] [02miraheze/ssl] 07WikiTideSSLBot 035232eec - Bot: Update SSL cert for atlas.starworld.zone [05:39:27] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 172.31 ms [05:40:43] RECOVERY - cp41 Current Load on cp41 is OK: LOAD OK - total load average: 5.02, 6.44, 6.42 [05:41:28] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 170.37 ms [05:45:10] PROBLEM - antiguabarbudacalypso.com - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query antiguabarbudacalypso.com. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [05:56:37] PROBLEM - dragonquestwiki.com - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query dragonquestwiki.com. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [05:58:18] RECOVERY - thelonsdalebattalion.co.uk - reverse DNS on sslhost is OK: SSL OK - thelonsdalebattalion.co.uk reverse DNS resolves to cp36.wikitide.net - NS RECORDS OK [06:04:58] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 10.29, 8.28, 7.00 [06:05:18] PROBLEM - cp51 Puppet on cp51 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [06:06:29] RECOVERY - ru.countryhumans.polandball.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'ru.countryhumans.polandball.wiki' will expire on Thu 18 Jul 2024 04:37:23 AM GMT +0000. [06:06:59] PROBLEM - apeirology.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - apeirology.com All nameservers failed to answer the query. [06:07:54] RECOVERY - atlas.starworld.zone - LetsEncrypt on sslhost is OK: OK - Certificate 'atlas.starworld.zone' will expire on Thu 18 Jul 2024 04:38:48 AM GMT +0000. [06:08:14] PROBLEM - grayravens.com - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query grayravens.com. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [06:08:49] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 7.84, 7.84, 7.09 [06:10:44] RECOVERY - cp41 Current Load on cp41 is OK: LOAD OK - total load average: 4.29, 6.73, 6.78 [06:17:34] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 7.38, 7.62, 7.14 [06:19:26] PROBLEM - corru.wiki - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query corru.wiki. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [06:23:19] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 9.61, 7.85, 7.29 [06:25:49] RECOVERY - dragonquestwiki.com - reverse DNS on sslhost is OK: SSL OK - dragonquestwiki.com reverse DNS resolves to cp36.wikitide.net - NS RECORDS OK [06:29:04] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 4.63, 6.81, 7.12 [06:31:49] RECOVERY - cp51 Puppet on cp51 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:32:54] RECOVERY - cp41 Current Load on cp41 is OK: LOAD OK - total load average: 4.69, 5.88, 6.68 [06:34:19] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.mahdiruiz.line.pm. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [06:35:51] RECOVERY - apeirology.com - reverse DNS on sslhost is OK: SSL OK - apeirology.com reverse DNS resolves to cp36.wikitide.net - NS RECORDS OK [06:37:43] RECOVERY - grayravens.com - reverse DNS on sslhost is OK: SSL OK - grayravens.com reverse DNS resolves to cp37.wikitide.net - NS RECORDS OK [06:39:38] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 7.78, 7.09, 6.93 [06:43:29] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 9.49, 7.57, 7.10 [06:45:24] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 7.45, 7.38, 7.07 [06:48:40] RECOVERY - corru.wiki - reverse DNS on sslhost is OK: SSL OK - corru.wiki reverse DNS resolves to cp37.wikitide.net - NS RECORDS OK [06:49:14] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 8.54, 7.68, 7.24 [06:54:12] !log [@test151] starting deploy of {'config': True} to test151 [06:54:12] !log [@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [06:55:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:58:50] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 6.85, 7.48, 7.69 [07:02:43] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 8.90, 7.90, 7.79 [07:03:35] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp36.wikitide.net - CNAME OK [07:10:43] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 5.82, 7.70, 7.90 [07:13:32] RECOVERY - antiguabarbudacalypso.com - reverse DNS on sslhost is OK: SSL OK - antiguabarbudacalypso.com reverse DNS resolves to cp37.wikitide.net - NS RECORDS OK [07:17:13] PROBLEM - rct.wiki - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query rct.wiki. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [07:17:20] [Grafana] !sre FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [07:18:43] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 8.36, 8.23, 8.02 [07:19:41] PROBLEM - wiki.mahdiruiz.line.pm - LetsEncrypt on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket [07:20:43] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 5.98, 7.64, 7.84 [07:21:14] PROBLEM - mh142.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - mh142.com All nameservers failed to answer the query. [07:24:44] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 8.20, 7.86, 7.86 [07:24:44] PROBLEM - webkinzguide.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - webkinzguide.com All nameservers failed to answer the query. [07:26:43] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 5.26, 6.93, 7.52 [07:27:20] [Grafana] !sre RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [07:30:55] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 166.62 ms [07:32:56] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 166.23 ms [07:36:43] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 8.22, 7.66, 7.63 [07:38:43] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 6.55, 7.30, 7.50 [07:45:11] PROBLEM - antiguabarbudacalypso.com - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.404 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out. [07:45:49] RECOVERY - rct.wiki - reverse DNS on sslhost is OK: SSL OK - rct.wiki reverse DNS resolves to cp36.wikitide.net - NS RECORDS OK [07:49:13] RECOVERY - wiki.mahdiruiz.line.pm - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.mahdiruiz.line.pm' will expire on Fri 14 Jun 2024 04:28:50 PM GMT +0000. [07:50:41] RECOVERY - mh142.com - reverse DNS on sslhost is OK: SSL OK - mh142.com reverse DNS resolves to cp37.wikitide.net - NS RECORDS OK [07:50:44] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 9.93, 8.98, 8.08 [07:52:17] PROBLEM - cp37 Disk Space on cp37 is CRITICAL: DISK CRITICAL - free space: / 5309MiB (5% inode=98%); [07:53:26] RECOVERY - webkinzguide.com - reverse DNS on sslhost is OK: SSL OK - webkinzguide.com reverse DNS resolves to cp36.wikitide.net - NS RECORDS OK [08:04:29] PROBLEM - landofliberos.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - landofliberos.com All nameservers failed to answer the query. [08:14:35] RECOVERY - antiguabarbudacalypso.com - reverse DNS on sslhost is OK: SSL OK - antiguabarbudacalypso.com reverse DNS resolves to cp37.wikitide.net - NS RECORDS OK [08:17:41] PROBLEM - yoshipedia.com - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query yoshipedia.com. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [08:33:51] RECOVERY - landofliberos.com - reverse DNS on sslhost is OK: SSL OK - landofliberos.com reverse DNS resolves to cp37.wikitide.net - NS RECORDS OK [08:46:43] RECOVERY - yoshipedia.com - reverse DNS on sslhost is OK: SSL OK - yoshipedia.com reverse DNS resolves to cp36.wikitide.net - NS RECORDS OK [09:22:42] PROBLEM - www.project-patterns.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - www.project-patterns.com All nameservers failed to answer the query. [09:32:57] PROBLEM - cp51 Puppet on cp51 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [09:40:10] PROBLEM - fallofsanctuary.com - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query fallofsanctuary.com. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [09:44:16] PROBLEM - www.durawiki.com - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query durawiki.com. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [09:49:09] PROBLEM - lostmediawiki.ru - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - lostmediawiki.ru All nameservers failed to answer the query. [09:52:15] RECOVERY - www.project-patterns.com - reverse DNS on sslhost is OK: SSL OK - www.project-patterns.com reverse DNS resolves to cp37.wikitide.net - NS RECORDS OK [09:59:57] PROBLEM - polcompball.wikitide.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wikitide.org. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [10:01:30] RECOVERY - cp51 Puppet on cp51 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [10:05:43] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query line.pm. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [10:10:00] RECOVERY - fallofsanctuary.com - reverse DNS on sslhost is OK: SSL OK - fallofsanctuary.com reverse DNS resolves to cp37.wikitide.net - NS RECORDS OK [10:12:51] RECOVERY - www.durawiki.com - reverse DNS on sslhost is OK: SSL OK - www.durawiki.com reverse DNS resolves to cp36.wikitide.net - NS RECORDS OK [10:18:21] RECOVERY - lostmediawiki.ru - reverse DNS on sslhost is OK: SSL OK - lostmediawiki.ru reverse DNS resolves to cp36.wikitide.net - NS RECORDS OK [10:29:15] RECOVERY - polcompball.wikitide.org - reverse DNS on sslhost is OK: SSL OK - polcompball.wikitide.org reverse DNS resolves to cp36.wikitide.net - NS RECORDS OK [11:29:49] PROBLEM - db162 MariaDB on db162 is UNKNOWN: check_mysql: Invalid hostname/address - db162.wikitide.netUsage: check_mysql [-d database] [-H host] [-P port] [-s socket] [-u user] [-p password] [-S] [-l] [-a cert] [-k key] [-C ca-cert] [-D ca-dir] [-L ciphers] [-f optfile] [-g group] [11:31:46] PROBLEM - db162 MariaDB on db162 is CRITICAL: Access denied for user 'icinga'@'2602:294:0:b12::110' (using password: YES) [11:33:31] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp41.wikitide.net - CNAME OK [11:40:03] [02MirahezeMagic] 07waki285 synchronize pull request 03#489: Add electionadmin - 13https://github.com/miraheze/MirahezeMagic/pull/489 [11:43:55] PROBLEM - tdr.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'tdr.wiki' expires in 15 day(s) (Sun 05 May 2024 11:19:28 AM GMT +0000). [11:44:13] [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/5232eec7556b...359fa619ef6b [11:44:16] [02miraheze/ssl] 07WikiTideSSLBot 03359fa61 - Bot: Update SSL cert for tdr.wiki [12:13:22] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 27.30, 23.18, 17.10 [12:13:40] RECOVERY - tdr.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'tdr.wiki' will expire on Thu 18 Jul 2024 10:44:07 AM GMT +0000. [12:18:35] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 25.72, 19.63, 14.63 [12:20:35] RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 17.61, 19.33, 15.18 [12:20:51] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 23.31, 19.75, 15.23 [12:22:51] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 18.79, 18.92, 15.45 [12:24:35] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 20.43, 20.76, 16.76 [12:26:23] PROBLEM - franchise.franchising.org.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - franchise.franchising.org.ua All nameservers failed to answer the query. [12:28:51] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.90, 20.61, 17.18 [12:30:59] PROBLEM - volunteerforukraine.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - volunteerforukraine.com All nameservers failed to answer the query. [12:34:51] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 15.80, 18.91, 17.61 [12:40:51] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 20.46, 20.91, 18.91 [12:42:35] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 24.96, 22.38, 20.32 [12:45:00] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [12:46:35] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 23.91, 23.40, 21.20 [12:46:51] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 18.64, 20.33, 19.35 [12:48:08] PROBLEM - antiguabarbudacalypso.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - antiguabarbudacalypso.com All nameservers failed to answer the query. [12:48:23] PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 20.52, 19.91, 17.30 [12:50:24] RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 17.73, 18.89, 17.23 [12:50:51] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 23.00, 21.67, 20.08 [12:52:35] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 25.89, 23.48, 21.94 [12:54:08] !log [@test151] starting deploy of {'config': True} to test151 [12:54:09] !log [@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [12:54:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:54:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:56:21] RECOVERY - franchise.franchising.org.ua - reverse DNS on sslhost is OK: SSL OK - franchise.franchising.org.ua reverse DNS resolves to cp37.wikitide.net - CNAME OK [12:56:35] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 19.39, 23.64, 22.54 [12:58:51] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 11.74, 17.55, 19.16 [12:59:48] RECOVERY - volunteerforukraine.com - reverse DNS on sslhost is OK: SSL OK - volunteerforukraine.com reverse DNS resolves to cp37.wikitide.net - NS RECORDS OK [13:00:00] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [13:00:35] RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 11.91, 17.35, 20.18 [13:08:28] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 23.02, 22.06, 23.94 [13:08:53] PROBLEM - www.permanentfuturelab.wiki - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query permanentfuturelab.wiki. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [13:13:39] PROBLEM - portalsofphereon.com - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query portalsofphereon.com. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [13:17:32] RECOVERY - antiguabarbudacalypso.com - reverse DNS on sslhost is OK: SSL OK - antiguabarbudacalypso.com reverse DNS resolves to cp37.wikitide.net - NS RECORDS OK [13:22:28] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.68, 22.11, 22.53 [13:26:28] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 23.43, 23.48, 23.01 [13:28:28] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 28.59, 25.21, 23.69 [13:34:28] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 13.29, 22.19, 23.35 [13:38:06] RECOVERY - www.permanentfuturelab.wiki - reverse DNS on sslhost is OK: SSL OK - www.permanentfuturelab.wiki reverse DNS resolves to cp37.wikitide.net - NS RECORDS OK [13:38:28] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 8.70, 14.40, 19.84 [13:42:56] RECOVERY - portalsofphereon.com - reverse DNS on sslhost is OK: SSL OK - portalsofphereon.com reverse DNS resolves to cp37.wikitide.net - NS RECORDS OK [14:07:09] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 22.01, 16.44, 12.36 [14:08:28] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 29.82, 22.23, 16.48 [14:11:07] RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 18.08, 17.49, 13.73 [14:12:28] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 20.42, 22.78, 18.16 [14:16:28] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.35, 22.50, 19.00 [14:18:28] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 17.84, 21.06, 18.93 [14:20:28] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 16.96, 19.85, 18.76 [14:22:15] PROBLEM - christipedia.nl - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.403 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out. [14:22:35] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 167.59 ms [14:24:36] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 165.99 ms [14:29:22] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 26.25, 23.32, 20.70 [14:31:20] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.69, 22.61, 20.73 [14:33:18] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 28.29, 24.58, 21.67 [14:33:52] PROBLEM - os151 Current Load on os151 is WARNING: LOAD WARNING - total load average: 3.75, 3.16, 2.25 [14:35:52] RECOVERY - os151 Current Load on os151 is OK: LOAD OK - total load average: 2.70, 2.95, 2.28 [14:39:09] !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/initSiteStats.php --wiki=epicduelwikiwiki --update (END - exit=0) [14:39:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:42:45] PROBLEM - os151 Disk Space on os151 is WARNING: DISK WARNING - free space: / 22843MiB (10% inode=99%); [14:48:33] RECOVERY - os151 Disk Space on os151 is OK: DISK OK - free space: / 31096MiB (14% inode=99%); [14:51:28] RECOVERY - christipedia.nl - reverse DNS on sslhost is OK: SSL OK - christipedia.nl reverse DNS resolves to cp36.wikitide.net - NS RECORDS OK [14:58:14] [02miraheze/mediawiki-repos] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mediawiki-repos/compare/c01174c1613c...d70ff5066852 [14:58:16] [02miraheze/mediawiki-repos] 07AgentIsai 03d70ff50 - T11918: Install OreDict and Tilesheets [14:58:43] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 22.07, 21.14, 19.08 [15:00:42] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 25.36, 22.21, 19.69 [15:03:00] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [15:05:23] PROBLEM - trollpasta.com - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query trollpasta.com. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [15:06:59] [02miraheze/mw-config] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/897e816ddc8a...ad9bf9ad5c24 [15:07:02] [02miraheze/mw-config] 07AgentIsai 03ad9bf9a - T11918: Allow OreDict and Tilesheets to be enabled through ManageWiki [15:07:40] !log [@mwtask171] starting deploy of {'config': True} to all [15:07:58] !log [@mwtask171] DEPLOY ABORTED: Canary check failed for publictestwiki.com@mw181.wikitide.net [15:08:13] miraheze/mw-config - AgentIsai the build has errored. [15:08:38] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 15.42, 23.19, 21.79 [15:09:25] [02miraheze/mw-config] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/ad9bf9ad5c24...d79ee04ba748 [15:09:27] [02miraheze/mw-config] 07AgentIsai 03d79ee04 - Fix indentation [15:09:53] PROBLEM - mw181 MediaWiki Rendering on mw181 is UNKNOWN: HTTP UNKNOWN: Failed to unchunk message body [15:09:56] PROBLEM - mw151 MediaWiki Rendering on mw151 is UNKNOWN: HTTP UNKNOWN: Failed to unchunk message body [15:09:56] PROBLEM - mw171 MediaWiki Rendering on mw171 is UNKNOWN: HTTP UNKNOWN: Failed to unchunk message body [15:10:06] PROBLEM - mw181 HTTPS on mw181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 500 [15:10:08] PROBLEM - cp27 HTTPS on cp27 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 500 [15:10:13] PROBLEM - mw161 MediaWiki Rendering on mw161 is UNKNOWN: HTTP UNKNOWN: Failed to unchunk message body [15:10:20] miraheze/mw-config - AgentIsai the build passed. [15:10:35] PROBLEM - mwtask171 MediaWiki Rendering on mwtask171 is UNKNOWN: HTTP UNKNOWN: Failed to unchunk message body [15:10:37] RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 5.56, 16.91, 19.66 [15:10:46] PROBLEM - mw162 HTTPS on mw162 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 500 [15:10:48] PROBLEM - cp36 HTTPS on cp36 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 500 [15:10:49] PROBLEM - mw151 HTTPS on mw151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 500 [15:10:49] PROBLEM - mw172 MediaWiki Rendering on mw172 is UNKNOWN: HTTP UNKNOWN: Failed to unchunk message body [15:10:49] PROBLEM - cp41 HTTPS on cp41 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 500 [15:10:57] PROBLEM - mw161 HTTPS on mw161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 500 [15:11:17] Uh oh [15:11:19] PROBLEM - mwtask171 HTTPS on mwtask171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 500 [15:11:19] PROBLEM - mw171 HTTPS on mw171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 500 [15:11:26] PROBLEM - mw162 MediaWiki Rendering on mw162 is UNKNOWN: HTTP UNKNOWN: Failed to unchunk message body [15:11:26] PROBLEM - cp26 HTTPS on cp26 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 500 [15:11:35] PROBLEM - mwtask171 Puppet on mwtask171 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[MediaWiki Config Sync] [15:11:39] PROBLEM - mw152 HTTPS on mw152 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 500 [15:11:43] PROBLEM - mw152 MediaWiki Rendering on mw152 is UNKNOWN: HTTP UNKNOWN: Failed to unchunk message body [15:11:46] PROBLEM - mw172 HTTPS on mw172 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 500 [15:12:11] !log [agent@mwtask181] starting deploy of {'folders': '1.41/extensions/OreDict,1.41/extensions/Tilesheets'} to all [15:12:13] !log [agent@mwtask181] DEPLOY ABORTED: Canary check failed for publictestwiki.com@mw151.wikitide.net [15:12:31] !log [agent@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [15:12:39] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 4.80, 14.38, 22.47 [15:12:43] !log [agent@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 12s [15:12:46] RECOVERY - mw162 HTTPS on mw162 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 285 bytes in 0.066 second response time [15:12:46] !log [agent@mwtask181] starting deploy of {'folders': '1.41/extensions/OreDict,1.41/extensions/Tilesheets'} to all [15:12:47] RECOVERY - cp36 HTTPS on cp36 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3761 bytes in 0.077 second response time [15:12:48] RECOVERY - cp41 HTTPS on cp41 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3783 bytes in 0.983 second response time [15:12:49] RECOVERY - mw151 HTTPS on mw151 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 285 bytes in 0.062 second response time [15:12:49] RECOVERY - mw172 MediaWiki Rendering on mw172 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.550 second response time [15:12:52] PROBLEM - cp41 Disk Space on cp41 is WARNING: DISK WARNING - free space: / 10067MiB (10% inode=98%); [15:12:57] RECOVERY - mw161 HTTPS on mw161 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 285 bytes in 0.062 second response time [15:13:00] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [15:13:11] !log [agent@mwtask181] finished deploy of {'folders': '1.41/extensions/OreDict,1.41/extensions/Tilesheets'} to all - SUCCESS in 24s [15:13:19] RECOVERY - mwtask171 HTTPS on mwtask171 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 285 bytes in 0.072 second response time [15:13:19] RECOVERY - mw171 HTTPS on mw171 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 285 bytes in 0.063 second response time [15:13:21] RECOVERY - cp26 HTTPS on cp26 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3758 bytes in 0.899 second response time [15:13:26] RECOVERY - mw162 MediaWiki Rendering on mw162 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.563 second response time [15:13:36] !log [agent@mwtask181] starting deploy of {'extension_list': True, 'versions': '1.41'} to all [15:13:39] RECOVERY - mw152 HTTPS on mw152 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 285 bytes in 0.070 second response time [15:13:42] RECOVERY - mw181 MediaWiki Rendering on mw181 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.937 second response time [15:13:43] RECOVERY - mw152 MediaWiki Rendering on mw152 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.294 second response time [15:13:46] RECOVERY - mw172 HTTPS on mw172 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 285 bytes in 0.118 second response time [15:13:48] !log [agent@mwtask181] finished deploy of {'extension_list': True, 'versions': '1.41'} to all - SUCCESS in 12s [15:13:56] RECOVERY - mw171 MediaWiki Rendering on mw171 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.291 second response time [15:13:56] RECOVERY - mw151 MediaWiki Rendering on mw151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.318 second response time [15:14:03] RECOVERY - mw181 HTTPS on mw181 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 285 bytes in 0.104 second response time [15:14:05] RECOVERY - cp27 HTTPS on cp27 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3759 bytes in 0.899 second response time [15:14:13] RECOVERY - mw161 MediaWiki Rendering on mw161 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.522 second response time [15:14:35] RECOVERY - mwtask171 MediaWiki Rendering on mwtask171 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.307 second response time [15:17:21] PROBLEM - antiguabarbudacalypso.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - antiguabarbudacalypso.com All nameservers failed to answer the query. [15:20:31] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 27.02, 20.68, 21.96 [15:23:26] !log [@test151] starting deploy of {'config': True} to test151 [15:23:27] !log [@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [15:23:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:23:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:24:35] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 29.38, 23.50, 20.31 [15:26:35] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 23.42, 23.73, 20.82 [15:28:35] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 24.08, 24.05, 21.30 [15:29:43] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.21, 20.12, 18.45 [15:31:39] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 27.73, 22.72, 19.59 [15:34:11] RECOVERY - trollpasta.com - reverse DNS on sslhost is OK: SSL OK - trollpasta.com reverse DNS resolves to cp36.wikitide.net - NS RECORDS OK [15:35:12] PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 20.04, 20.49, 18.67 [15:37:11] PROBLEM - mw172 Current Load on mw172 is CRITICAL: LOAD CRITICAL - total load average: 25.03, 22.20, 19.53 [15:37:36] !log [@mwtask171] starting deploy of {'config': True} to all [15:38:23] !log [@mwtask171] finished deploy of {'config': True} to all - SUCCESS in 47s [15:39:10] RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 5.03, 16.27, 17.74 [15:39:34] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 23.99, 19.53, 16.02 [15:39:35]