[00:00:00] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [00:01:03] PROBLEM - wiki.sheepservermc.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.sheepservermc.net All nameservers failed to answer the query. [00:01:25] PROBLEM - wiki.mobilityengineer.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mobilityengineer.com All nameservers failed to answer the query. [00:01:34] PROBLEM - puritwiki.p-e.kr - LetsEncrypt on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket [00:01:45] PROBLEM - wiki.tmyt105.leyhp.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.tmyt105.leyhp.com All nameservers failed to answer the query. [00:03:34] PROBLEM - wiki.villagecollaborative.net - LetsEncrypt on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket [00:04:45] PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call [00:05:25] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 18.79, 23.01, 22.40 [00:06:44] RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.074 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206 [00:10:00] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [00:15:20] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 25.18, 21.48, 21.37 [00:17:19] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 23.43, 21.59, 21.40 [00:19:19] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 27.31, 23.43, 22.08 [00:19:45] PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:22:50] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 20.83, 19.70, 17.75 [00:23:51] RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0) [00:24:46] RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 17.84, 18.63, 17.57 [00:26:37] PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call [00:27:27] PROBLEM - franchise.franchising.org.ua - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for franchise.franchising.org.ua could not be found [00:27:37] RECOVERY - wiki.kscucf.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.kscucf.org' will expire on Sun 15 Sep 2024 10:29:47 PM GMT +0000. [00:29:14] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 17.40, 23.10, 23.12 [00:31:13] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 25.97, 23.83, 23.33 [00:32:25] [Grafana] FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [00:33:12] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 20.86, 22.59, 22.94 [00:35:09] PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:37:10] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.11, 22.12, 22.58 [00:39:26] RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 7.541 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206 [00:39:57] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.26, 3.43, 3.98 [00:41:15] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 21.91, 19.94, 18.40 [00:41:57] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 5.31, 4.24, 4.22 [00:42:25] [Grafana] FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [00:42:38] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 21.52, 19.17, 18.05 [00:43:13] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 24.72, 21.43, 19.11 [00:43:19] RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0) [00:43:58] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.11, 3.52, 3.95 [00:44:02] PROBLEM - mw162 Current Load on mw162 is CRITICAL: LOAD CRITICAL - total load average: 24.95, 20.50, 17.93 [00:44:38] PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 25.43, 21.68, 19.12 [00:45:11] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 21.58, 21.67, 19.51 [00:45:53] PROBLEM - mw172 Current Load on mw172 is CRITICAL: LOAD CRITICAL - total load average: 25.38, 20.83, 18.50 [00:45:57] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 3.15, 3.88, 4.06 [00:45:58] RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 19.95, 19.85, 17.98 [00:46:38] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 20.02, 21.25, 19.29 [00:47:49] PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 22.68, 21.63, 19.09 [00:49:46] RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 15.82, 19.59, 18.66 [00:50:15] RECOVERY - wiki.kscucf.org - reverse DNS on sslhost is OK: SSL OK - wiki.kscucf.org reverse DNS resolves to cp36.wikitide.net - CNAME OK [00:50:38] RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 18.86, 20.21, 19.34 [00:51:04] RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 16.15, 19.52, 19.46 [00:53:03] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 23.07, 22.48, 23.59 [00:55:02] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 26.04, 24.09, 24.06 [00:56:30] PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call [00:58:30] RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 1.809 second response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206 [01:00:59] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 20.39, 23.06, 23.79 [01:07:49] RECOVERY - www.kinitopedia.lol - reverse DNS on sslhost is OK: SSL OK - www.kinitopedia.lol reverse DNS resolves to cp36.wikitide.net - CNAME OK [01:07:57] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 0.24, 2.28, 3.65 [01:08:55] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 14.25, 16.28, 20.16 [01:09:57] RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 0.10, 1.53, 3.21 [01:21:45] RECOVERY - wiki.wikimedia.cat - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.wikimedia.cat' will expire on Thu 10 Oct 2024 07:43:36 AM GMT +0000. [01:29:02] RECOVERY - wiki.sheepservermc.net - reverse DNS on sslhost is OK: SSL OK - wiki.sheepservermc.net reverse DNS resolves to cp36.wikitide.net - CNAME OK [01:30:02] RECOVERY - puritwiki.p-e.kr - LetsEncrypt on sslhost is OK: OK - Certificate 'puritwiki.p-e.kr' will expire on Sat 19 Oct 2024 02:09:03 PM GMT +0000. [01:31:10] PROBLEM - db161 Current Load on db161 is CRITICAL: LOAD CRITICAL - total load average: 34.47, 16.05, 6.72 [01:34:49] RECOVERY - wiki.astralprojections.org - reverse DNS on sslhost is OK: SSL OK - wiki.astralprojections.org reverse DNS resolves to cp36.wikitide.net - CNAME OK [01:36:41] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 25.69, 22.00, 20.02 [01:38:41] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.68, 21.94, 20.25 [01:39:10] RECOVERY - db161 Current Load on db161 is OK: LOAD OK - total load average: 0.69, 8.73, 8.58 [01:39:46] PROBLEM - www.kinitopedia.lol - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - www.kinitopedia.lol All nameservers failed to answer the query. [01:40:28] RECOVERY - wiki.wikimedia.cat - reverse DNS on sslhost is OK: SSL OK - wiki.wikimedia.cat reverse DNS resolves to cp36.wikitide.net - CNAME OK [01:45:30] RECOVERY - www.thesimswiki.com - reverse DNS on sslhost is OK: SSL OK - www.thesimswiki.com reverse DNS resolves to cp36.wikitide.net - CNAME OK [01:46:37] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 19.06, 20.24, 20.01 [01:48:58] RECOVERY - wiki.tmyt105.leyhp.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.tmyt105.leyhp.com' will expire on Thu 12 Sep 2024 12:27:49 PM GMT +0000. [01:51:34] RECOVERY - wiki.joust.ro - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.joust.ro' will expire on Wed 09 Oct 2024 09:25:14 PM GMT +0000. [01:51:44] RECOVERY - wiki.eggsdstudios.com - reverse DNS on sslhost is OK: SSL OK - wiki.eggsdstudios.com reverse DNS resolves to cp36.wikitide.net - CNAME OK [02:00:28] RECOVERY - wiki.tmyt105.leyhp.com - reverse DNS on sslhost is OK: SSL OK - wiki.tmyt105.leyhp.com reverse DNS resolves to cp36.wikitide.net - CNAME OK [02:00:33] RECOVERY - wiki.nowchess.org - reverse DNS on sslhost is OK: SSL OK - wiki.nowchess.org reverse DNS resolves to cp36.wikitide.net - CNAME OK [02:00:46] RECOVERY - wiki.corgicam.tv - reverse DNS on sslhost is OK: SSL OK - wiki.corgicam.tv reverse DNS resolves to cp36.wikitide.net - CNAME OK [02:01:44] RECOVERY - wiki.villagecollaborative.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.villagecollaborative.net' will expire on Wed 09 Oct 2024 09:20:12 PM GMT +0000. [02:08:00] RECOVERY - tno.wiki - reverse DNS on sslhost is OK: SSL OK - tno.wiki reverse DNS resolves to cp36.wikitide.net - CNAME OK [02:08:00] RECOVERY - wiki.junkstore.xyz - reverse DNS on sslhost is OK: SSL OK - wiki.junkstore.xyz reverse DNS resolves to cp36.wikitide.net - CNAME OK [02:08:05] RECOVERY - wiki.ate42.ru - reverse DNS on sslhost is OK: SSL OK - wiki.ate42.ru reverse DNS resolves to cp36.wikitide.net - CNAME OK [02:12:28] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 23.63, 21.02, 19.62 [02:14:27] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 19.28, 20.14, 19.46 [02:17:18] RECOVERY - minescape.wiki - reverse DNS on sslhost is OK: SSL OK - minescape.wiki reverse DNS resolves to cp36.wikitide.net - CNAME FLAT [02:17:19] RECOVERY - vise.dayid.org - reverse DNS on sslhost is OK: SSL OK - vise.dayid.org reverse DNS resolves to cp36.wikitide.net - CNAME OK [02:18:24] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 25.76, 22.99, 20.78 [02:20:28] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 23.09, 19.46, 16.83 [02:22:22] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.42, 23.15, 21.39 [02:22:26] RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 20.33, 19.19, 17.03 [02:22:53] RECOVERY - gufengcheng.top - reverse DNS on sslhost is OK: SSL OK - gufengcheng.top reverse DNS resolves to cp36.wikitide.net - CNAME OK [02:26:09] RECOVERY - wiki.joust.ro - reverse DNS on sslhost is OK: SSL OK - wiki.joust.ro reverse DNS resolves to cp36.wikitide.net - CNAME OK [02:26:20] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 15.22, 19.26, 20.26 [02:29:18] RECOVERY - wiki.mobilityengineer.com - reverse DNS on sslhost is OK: SSL OK - wiki.mobilityengineer.com reverse DNS resolves to cp36.wikitide.net - CNAME OK [02:36:10] RECOVERY - wiki.digitalcandela.com - reverse DNS on sslhost is OK: SSL OK - wiki.digitalcandela.com reverse DNS resolves to cp36.wikitide.net - CNAME OK [02:37:25] [Grafana] FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [02:39:33] RECOVERY - www.kinitopedia.lol - reverse DNS on sslhost is OK: SSL OK - www.kinitopedia.lol reverse DNS resolves to cp36.wikitide.net - CNAME OK [02:50:18] RECOVERY - puritwiki.p-e.kr - reverse DNS on sslhost is OK: SSL OK - puritwiki.p-e.kr reverse DNS resolves to cp36.wikitide.net - CNAME OK [02:51:07] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 20.35, 20.76, 20.36 [02:53:06] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 16.05, 18.84, 19.69 [02:57:03] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 19.87, 20.53, 20.25 [02:57:58] PROBLEM - wiki.joust.ro - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.joust.ro All nameservers failed to answer the query. [03:00:19] PROBLEM - cp27 Varnish Backends on cp27 is CRITICAL: 1 backends are down. mw172 [03:00:21] PROBLEM - cp36 Varnish Backends on cp36 is CRITICAL: 1 backends are down. mw172 [03:00:26] PROBLEM - cp26 Varnish Backends on cp26 is CRITICAL: 1 backends are down. mw172 [03:01:01] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 28.15, 23.71, 21.51 [03:01:16] RECOVERY - db151 Backups SQL on db151 is OK: FILE_AGE OK: /var/log/sql-backup.log is 75 seconds old and 0 bytes [03:01:19] PROBLEM - wiki.mobilityengineer.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mobilityengineer.com All nameservers failed to answer the query. [03:02:17] RECOVERY - cp27 Varnish Backends on cp27 is OK: All 19 backends are healthy [03:02:21] RECOVERY - cp36 Varnish Backends on cp36 is OK: All 19 backends are healthy [03:02:26] RECOVERY - cp26 Varnish Backends on cp26 is OK: All 19 backends are healthy [03:03:58] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 6.13, 2.99, 1.24 [03:05:57] RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 2.93, 2.63, 1.31 [03:06:36] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 20.75, 20.41, 18.28 [03:07:25] [Grafana] FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [03:07:55] PROBLEM - wiki.digitalcandela.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.digitalcandela.com All nameservers failed to answer the query. [03:08:34] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 27.01, 23.07, 19.53 [03:08:55] PROBLEM - mw161 Current Load on mw161 is CRITICAL: LOAD CRITICAL - total load average: 25.48, 21.12, 17.55 [03:09:35] PROBLEM - mw162 Current Load on mw162 is CRITICAL: LOAD CRITICAL - total load average: 26.10, 21.53, 17.74 [03:09:40] PROBLEM - mw172 Current Load on mw172 is CRITICAL: LOAD CRITICAL - total load average: 25.52, 20.86, 17.43 [03:10:03] PROBLEM - tno.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - tno.wiki All nameservers failed to answer the query. [03:10:38] PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 24.74, 22.16, 18.92 [03:12:30] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 20.87, 23.56, 20.64 [03:12:53] PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 23.67, 22.85, 19.08 [03:13:50] [02mw-config] 07OAuthority pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/a182ac555cb2...c02ca1698d8a [03:13:53] [02mw-config] 07OAuthority 03c02ca16 - Add back for now [03:14:38] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 19.45, 22.28, 19.87 [03:14:45] miraheze/mw-config - OAuthority the build passed. [03:15:23] PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 17.89, 21.49, 19.05 [03:15:33] PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 18.86, 21.42, 18.94 [03:16:52] RECOVERY - mw161 Current Load on mw161 is OK: LOAD OK - total load average: 13.87, 19.36, 18.65 [03:17:19] RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 16.72, 19.97, 18.78 [03:18:24] RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 18.93, 20.06, 20.03 [03:18:38] RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 17.87, 19.89, 19.43 [03:19:33] RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 17.12, 20.25, 19.13 [03:22:51] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.45, 22.87, 23.85 [03:23:52] !log [@test151] starting deploy of {'config': True} to test151 [03:23:53] !log [@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [03:23:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:24:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:24:50] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 28.11, 24.68, 24.38 [03:26:12] RECOVERY - wiki.thesimswiki.com - reverse DNS on sslhost is OK: SSL OK - wiki.thesimswiki.com reverse DNS resolves to cp36.wikitide.net - CNAME OK [03:27:49] RECOVERY - wiki.joust.ro - reverse DNS on sslhost is OK: SSL OK - wiki.joust.ro reverse DNS resolves to cp36.wikitide.net - CNAME OK [03:28:28] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query pt.eu.org. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [03:30:37] RECOVERY - wiki.mobilityengineer.com - reverse DNS on sslhost is OK: SSL OK - wiki.mobilityengineer.com reverse DNS resolves to cp36.wikitide.net - CNAME OK [03:31:38] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 6.22, 4.35, 3.17 [03:32:21] !log [@mwtask181] starting deploy of {'config': True} to all [03:32:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:32:45] !log [@mwtask181] finished deploy of {'config': True} to all - SUCCESS in 24s [03:32:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:35:01] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 20.58, 20.36, 20.08 [03:36:59] RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 16.70, 19.20, 19.70 [03:37:35] !log [@mwtask171] starting deploy of {'config': True} to all [03:37:36] RECOVERY - wiki.digitalcandela.com - reverse DNS on sslhost is OK: SSL OK - wiki.digitalcandela.com reverse DNS resolves to cp36.wikitide.net - CNAME OK [03:37:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:37:50] RECOVERY - wiki.yahia.xyz - reverse DNS on sslhost is OK: SSL OK - wiki.yahia.xyz reverse DNS resolves to cp36.wikitide.net - CNAME OK [03:37:50] !log [@mwtask171] finished deploy of {'config': True} to all - SUCCESS in 15s [03:38:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:39:03] RECOVERY - tno.wiki - reverse DNS on sslhost is OK: SSL OK - tno.wiki reverse DNS resolves to cp36.wikitide.net - CNAME OK [03:39:24] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.75, 3.78, 3.48 [03:40:43] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 16.87, 21.20, 23.69 [03:41:21] RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 2.23, 3.10, 3.26 [03:42:08] RECOVERY - wiki.villagecollaborative.net - reverse DNS on sslhost is OK: SSL OK - wiki.villagecollaborative.net reverse DNS resolves to cp36.wikitide.net - CNAME OK [03:56:35] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 15.85, 17.41, 20.08 [03:57:55] RECOVERY - wiki.gab.pt.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.gab.pt.eu.org reverse DNS resolves to cp36.wikitide.net - CNAME OK [03:58:45] PROBLEM - wiki.cubestudios.xyz - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.cubestudios.xyz All nameservers failed to answer the query. [04:00:00] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 8.66, 5.18, 3.96 [04:00:49] PROBLEM - legacygt.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - legacygt.wiki All nameservers failed to answer the query. [04:03:57] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 1.55, 3.42, 3.51 [04:05:57] RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 1.27, 3.00, 3.37 [04:10:35] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 29.05, 23.01, 20.70 [04:16:35] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 20.35, 24.00, 22.09 [04:19:54] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 11.13, 6.18, 4.36 [04:22:35] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.89, 22.23, 21.70 [04:23:47] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 1.62, 4.00, 3.88 [04:24:35] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 18.75, 20.96, 21.31 [04:28:21] RECOVERY - wiki.cubestudios.xyz - reverse DNS on sslhost is OK: SSL OK - wiki.cubestudios.xyz reverse DNS resolves to cp36.wikitide.net - CNAME OK [04:29:34] RECOVERY - legacygt.wiki - reverse DNS on sslhost is OK: SSL OK - legacygt.wiki reverse DNS resolves to cp36.wikitide.net - CNAME OK [04:29:37] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 4.74, 3.54, 3.67 [04:31:34] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.31, 3.10, 3.48 [04:33:31] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 7.21, 4.63, 3.99 [04:33:47] PROBLEM - ru-teirailway.f5.si - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - ru-teirailway.f5.si All nameservers failed to answer the query. [04:35:28] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.04, 3.74, 3.74 [04:37:25] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 6.70, 4.89, 4.16 [04:41:18] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 1.31, 3.47, 3.79 [04:43:15] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 4.10, 3.88, 3.91 [04:50:35] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 13.92, 17.66, 19.75 [04:52:25] [Grafana] RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [04:53:22] PROBLEM - wiki.stag.lol - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.stag.lol All nameservers failed to answer the query. [04:54:56] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.11, 3.48, 3.79 [04:57:18] PROBLEM - ao90.pinho.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - ao90.pinho.org All nameservers failed to answer the query. [04:57:25] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [04:57:29] PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call [04:58:35] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.56, 21.10, 20.32 [04:59:57] PROBLEM - wiki.cubestudios.xyz - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.cubestudios.xyz All nameservers failed to answer the query. [05:00:23] [Grafana] FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [05:00:49] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 4.78, 3.94, 3.87 [05:00:50] PROBLEM - legacygt.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - legacygt.wiki All nameservers failed to answer the query. [05:01:36] RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.098 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206 [05:02:35] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 18.31, 19.55, 19.90 [05:04:43] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 0.91, 2.89, 3.51 [05:05:23] [Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [05:06:39] RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 0.16, 1.98, 3.10 [05:08:06] PROBLEM - wiki.artmechanicum.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.artmechanicum.com All nameservers failed to answer the query. [05:09:06] PROBLEM - wiki.pixlies.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.pixlies.net All nameservers failed to answer the query. [05:19:32] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.76, 19.63, 18.69 [05:19:40] PROBLEM - wiki.artmechanicum.com - LetsEncrypt on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket [05:21:31] PROBLEM - wiki.case-clicker.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.case-clicker.com All nameservers failed to answer the query. [05:23:31] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 29.21, 22.68, 19.97 [05:25:22] PROBLEM - wiki.tulpa.info - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.tulpa.info All nameservers failed to answer the query. [05:25:30] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 23.78, 21.90, 19.94 [05:29:28] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 14.51, 18.58, 19.07 [05:31:20] PROBLEM - data.nonbinary.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - data.nonbinary.wiki All nameservers failed to answer the query. [05:46:16] PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [05:47:25] [Grafana] FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [05:57:25] [Grafana] RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [06:08:08] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.37, 21.00, 19.16 [06:12:06] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 19.14, 19.97, 19.17 [06:14:58] RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:16:04] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.04, 22.71, 20.51 [06:17:46] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 21.80, 19.69, 17.49 [06:18:03] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.73, 22.43, 20.69 [06:19:46] RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 18.50, 18.98, 17.49 [06:22:01] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 26.02, 23.37, 21.36 [06:25:59] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.12, 23.71, 22.04 [06:37:53] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 26.19, 21.80, 21.36 [06:39:52] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.12, 21.75, 21.42 [06:45:50] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 16.25, 18.52, 20.03 [06:59:42] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 20.43, 19.03, 19.12 [07:01:41] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 18.23, 19.10, 19.16 [07:04:00] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 5.11, 2.45, 1.01 [07:05:14] PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call [07:06:19] PROBLEM - prometheus151 APT on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:06:20] [Grafana] FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [07:07:13] RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.070 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206 [07:07:57] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.66, 3.65, 1.91 [07:08:19] RECOVERY - prometheus151 APT on prometheus151 is OK: APT OK: 51 packages available for upgrade (0 critical updates). [07:09:57] RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 0.68, 2.53, 1.70 [07:11:20] [Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [07:17:12] PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [07:41:21] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.52, 21.15, 19.12 [07:43:20] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 19.84, 20.96, 19.34 [07:45:19] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 17.35, 19.88, 19.15 [07:58:36] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 25.90, 23.07, 18.79 [07:59:10] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 25.91, 23.57, 21.25 [08:00:34] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 15.67, 20.53, 18.39 [08:01:09] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 16.74, 21.13, 20.65 [08:02:31] RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 14.28, 18.28, 17.82 [08:03:09] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 16.52, 19.45, 20.08 [08:14:20] RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [08:21:18] RECOVERY - wiki.stag.lol - reverse DNS on sslhost is OK: SSL OK - wiki.stag.lol reverse DNS resolves to cp36.wikitide.net - CNAME OK [08:23:39] PROBLEM - wiki.joust.ro - LetsEncrypt on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket [08:23:49] PROBLEM - wiki.wikimedia.cat - LetsEncrypt on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket [08:24:32] RECOVERY - wiki.tulpa.info - reverse DNS on sslhost is OK: SSL OK - wiki.tulpa.info reverse DNS resolves to cp36.wikitide.net - CNAME OK [08:24:53] PROBLEM - gufengcheng.top - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - gufengcheng.top All nameservers failed to answer the query. [08:25:39] RECOVERY - ao90.pinho.org - reverse DNS on sslhost is OK: SSL OK - ao90.pinho.org reverse DNS resolves to cp36.wikitide.net - CNAME OK [08:27:12] RECOVERY - wiki.cubestudios.xyz - reverse DNS on sslhost is OK: SSL OK - wiki.cubestudios.xyz reverse DNS resolves to cp36.wikitide.net - CNAME OK [08:27:56] RECOVERY - data.nonbinary.wiki - reverse DNS on sslhost is OK: SSL OK - data.nonbinary.wiki reverse DNS resolves to cp36.wikitide.net - CNAME OK [08:28:46] RECOVERY - legacygt.wiki - reverse DNS on sslhost is OK: SSL OK - legacygt.wiki reverse DNS resolves to cp36.wikitide.net - CNAME OK [08:28:55] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 26.16, 21.71, 19.26 [08:30:54] RECOVERY - ru-teirailway.f5.si - reverse DNS on sslhost is OK: SSL OK - ru-teirailway.f5.si reverse DNS resolves to cp36.wikitide.net - CNAME OK [08:32:54] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 17.62, 20.88, 19.60 [08:33:41] RECOVERY - wiki.artmechanicum.com - reverse DNS on sslhost is OK: SSL OK - wiki.artmechanicum.com reverse DNS resolves to cp36.wikitide.net - CNAME OK [08:36:52] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 18.57, 19.71, 19.40 [08:37:05] RECOVERY - wiki.pixlies.net - reverse DNS on sslhost is OK: SSL OK - wiki.pixlies.net reverse DNS resolves to cp36.wikitide.net - CNAME OK [08:37:49] [02MediaWikiDebugJS] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/MediaWikiDebugJS/compare/69a335226eef...0c15d5d2b89f [08:37:50] [02MediaWikiDebugJS] 07Universal-Omega 030c15d5d - Add support for abxy and minor cleanup [08:47:36] RECOVERY - wiki.artmechanicum.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.artmechanicum.com' will expire on Fri 11 Oct 2024 04:38:59 PM GMT +0000. [08:48:57] RECOVERY - wiki.case-clicker.com - reverse DNS on sslhost is OK: SSL OK - wiki.case-clicker.com reverse DNS resolves to cp36.wikitide.net - CNAME OK [08:52:41] RECOVERY - wiki.joust.ro - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.joust.ro' will expire on Wed 09 Oct 2024 09:25:14 PM GMT +0000. [08:52:43] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 23.55, 20.74, 19.83 [08:53:01] RECOVERY - wiki.wikimedia.cat - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.wikimedia.cat' will expire on Thu 10 Oct 2024 07:43:36 AM GMT +0000. [08:53:44] RECOVERY - gufengcheng.top - reverse DNS on sslhost is OK: SSL OK - gufengcheng.top reverse DNS resolves to cp36.wikitide.net - CNAME OK [08:54:42] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 18.53, 20.22, 19.78 [09:22:44] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 6.25, 4.45, 2.81 [09:24:41] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.46, 3.70, 2.71 [09:26:35] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.87, 20.30, 19.16 [09:26:39] RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 2.25, 2.97, 2.55 [09:28:35] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 16.57, 19.43, 19.02 [09:29:29] PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call [09:30:36] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 7.27, 5.27, 3.58 [09:31:27] RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.066 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206 [09:36:00] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [09:44:38] PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 29.51, 20.31, 16.21 [09:46:38] RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 15.55, 18.33, 16.01 [09:52:04] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.08, 3.50, 3.85 [09:55:59] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 6.99, 4.61, 4.14 [09:57:27] PROBLEM - franchise.franchising.org.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - franchise.franchising.org.ua All nameservers failed to answer the query. [09:57:57] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.31, 3.71, 3.87 [10:03:57] RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 1.18, 2.38, 3.27 [10:06:00] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [10:07:15] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 18.34, 20.88, 23.39 [10:09:58] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 8.99, 5.08, 4.00 [10:13:57] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.08, 3.94, 3.76 [10:15:15] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 28.98, 24.07, 23.44 [10:17:57] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 4.74, 3.57, 3.60 [10:18:12] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.70, 19.88, 17.72 [10:19:57] RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 2.91, 3.01, 3.37 [10:20:11] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 19.03, 19.47, 17.84 [10:27:26] PROBLEM - franchise.franchising.org.ua - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for franchise.franchising.org.ua could not be found [10:27:57] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 3.78, 4.22, 3.74 [10:35:57] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.55, 3.89, 3.81 [10:37:57] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 6.32, 4.99, 4.23 [10:41:58] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.52, 3.43, 3.75 [10:49:44] PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call [10:49:58] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 6.02, 4.36, 3.98 [10:51:42] RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.074 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206 [10:52:20] [Grafana] FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [10:53:09] PROBLEM - prometheus151 APT on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:55:27] RECOVERY - prometheus151 APT on prometheus151 is OK: APT OK: 51 packages available for upgrade (0 critical updates). [10:56:53] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 20.75, 19.78, 18.28 [11:00:51] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 25.44, 21.83, 19.44 [11:02:50] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.07, 22.11, 19.89 [11:03:36] PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:04:25] PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call [11:06:26] RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 2.773 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206 [11:06:48] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 26.80, 23.14, 20.66 [11:07:20] [Grafana] FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [11:08:47] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.32, 22.79, 20.87 [11:09:37] RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0) [11:12:45] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 26.28, 23.88, 21.71 [11:18:40] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 20.80, 20.21, 18.12 [11:20:38] RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 19.22, 19.42, 18.05 [11:24:40] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 23.69, 23.32, 22.88 [11:27:20] [Grafana] RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [11:28:30] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [11:30:30] PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call [11:32:29] RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.066 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206 [11:36:55] PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call [11:38:54] RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.083 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206 [11:42:35] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 18.77, 18.60, 20.35 [11:43:30] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [11:49:58] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.05, 3.50, 3.99 [11:51:57] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 5.93, 4.49, 4.30 [11:53:31] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 20.95, 19.28, 19.60 [11:55:30] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 18.99, 19.01, 19.46 [11:59:57] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 1.40, 3.23, 3.86 [12:01:57] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 6.03, 4.37, 4.20 [12:09:57] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 1.66, 3.01, 3.71 [12:13:57] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 6.57, 4.13, 3.92 [12:18:26] PROBLEM - cp26 Varnish Backends on cp26 is CRITICAL: 1 backends are down. mw172 [12:19:01] PROBLEM - cp51 Varnish Backends on cp51 is CRITICAL: 2 backends are down. mw172 mw181 [12:19:46] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 22.10, 19.14, 16.48 [12:20:32] RECOVERY - cp26 Varnish Backends on cp26 is OK: All 19 backends are healthy [12:21:00] RECOVERY - cp51 Varnish Backends on cp51 is OK: All 19 backends are healthy [12:21:15] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 26.18, 23.19, 20.58 [12:21:46] RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 16.38, 17.67, 16.24 [12:23:14] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.13, 22.74, 20.75 [12:33:14] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 35.92, 24.43, 21.90 [12:35:13] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 18.46, 21.48, 21.13 [12:35:34] PROBLEM - cp27 Varnish Backends on cp27 is CRITICAL: 1 backends are down. mw181 [12:37:34] RECOVERY - cp27 Varnish Backends on cp27 is OK: All 19 backends are healthy [12:37:57] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 1.50, 3.27, 3.93 [12:39:11] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [12:41:58] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 6.84, 4.26, 4.11 [12:43:10] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 14.15, 18.61, 20.04 [12:59:11] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [13:05:57] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.42, 21.31, 19.72 [13:07:57] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 16.65, 20.27, 19.60 [13:11:54] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 25.43, 22.28, 20.53 [13:13:53] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 17.59, 21.03, 20.33 [13:14:11] [Grafana] FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [13:15:52] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 26.62, 23.12, 21.17 [13:17:57] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.75, 3.55, 4.00 [13:18:10] PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 22.99, 19.26, 17.81 [13:19:50] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 23.36, 23.75, 21.88 [13:19:57] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 4.91, 3.82, 4.02 [13:20:06] RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 18.02, 18.91, 17.88 [13:21:58] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.60, 3.12, 3.72 [13:23:37] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.andreijiroh.uk.eu.org All nameservers failed to answer the query. [13:23:48] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.80, 23.03, 21.93 [13:23:57] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 4.78, 3.88, 3.94 [13:25:57] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.14, 3.60, 3.86 [13:29:45] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.95, 23.69, 22.71 [13:29:57] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 4.41, 3.56, 3.76 [13:31:45] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.71, 24.01, 22.94 [13:33:44] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.70, 23.41, 22.83 [13:33:57] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.99, 3.82, 3.85 [13:37:42] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 25.92, 23.98, 23.14 [13:41:58] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 4.33, 3.53, 3.69 [13:43:39] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.09, 23.54, 23.28 [13:43:57] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.42, 3.83, 3.80 [13:45:58] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 6.90, 4.47, 4.00 [13:47:37] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.08, 22.97, 23.03 [13:47:57] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.94, 3.63, 3.74 [13:49:11] [Grafana] RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [13:49:36] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 20.92, 21.78, 22.56 [13:51:11] PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call [13:51:35] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 27.64, 23.45, 23.04 [13:51:58] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 5.63, 4.29, 3.96 [13:52:45] RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp36.wikitide.net - CNAME OK [13:53:35] PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:54:11] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [13:55:26] PROBLEM - prometheus151 Puppet on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:55:29] PROBLEM - prometheus151 APT on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:55:32] RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0) [13:55:34] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.41, 23.07, 23.07 [13:57:32] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.41, 23.22, 23.10 [13:58:00] RECOVERY - prometheus151 Puppet on prometheus151 is OK: OK: Puppet is currently enabled, last run 11 minutes ago with 0 failures [13:58:02] RECOVERY - prometheus151 APT on prometheus151 is OK: APT OK: 51 packages available for upgrade (0 critical updates). [14:01:31] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 19.35, 23.16, 23.26 [14:03:53] RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.099 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206 [14:04:11] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [14:11:47] PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:12:16]