[00:01:20] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 27.00, 23.32, 22.11 [00:01:23] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 5.27, 4.09, 3.73 [00:03:20] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 18.59, 21.45, 21.58 [00:07:00] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [00:07:20] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.57, 22.59, 21.99 [00:09:03] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.47, 3.62, 3.74 [00:09:20] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 20.24, 21.86, 21.82 [00:10:58] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 4.40, 3.61, 3.70 [00:15:20] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.16, 23.34, 22.50 [00:21:20] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 19.13, 22.75, 22.70 [00:27:20] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 25.49, 23.33, 22.83 [00:28:21] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 21.20, 19.28, 16.78 [00:28:31] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.11, 3.33, 3.92 [00:28:51] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.28, 20.31, 18.93 [00:29:20] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 19.45, 21.68, 22.27 [00:30:21] RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 17.06, 18.06, 16.62 [00:30:32] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 4.79, 3.66, 3.95 [00:32:51] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.23, 21.19, 19.50 [00:34:51] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.77, 21.77, 19.93 [00:35:20] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 25.62, 22.92, 22.51 [00:37:20] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 20.55, 21.56, 22.04 [00:39:23] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [00:40:31] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.49, 3.54, 3.86 [00:40:51] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 18.36, 19.99, 19.80 [00:42:32] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 5.10, 4.26, 4.10 [00:43:12] PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call [00:45:20] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.50, 22.26, 21.94 [00:47:11] RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.059 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206 [00:47:20] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.86, 22.10, 21.92 [00:47:27] RECOVERY - wiki.qadrishattari.xyz - reverse DNS on sslhost is OK: SSL OK - wiki.qadrishattari.xyz reverse DNS resolves to cp36.wikitide.net - CNAME OK [00:50:31] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 1.99, 3.45, 3.88 [00:54:23] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [00:58:31] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 6.40, 3.81, 3.71 [00:59:02] RECOVERY - wiki.qadrishattari.xyz - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.qadrishattari.xyz' will expire on Thu 10 Oct 2024 02:20:07 AM GMT +0000. [00:59:20] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 25.79, 23.08, 22.26 [00:59:41] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.36, 20.38, 19.51 [01:01:38] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 17.62, 19.34, 19.23 [01:02:25] [Grafana] FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [01:03:20] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.96, 22.59, 22.28 [01:05:20] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 26.12, 24.00, 22.84 [01:07:20] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.58, 22.88, 22.56 [01:09:20] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.84, 23.48, 22.81 [01:10:18] PROBLEM - os162 Current Load on os162 is CRITICAL: LOAD CRITICAL - total load average: 8.48, 7.80, 7.51 [01:10:32] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.46, 3.60, 3.84 [01:11:18] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.14, 19.56, 19.20 [01:12:18] PROBLEM - os162 Current Load on os162 is WARNING: LOAD WARNING - total load average: 7.60, 7.69, 7.50 [01:12:32] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 5.74, 4.45, 4.12 [01:13:15] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 17.11, 18.77, 18.97 [01:13:20] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 18.52, 21.85, 22.37 [01:20:32] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.48, 3.38, 3.81 [01:24:32] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 4.32, 3.56, 3.76 [01:24:57] PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call [01:26:31] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.75, 3.59, 3.77 [01:26:52] RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.075 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206 [01:27:25] [Grafana] FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [01:28:51] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 20.94, 20.01, 18.98 [01:30:31] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 4.99, 3.69, 3.73 [01:32:31] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.63, 3.89, 3.82 [01:32:51] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 17.54, 19.54, 19.10 [01:36:32] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 4.43, 3.70, 3.70 [01:37:20] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 25.26, 23.10, 22.49 [01:38:31] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.33, 3.83, 3.77 [01:41:20] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.53, 22.76, 22.50 [01:44:32] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 5.46, 3.44, 3.50 [01:46:31] RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 2.42, 3.09, 3.38 [01:52:31] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 4.89, 4.30, 3.80 [01:54:02] PROBLEM - prometheus151 Puppet on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [02:05:20] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 25.67, 23.46, 22.55 [02:05:46] RECOVERY - prometheus151 Puppet on prometheus151 is OK: OK: Puppet is currently enabled, last run 19 minutes ago with 0 failures [02:07:20] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 18.35, 21.73, 22.05 [02:07:25] [Grafana] FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [02:21:20] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 25.48, 23.65, 22.90 [02:23:20] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 17.83, 22.13, 22.48 [02:29:20] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 25.85, 23.45, 22.88 [02:30:51] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 23.70, 21.56, 19.33 [02:31:20] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.54, 22.69, 22.65 [02:34:51] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 19.11, 20.01, 19.23 [02:35:20] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.33, 22.19, 22.32 [02:37:20] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.44, 22.19, 22.33 [02:40:31] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.09, 3.53, 3.99 [02:42:32] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 5.46, 4.25, 4.20 [02:43:06] PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call [02:43:20] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.24, 22.95, 22.52 [02:44:51] PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:45:20] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.16, 22.68, 22.48 [02:45:28] PROBLEM - prometheus151 APT on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [02:45:29] PROBLEM - prometheus151 Puppet on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [02:46:56] RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0) [02:47:20] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.17, 22.80, 22.52 [02:47:25] [Grafana] FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [02:50:31] RECOVERY - prometheus151 Puppet on prometheus151 is OK: OK: Puppet is currently enabled, last run 35 minutes ago with 0 failures [02:50:33] RECOVERY - prometheus151 APT on prometheus151 is OK: APT OK: 51 packages available for upgrade (0 critical updates). [02:51:14] RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.063 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206 [02:52:25] [Grafana] FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [02:53:20] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 19.81, 22.54, 22.66 [03:03:20] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 26.13, 23.39, 22.87 [03:05:20] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 23.14, 22.98, 22.78 [03:08:31] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 0.20, 2.30, 3.76 [03:10:31] RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 0.04, 1.54, 3.31 [03:17:20] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 25.15, 23.25, 22.65 [03:19:20] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.40, 22.87, 22.59 [03:21:38] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.54, 19.83, 18.49 [03:23:35] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 18.11, 19.09, 18.39 [03:29:20] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.10, 22.16, 22.28 [03:33:20] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 18.09, 21.27, 22.01 [03:35:20] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.91, 22.92, 22.52 [03:37:20] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.55, 22.34, 22.36 [03:43:20] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.51, 23.01, 22.56 [03:45:20] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.55, 22.73, 22.53 [03:49:20] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.45, 22.67, 22.46 [03:51:20] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.24, 21.96, 22.20 [03:56:51] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.52, 20.53, 18.87 [03:58:51] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 20.84, 20.64, 19.13 [04:00:51] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 17.28, 19.21, 18.78 [04:05:20] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 25.61, 22.78, 22.30 [04:07:20] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 20.46, 21.60, 21.90 [04:09:20] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 25.25, 23.49, 22.59 [04:10:54] PROBLEM - franchise.franchising.org.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - franchise.franchising.org.ua All nameservers failed to answer the query. [04:15:20] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 18.06, 22.31, 22.52 [04:17:20] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 25.21, 23.63, 22.98 [04:21:20] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 20.88, 22.85, 22.88 [04:22:25] [Grafana] FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [04:23:20] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 25.56, 24.15, 23.36 [04:33:20] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 20.75, 22.51, 23.14 [04:37:20] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 25.83, 24.02, 23.59 [04:39:20] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 19.99, 22.53, 23.10 [04:40:53] PROBLEM - franchise.franchising.org.ua - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for franchise.franchising.org.ua could not be found [04:41:20] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.51, 23.21, 23.26 [04:44:21] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 25.90, 20.73, 19.02 [04:46:18] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.62, 21.58, 19.59 [04:48:15] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 17.27, 20.10, 19.29 [04:57:25] [Grafana] FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [04:59:20] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 23.21, 23.22, 23.83 [05:03:32] PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:04:31] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 4.48, 3.02, 1.32 [05:05:14] PROBLEM - ping6 on cp26 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 302.23 ms [05:06:05] PROBLEM - cp26 HTTPS on cp26 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [05:07:03] PROBLEM - cp26 Varnish Backends on cp26 is CRITICAL: 1 backends are down. mw151 [05:07:36] RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0) [05:07:59] !log [@mwtask171] starting deploy of {'config': True} to all [05:08:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:08:07] RECOVERY - cp26 HTTPS on cp26 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3820 bytes in 5.292 second response time [05:08:12] !log [@mwtask171] finished deploy of {'config': True} to all - SUCCESS in 13s [05:08:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:08:31] RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 3.02, 2.80, 1.60 [05:09:03] RECOVERY - cp26 Varnish Backends on cp26 is OK: All 19 backends are healthy [05:09:20] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.76, 23.23, 23.21 [05:09:21] PROBLEM - ping6 on cp26 is WARNING: PING WARNING - Packet loss = 0%, RTA = 299.12 ms [05:11:20] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.93, 23.13, 23.20 [05:11:23] PROBLEM - ping6 on cp26 is CRITICAL: PING CRITICAL - Packet loss = 0%, RTA = 301.10 ms [05:12:00] PROBLEM - mwtask171 Puppet on mwtask171 is CRITICAL: CRITICAL: Puppet has 3 failures. Last run 2 minutes ago with 3 failures. Failed resources (up to 3 shown): Exec[git_pull_ErrorPages],Exec[git_pull_femiwiki-deploy-1.42],Exec[git_pull_3d2png] [05:12:00] PROBLEM - phorge171 Puppet on phorge171 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_phorge-extensions] [05:14:30] PROBLEM - cp26 HTTPS on cp26 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [05:14:31] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 6.29, 3.91, 2.33 [05:15:20] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 25.36, 23.27, 23.17 [05:16:31] RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 3.32, 3.33, 2.30 [05:19:20] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.09, 23.00, 23.13 [05:21:43] PROBLEM - cp26 Varnish Backends on cp26 is CRITICAL: 2 backends are down. mw181 mw182 [05:22:57] RECOVERY - cp26 HTTPS on cp26 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3820 bytes in 6.455 second response time [05:25:09] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.45, 20.61, 18.54 [05:25:20] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.26, 23.41, 23.19 [05:26:40] RECOVERY - cp26 Varnish Backends on cp26 is OK: All 19 backends are healthy [05:27:06] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 19.01, 19.52, 18.37 [05:27:20] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 20.81, 22.34, 22.81 [05:29:20] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.23, 23.00, 22.99 [05:30:39] PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call [05:31:04] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 7.35, 4.67, 3.29 [05:32:12] PROBLEM - ping6 on cp26 is WARNING: PING WARNING - Packet loss = 0%, RTA = 298.08 ms [05:32:34] RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.066 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206 [05:32:59] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.47, 3.99, 3.23 [05:33:20] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 23.51, 23.62, 23.26 [05:34:15] RECOVERY - ping6 on cp26 is OK: PING OK - Packet loss = 0%, RTA = 182.55 ms [05:35:35] [02ImportDump] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/2ebf8acd329b...adfd5f0962a4 [05:35:38] [02ImportDump] 07BlankEclair 03adfd5f0 - T12434: Fix uploading dumps (#116) [05:35:41] [02ImportDump] 07Universal-Omega closed pull request 03#116: T12434: Fix uploading dumps - 13https://github.com/miraheze/ImportDump/pull/116 [05:35:50] [02ImportDump] 07Universal-Omega closed pull request 03#117: Update mediawiki/mediawiki-codesniffer requirement from 43.0.0 to 44.0.0 - 13https://github.com/miraheze/ImportDump/pull/117 [05:35:52] [02ImportDump] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/adfd5f0962a4...608b6c4f596b [05:35:54] [02ImportDump] 07dependabot[bot] 03608b6c4 - Update mediawiki/mediawiki-codesniffer requirement from 43.0.0 to 44.0.0 (#117) [05:35:56] [02ImportDump] 07Universal-Omega deleted branch 03dependabot/composer/mediawiki/mediawiki-codesniffer-44.0.0 - 13https://github.com/miraheze/ImportDump [05:35:57] [02ImportDump] 07Universal-Omega deleted branch 03dependabot/composer/mediawiki/mediawiki-codesniffer-44.0.0 [05:37:20] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 25.39, 24.21, 23.58 [05:38:43] RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 2.52, 3.24, 3.17 [05:39:20] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 23.39, 23.24, 23.28 [05:39:58] RECOVERY - mwtask171 Puppet on mwtask171 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [05:40:05] RECOVERY - phorge171 Puppet on phorge171 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [05:40:21] PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 22.50, 18.52, 15.76 [05:40:26] PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 22.13, 18.75, 15.80 [05:40:28] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 20.46, 17.74, 15.18 [05:42:19] RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 14.95, 17.42, 15.71 [05:42:26] RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 15.08, 17.12, 15.56 [05:42:27] PROBLEM - ping6 on cp26 is WARNING: PING WARNING - Packet loss = 0%, RTA = 205.30 ms [05:42:28] RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 14.61, 16.38, 14.99 [05:45:23] miraheze/ImportDump - Universal-Omega the build passed. [05:46:31] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 4.08, 3.83, 3.32 [05:46:32] RECOVERY - ping6 on cp26 is OK: PING OK - Packet loss = 0%, RTA = 198.71 ms [05:50:32] RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 2.43, 3.16, 3.17 [05:54:31] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.01, 3.68, 3.44 [05:58:31] RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 2.27, 3.25, 3.35 [06:10:18] PROBLEM - os162 Current Load on os162 is CRITICAL: LOAD CRITICAL - total load average: 8.24, 7.71, 7.49 [06:10:53] PROBLEM - franchise.franchising.org.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - franchise.franchising.org.ua All nameservers failed to answer the query. [06:12:18] PROBLEM - os162 Current Load on os162 is WARNING: LOAD WARNING - total load average: 7.53, 7.62, 7.48 [06:12:24] PROBLEM - ns2 NTP time on ns2 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H [-4|-6] [-w ] [-c ] [-v verbose] [-o ] [06:14:10] PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 22.66, 18.50, 16.12 [06:14:22] RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset -0.001496165991 secs [06:15:28] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 23.83, 21.40, 19.24 [06:16:10] RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 16.06, 17.75, 16.16 [06:17:20] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.38, 22.17, 21.78 [06:17:24] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 18.92, 19.96, 18.94 [06:17:51] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 3.80, 4.44, 3.67 [06:21:20] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.02, 22.35, 22.03 [06:23:20] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.90, 23.43, 22.47 [06:23:36] RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 2.36, 3.14, 3.33 [06:25:20] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 20.39, 22.80, 22.40 [06:27:20] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 26.23, 24.09, 22.91 [06:31:20] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 19.68, 22.31, 22.49 [06:33:13] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 7.79, 5.01, 3.92 [06:37:02] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 1.81, 3.58, 3.58 [06:37:54] !log [reception@mwtask181] starting deploy of {'versions': '1.42', 'upgrade_extensions': 'ImportDump'} to all [06:37:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:38:07] !log [reception@mwtask181] finished deploy of {'versions': '1.42', 'upgrade_extensions': 'ImportDump'} to all - SUCCESS in 13s [06:38:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:38:57] RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 1.33, 3.03, 3.39 [06:40:53] PROBLEM - franchise.franchising.org.ua - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for franchise.franchising.org.ua could not be found [06:44:38] PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [06:44:51] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.94, 18.58, 16.95 [06:45:20] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 25.04, 22.62, 22.09 [06:45:40] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.70, 3.91, 3.70 [06:46:51] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 18.79, 18.44, 17.10 [06:47:20] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 18.13, 21.10, 21.62 [06:49:29] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 7.02, 4.50, 3.91 [06:51:25] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.34, 3.66, 3.65 [06:55:15] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 6.96, 4.79, 4.05 [06:59:20] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.50, 23.44, 22.63 [07:01:20] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 20.41, 22.39, 22.35 [07:02:56] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 1.79, 3.56, 3.93 [07:04:52] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 6.11, 4.58, 4.26 [07:05:20] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.96, 22.90, 22.50 [07:07:20] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.27, 22.53, 22.43 [07:08:41] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 0.40, 2.65, 3.58 [07:09:20] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.10, 23.15, 22.67 [07:10:36] RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 0.09, 1.85, 3.19 [07:11:20] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 20.45, 22.47, 22.52 [07:13:20] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 26.59, 24.04, 23.08 [07:19:20] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.69, 23.94, 23.41 [07:21:20] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.94, 24.37, 23.63 [07:25:20] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 16.93, 22.55, 23.22 [07:26:10] PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 22.87, 19.23, 16.34 [07:26:21] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 22.85, 19.27, 16.40 [07:26:51] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.90, 19.65, 18.12 [07:28:10] RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 15.69, 17.86, 16.20 [07:28:21] RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 16.10, 17.91, 16.26 [07:28:51] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 17.19, 19.27, 18.21 [07:36:06] PROBLEM - ping6 on cp26 is CRITICAL: PING CRITICAL - Packet loss = 0%, RTA = 325.43 ms [07:37:20] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 26.18, 23.71, 23.21 [07:41:20] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 23.92, 23.74, 23.33 [07:44:21] PROBLEM - ping6 on cp26 is WARNING: PING WARNING - Packet loss = 0%, RTA = 241.97 ms [07:50:30] PROBLEM - ping6 on cp26 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 275.07 ms [07:51:20] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.32, 23.81, 23.52 [07:52:32] RECOVERY - ping6 on cp26 is OK: PING OK - Packet loss = 0%, RTA = 180.55 ms [07:53:20] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 23.99, 23.73, 23.52 [07:56:43] PROBLEM - ping6 on cp26 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 272.90 ms [07:59:20] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.39, 23.23, 23.26 [08:00:49] PROBLEM - ping6 on cp26 is WARNING: PING WARNING - Packet loss = 0%, RTA = 274.21 ms [08:01:20] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.49, 23.33, 23.32 [08:02:53] PROBLEM - ping6 on cp26 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 274.05 ms [08:05:07] PROBLEM - cp26 Varnish Backends on cp26 is CRITICAL: 1 backends are down. mw181 [08:05:20] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.47, 23.62, 23.39 [08:06:51] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 23.91, 22.11, 19.28 [08:07:13] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 22.19, 18.56, 15.93 [08:07:20] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.02, 22.41, 22.93 [08:08:26] PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 21.05, 18.49, 15.84 [08:09:05] RECOVERY - cp26 Varnish Backends on cp26 is OK: All 19 backends are healthy [08:09:06] PROBLEM - ping6 on cp26 is WARNING: PING WARNING - Packet loss = 0%, RTA = 276.10 ms [08:09:11] RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 17.84, 18.11, 16.09 [08:10:26] RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 14.84, 17.18, 15.71 [08:11:09] RECOVERY - ping6 on cp26 is OK: PING OK - Packet loss = 0%, RTA = 174.00 ms [08:15:20] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.77, 23.28, 23.09 [08:15:50] RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [08:17:20] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 20.28, 22.18, 22.73 [08:18:51] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.27, 22.75, 20.85 [08:20:51] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.82, 22.57, 21.04 [08:21:20] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 25.57, 23.41, 23.09 [08:24:51] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 18.14, 20.03, 20.35 [08:25:20] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 23.03, 23.93, 23.43 [08:27:20] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 27.33, 25.10, 23.91 [08:28:51] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 23.81, 22.01, 21.09 [08:31:20] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 17.81, 22.79, 23.34 [08:32:51] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.26, 22.54, 21.47 [08:36:51] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.06, 23.31, 22.16 [08:43:20] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.48, 23.24, 23.29 [08:45:20] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.06, 22.66, 23.07 [08:46:42] PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [08:49:20] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 25.58, 24.06, 23.55 [08:50:51] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 25.14, 22.95, 21.85 [08:52:51] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 20.63, 22.62, 21.91 [08:53:20] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.88, 23.86, 23.63 [08:55:20] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 25.19, 24.29, 23.80 [08:57:20] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.39, 23.52, 23.58 [08:57:25] [Grafana] FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [09:05:20] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 26.78, 24.38, 23.78 [09:06:31] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 6.92, 3.69, 1.62 [09:06:32] PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call [09:06:51] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 25.31, 22.37, 21.51 [09:08:26] RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.072 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206 [09:08:51] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 20.78, 20.90, 21.03 [09:09:12] PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 22.43, 20.74, 18.16 [09:10:31] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.51, 3.64, 2.14 [09:11:07] RECOVERY - mw161 Current Load on mw161 is OK: LOAD OK - total load average: 16.60, 18.84, 17.75 [09:11:20] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.39, 22.77, 23.21 [09:12:31] RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 3.21, 3.38, 2.22 [09:13:20] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.19, 23.51, 23.44 [09:15:32] RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [09:16:51] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 25.32, 21.73, 21.21 [09:18:51] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 23.35, 22.21, 21.45 [09:19:20] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.52, 23.43, 23.53 [09:25:20] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.03, 23.10, 23.24 [09:27:06] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 3.82, 4.12, 3.15 [09:27:20] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 20.47, 22.81, 23.16 [09:30:56] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.03, 3.61, 3.16 [09:31:20] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.19, 23.36, 23.26 [09:33:20] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.42, 22.97, 23.15 [09:37:50] PROBLEM - ns2 NTP time on ns2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:38:36] RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 2.32, 2.88, 3.04 [09:39:48] RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset -0.0006105005741 secs [09:44:16] PROBLEM - ns2 NTP time on ns2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:45:20] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 26.03, 23.28, 22.87 [09:46:14] RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset 0.0001010894775 secs [09:46:17] PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_dns] [09:47:20] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.55, 23.00, 22.82 [09:48:33] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 4.50, 4.66, 3.80 [09:52:32] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.45, 3.55, 3.55 [09:54:33] RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 1.53, 2.96, 3.34 [09:55:20] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.75, 23.41, 22.88 [09:58:32] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 4.86, 3.85, 3.64 [09:58:51] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.16, 21.85, 21.53 [10:00:32] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.38, 3.41, 3.49 [10:02:51] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 20.28, 22.33, 21.93 [10:03:20] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.29, 23.99, 23.51 [10:04:32] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 4.29, 3.64, 3.57 [10:04:51] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.87, 23.45, 22.39 [10:05:20] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 25.29, 24.60, 23.79 [10:05:41] PROBLEM - ns2 NTP time on ns2 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H [-4|-6] [-w ] [-c ] [-v verbose] [-o ] [10:05:52] PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call [10:06:51] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 16.55, 21.52, 21.87 [10:07:39] RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset 0.0001163184643 secs [10:07:47] RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.228 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206 [10:09:20] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.26, 23.85, 23.72 [10:10:31] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.57, 3.94, 3.88 [10:12:31] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 4.29, 3.87, 3.84 [10:14:26] PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 21.15, 20.14, 18.15 [10:14:31] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.43, 3.38, 3.64 [10:14:51] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.39, 22.85, 22.22 [10:16:26] RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 18.98, 19.31, 18.06 [10:16:32] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 4.13, 3.27, 3.55 [10:16:51] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 20.51, 21.92, 21.97 [10:18:31] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.45, 2.97, 3.39 [10:19:20] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 25.83, 23.01, 23.00 [10:20:31] RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 2.33, 2.89, 3.32 [10:22:11] PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 22.42, 19.93, 18.02 [10:23:02] PROBLEM - sonicpedia.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'sonicpedia.org' expires in 15 day(s) (Sun 01 Sep 2024 10:19:43 AM GMT +0000). [10:23:15] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/627d3eb90437...539bf0bd9841 [10:23:18] [02ssl] 07WikiTideSSLBot 03539bf0b - Bot: Update SSL cert for sonicpedia.org [10:24:10] RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 17.12, 19.25, 18.04 [10:28:51] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 25.65, 24.21, 23.01 [10:30:10] PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 19.55, 20.53, 18.97 [10:30:51] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.31, 23.60, 22.96 [10:31:20] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.14, 23.82, 23.64 [10:32:10] RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 16.34, 19.12, 18.65 [10:33:20] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 23.85, 24.14, 23.79 [10:35:20] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 20.79, 22.81, 23.35 [10:36:28] PROBLEM - ns2 NTP time on ns2 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H [-4|-6] [-w ] [-c ] [-v verbose] [-o ] [10:37:20] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 26.78, 24.44, 23.88 [10:37:37] PROBLEM - wiki.susqu.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.susqu.org' expires in 15 day(s) (Sun 01 Sep 2024 10:34:23 AM GMT +0000). [10:37:49] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/539bf0bd9841...7bd2f0cca923 [10:37:52] [02ssl] 07WikiTideSSLBot 037bd2f0c - Bot: Update SSL cert for wiki.susqu.org [10:38:26] RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset 0.0002663731575 secs [10:38:48] PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 22.52, 21.23, 19.78 [10:39:43] PROBLEM - cp41 Varnish Backends on cp41 is CRITICAL: 1 backends are down. mw161 [10:40:29] PROBLEM - cp27 Varnish Backends on cp27 is CRITICAL: 1 backends are down. mw181 [10:40:39] PROBLEM - cp26 Varnish Backends on cp26 is CRITICAL: 1 backends are down. mw181 [10:40:44] RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 15.65, 19.42, 19.32 [10:40:54] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/improtDump.php --wiki=guerriersdelamontagnewiki --username-prefix=wikia:guerriersdelamontagne/fr /home/reception/frdesguerriersdelamontagne_pages_full.xml (END - exit=256) [10:41:11] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/importtDump.php --wiki=guerriersdelamontagnewiki --username-prefix=wikia:guerriersdelamontagne/fr /home/reception/frdesguerriersdelamontagne_pages_full.xml (END - exit=256) [10:41:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:41:20] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 23.23, 23.74, 23.76 [10:41:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:41:23] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/importDump.php --wiki=guerriersdelamontagnewiki --username-prefix=wikia:guerriersdelamontagne/fr /home/reception/frdesguerriersdelamontagne_pages_full.xml (START) [10:41:32] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 20.98, 20.64, 19.06 [10:41:38]