[00:01:37] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 1.83, 3.81, 3.81 [00:05:19] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.73, 3.50, 3.12 [00:05:37] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.60, 2.45, 3.24 [00:09:19] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.33, 2.73, 2.94 [00:14:52] RECOVERY - mw122 NTP time on mw122 is OK: NTP OK: Offset 0.0934638083 secs [00:18:29] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.28, 4.00, 3.62 [00:23:04] PROBLEM - test131 NTP time on test131 is CRITICAL: NTP CRITICAL: Offset 0.6715666056 secs [00:31:55] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 1.12, 3.03, 3.66 [00:32:17] PROBLEM - housing.wiki - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query housing.wiki. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [00:33:50] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 0.68, 2.28, 3.32 [00:37:13] RECOVERY - test131 NTP time on test131 is OK: NTP OK: Offset -0.001155406237 secs [01:02:05] RECOVERY - housing.wiki - reverse DNS on sslhost is OK: SSL OK - housing.wiki reverse DNS resolves to cp32.miraheze.org - CNAME FLAT [01:03:37] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.56, 2.90, 2.57 [01:05:37] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.75, 2.71, 2.53 [01:06:36] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 7.19, 4.69, 3.14 [01:22:24] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.11, 3.47, 2.99 [01:22:41] PROBLEM - cp32 SSH on cp32 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:23:04] PROBLEM - cp32 PowerDNS Recursor on cp32 is CRITICAL: CRITICAL - Plugin timed out while executing system call [01:23:35] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 108.175.15.182/cpweb, 2607:f1c0:1800:8100::1/cpweb [01:23:45] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 108.175.15.182/cpweb, 2607:f1c0:1800:8100::1/cpweb [01:23:53] PROBLEM - cp32 HTTPS on cp32 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:25:21] PROBLEM - cp32 Varnish Backends on cp32 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [01:25:21] PROBLEM - cp32 HTTP 4xx/5xx ERROR Rate on cp32 is CRITICAL: CRITICAL - NGINX Error Rate is 74% [01:25:22] PROBLEM - cp32 Puppet on cp32 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [01:25:48] RECOVERY - cp32 HTTPS on cp32 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3425 bytes in 0.947 second response time [01:27:22] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [01:27:41] PROBLEM - cp32 HTTP 4xx/5xx ERROR Rate on cp32 is WARNING: WARNING - NGINX Error Rate is 44% [01:28:06] RECOVERY - cp32 Varnish Backends on cp32 is OK: All 14 backends are healthy [01:29:50] RECOVERY - cp32 HTTP 4xx/5xx ERROR Rate on cp32 is OK: OK - NGINX Error Rate is 7% [01:30:26] RECOVERY - cp32 Puppet on cp32 is OK: OK: Puppet is currently enabled, last run 52 minutes ago with 0 failures [01:31:06] RECOVERY - cp32 SSH on cp32 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [01:31:23] RECOVERY - cp32 PowerDNS Recursor on cp32 is OK: DNS OK: 2.003 seconds response time. miraheze.org returns 108.175.15.182,2607:f1c0:1800:26f::1,2607:f1c0:1800:8100::1,74.208.203.152 [01:31:45] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [01:37:45] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.38, 3.62, 3.91 [01:43:37] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.95, 2.59, 3.34 [01:45:18] PROBLEM - graylog121 Current Load on graylog121 is WARNING: WARNING - load average: 3.55, 2.77, 1.91 [01:45:40] PROBLEM - cp22 NTP time on cp22 is WARNING: NTP WARNING: Offset 0.1088366807 secs [01:47:17] RECOVERY - graylog121 Current Load on graylog121 is OK: OK - load average: 2.52, 2.63, 1.96 [01:49:40] RECOVERY - cp22 NTP time on cp22 is OK: NTP OK: Offset 0.09586620331 secs [02:25:19] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 0.97, 2.03, 3.72 [02:25:37] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.48, 2.99, 2.63 [02:27:37] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.68, 2.96, 2.68 [02:29:19] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.26, 1.67, 3.19 [02:37:23] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 9.33, 5.57, 4.17 [02:45:19] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.76, 3.60, 3.91 [02:51:19] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 6.50, 3.64, 3.67 [02:53:19] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.76, 3.60, 3.65 [02:55:19] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.69, 2.89, 3.38 [03:01:11] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 13.88, 6.62, 2.95 [03:03:37] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.62, 3.21, 2.78 [03:05:08] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 0.56, 3.20, 2.37 [03:05:37] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.25, 2.77, 2.66 [03:07:19] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 7.08, 5.89, 4.40 [03:15:19] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.52, 3.27, 3.80 [03:19:19] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.66, 2.34, 3.29 [03:23:19] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 6.78, 5.82, 4.49 [03:27:19] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.56, 3.48, 3.82 [03:33:19] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.36, 2.57, 3.32 [03:37:19] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.79, 3.78, 3.60 [03:41:19] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.40, 3.55, 3.60 [03:43:19] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.27, 2.76, 3.30 [03:53:19] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.11, 3.50, 3.14 [03:55:37] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 6.04, 4.13, 2.83 [03:57:19] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.03, 3.91, 3.44 [03:57:37] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.92, 3.97, 2.95 [03:59:19] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.31, 3.30, 3.27 [03:59:37] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.05, 4.06, 3.11 [04:01:12] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 5.38, 4.67, 2.43 [04:01:37] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.44, 3.09, 2.86 [04:03:08] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.08, 3.24, 2.18 [04:09:19] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 8.93, 5.12, 3.82 [04:13:19] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.89, 3.77, 3.58 [04:14:54] PROBLEM - ns1 NTP time on ns1 is WARNING: NTP WARNING: Offset 0.1338487864 secs [04:19:19] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.30, 2.58, 3.16 [04:20:54] RECOVERY - ns1 NTP time on ns1 is OK: NTP OK: Offset 0.07931908965 secs [04:37:19] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 7.04, 5.87, 3.88 [04:45:19] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.85, 3.44, 3.70 [04:47:45] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 8.25, 4.13, 2.78 [04:53:19] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.49, 3.63, 3.63 [04:55:19] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.46, 2.96, 3.37 [05:01:37] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 1.24, 3.02, 3.58 [05:03:37] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.04, 2.35, 3.26 [05:24:24] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.98, 4.38, 3.45 [05:30:10] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.50, 3.83, 3.57 [05:32:05] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.22, 2.95, 3.28 [05:37:42] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.71, 3.30, 2.57 [05:39:37] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.31, 2.60, 2.41 [05:42:50] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.44, 9.88, 6.87 [05:44:50] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.22, 8.94, 6.91 [05:49:37] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 6.26, 5.11, 3.74 [05:55:37] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.31, 3.77, 3.62 [05:57:37] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.75, 3.03, 3.36 [06:01:21] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 24.30, 9.90, 4.14 [06:05:37] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.00, 4.03, 3.58 [06:07:37] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.44, 3.02, 3.26 [06:13:08] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 0.46, 3.05, 3.63 [06:15:08] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.13, 2.42, 3.33 [06:21:37] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 6.83, 6.55, 4.45 [06:22:43] PROBLEM - cloud13 Puppet on cloud13 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [06:24:11] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 6.66, 4.92, 3.37 [06:31:53] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.53, 3.91, 3.78 [06:33:40] PROBLEM - cp22 NTP time on cp22 is WARNING: NTP WARNING: Offset 0.1156219542 secs [06:33:48] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 6.10, 4.44, 3.96 [06:41:37] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 1.54, 2.69, 3.77 [06:41:40] RECOVERY - cp22 NTP time on cp22 is OK: NTP OK: Offset 0.09045353532 secs [06:45:22] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 0.72, 2.42, 3.69 [06:47:38] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.62, 3.04, 3.50 [06:49:19] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.82, 2.00, 3.21 [06:49:37] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.48, 3.35, 3.58 [06:50:43] RECOVERY - cloud13 Puppet on cloud13 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [06:53:20] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.57, 3.64, 3.61 [06:53:37] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.16, 3.73, 3.66 [06:55:19] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.13, 3.72, 3.66 [06:59:19] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.95, 2.70, 3.25 [06:59:37] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.72, 3.84, 3.84 [07:03:37] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.27, 2.60, 3.34 [07:06:02] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 8.82, 5.76, 3.17 [07:09:19] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 8.26, 6.55, 4.57 [07:09:53] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 0.63, 2.99, 2.62 [07:21:40] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 9.70, 5.31, 3.82 [07:29:19] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.18, 2.78, 3.96 [07:33:19] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.10, 2.13, 3.40 [07:33:37] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 1.22, 3.15, 3.78 [07:35:37] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 0.61, 2.27, 3.39 [07:37:25] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 16.08, 7.44, 5.13 [07:42:50] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.21, 9.55, 7.01 [07:44:50] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.43, 9.29, 7.24 [07:47:19] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.34, 2.56, 3.74 [07:49:19] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 0.94, 1.95, 3.37 [08:07:19] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.36, 3.63, 3.05 [08:09:19] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.07, 3.51, 3.08 [08:11:19] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.35, 2.71, 2.83 [08:39:19] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.96, 3.77, 2.83 [08:41:19] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.09, 3.23, 2.75 [08:54:24] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.50, 2.75, 1.99 [08:56:19] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 3.18, 2.86, 2.12 [09:00:38] PROBLEM - cp33 NTP time on cp33 is WARNING: NTP WARNING: Offset 0.2231641412 secs [09:01:19] !log [universalomega@mwtask141] Upload wikibackup from December 2021 for hkrailwiki to Swift miraheze-hkrailwiki-dumps-backup then delete once user downloaded it. [09:01:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:01:34] !log [universalomega@mwtask141] 2022 [09:01:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:07:21] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 13.42, 6.68, 3.70 [09:11:43] !log [universalomega@db142] DROP hkrailwiki [09:11:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:12:15] !log [universalomega@puppet141] sudo -u www-data rm /srv/mediawiki/cache/*hkrailwiki* on mw*, test131, and mwtask141 [09:12:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:12:23] !log [universalomega@db101] MariaDB [mhglobal]> DELETE FROM cw_wikis WHERE wiki_dbname='hkrailwiki'; [09:12:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:12:28] !log [universalomega@db101] MariaDB [mhglobal]> DELETE FROM mw_settings WHERE s_dbname='hkrailwiki'; [09:12:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:12:34] !log [universalomega@db101] MariaDB [mhglobal]> DELETE FROM mw_namespaces WHERE ns_dbname='hkrailwiki'; [09:12:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:12:40] !log [universalomega@db101] MariaDB [mhglobal]> DELETE FROM mw_permissions WHERE perm_dbname='hkrailwiki'; [09:12:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:12:45] !log [universalomega@db101] MariaDB [mhglobal]> DELETE FROM localnames WHERE ln_wiki='hkrailwiki'; [09:12:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:12:50] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 15.71, 11.06, 7.89 [09:12:50] !log [universalomega@db101] MariaDB [mhglobal]> DELETE FROM localuser WHERE lu_wiki='hkrailwiki'; [09:12:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:12:58] !log [universalomega@db101] MariaDB [metawiki]> DELETE FROM echo_unread_wikis WHERE euw_wiki='hkrailwiki'; [09:13:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:14:07] !log [universalomega@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildImages.php --wiki=hkrailwiki --missing (START) [09:14:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:14:50] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.15, 9.74, 7.80 [09:17:04] !log [universalomega@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=hkrailwiki /home/reception/wikibackups1812/hkrailwiki.xml (START) [09:17:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:21:19] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.01, 2.79, 3.75 [09:23:19] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 0.68, 2.14, 3.39 [09:26:06] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 6.74, 4.38, 2.57 [09:30:38] RECOVERY - cp33 NTP time on cp33 is OK: NTP OK: Offset 0.02885067463 secs [09:35:07] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.61, 8.83, 7.45 [09:35:22] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 6.79, 4.14, 3.28 [09:37:07] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.12, 7.48, 7.11 [09:39:37] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 0.84, 3.52, 3.66 [09:41:37] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 0.54, 2.57, 3.30 [09:50:35] [Grafana] !sre FIRING: The mediawiki job queue has more than 2500 unclaimed jobs https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [09:51:19] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.44, 2.69, 3.79 [09:55:19] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 0.75, 1.73, 3.15 [09:55:37] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.41, 3.24, 2.80 [09:57:37] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.67, 2.71, 2.66 [10:21:37] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.20, 3.43, 2.36 [10:27:20] !log [universalomega@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildImages.php --wiki=hkrailwiki --missing (END - exit=0) [10:27:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:29:37] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.50, 4.00, 3.12 [10:31:37] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.49, 2.99, 2.85 [10:34:09] !log [universalomega@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=hkrailwiki /home/reception/wikibackups1812/hkrailwiki.xml (END - exit=0) [10:34:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:35:50] PROBLEM - otogebase.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for otogebase.com could not be found [10:37:53] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 6.70, 4.40, 2.88 [10:41:44] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.83, 3.53, 2.87 [10:43:39] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 0.89, 2.59, 2.60 [10:58:33] PROBLEM - otogebase.com - LetsEncrypt on sslhost is CRITICAL: connect to address otogebase.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [10:59:08] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 5.32, 3.33, 1.97 [11:10:12] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.45, 3.68, 2.56 [11:12:07] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.67, 3.87, 2.78 [11:13:08] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.30, 3.02, 3.28 [11:14:02] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 0.97, 2.77, 2.50 [11:42:52] PROBLEM - mw141 MediaWiki Rendering on mw141 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 5796 bytes in 0.010 second response time [11:43:24] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:44:24] PROBLEM - db121 Current Load on db121 is CRITICAL: CRITICAL - load average: 45.91, 55.29, 25.79 [11:44:50] RECOVERY - mw141 MediaWiki Rendering on mw141 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.438 second response time [11:45:20] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.278 second response time [11:45:27] PROBLEM - cp22 Varnish Backends on cp22 is CRITICAL: 1 backends are down. mw121 [11:45:37] PROBLEM - cp33 Varnish Backends on cp33 is CRITICAL: 3 backends are down. mw121 mw132 mw141 [11:45:47] PROBLEM - cp23 Varnish Backends on cp23 is CRITICAL: 3 backends are down. mw122 mw131 mw141 [11:46:16] PROBLEM - cp32 Varnish Backends on cp32 is CRITICAL: 5 backends are down. mw121 mw131 mw132 mw141 mw142 [11:47:07] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 16.37, 10.15, 7.02 [11:47:27] RECOVERY - cp22 Varnish Backends on cp22 is OK: All 14 backends are healthy [11:47:37] RECOVERY - cp33 Varnish Backends on cp33 is OK: All 14 backends are healthy [11:47:47] RECOVERY - cp23 Varnish Backends on cp23 is OK: All 14 backends are healthy [11:48:16] RECOVERY - cp32 Varnish Backends on cp32 is OK: All 14 backends are healthy [11:49:07] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 5.50, 8.34, 6.76 [11:50:35] [Grafana] !sre RESOLVED: High Job Queue Backlog https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [12:05:24] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 0.68, 1.44, 7.15 [12:07:24] RECOVERY - db121 Current Load on db121 is OK: OK - load average: 0.73, 1.21, 6.37 [13:22:50] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.85, 9.29, 7.79 [13:24:50] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.24, 10.23, 8.34 [13:25:48] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.69, 3.18, 1.99 [13:26:50] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.48, 8.78, 8.04 [13:27:44] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 3.18, 3.08, 2.09 [13:31:37] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.44, 4.09, 2.75 [13:33:37] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.81, 3.34, 2.65 [13:35:19] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.69, 3.14, 2.52 [13:37:19] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 6.31, 4.29, 3.00 [13:39:19] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.78, 3.54, 2.88 [13:41:19] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 3.00, 3.39, 2.90 [13:45:40] PROBLEM - cp22 NTP time on cp22 is WARNING: NTP WARNING: Offset 0.363907665 secs [14:05:40] RECOVERY - cp22 NTP time on cp22 is OK: NTP OK: Offset -0.02452746034 secs [14:07:19] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 6.00, 4.31, 2.87 [14:12:50] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 19.78, 12.86, 9.29 [14:15:19] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.57, 3.46, 3.31 [14:15:59] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.02, 10.48, 8.66 [14:17:19] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.88, 2.83, 3.09 [14:17:54] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.92, 9.20, 8.41 [14:18:50] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 6.87, 10.88, 9.90 [14:20:50] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.04, 9.91, 9.68 [14:27:37] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.14, 3.27, 2.75 [14:29:37] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.64, 2.74, 2.63 [15:29:08] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 4.96, 3.58, 2.11 [15:38:06] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.01, 3.72, 2.74 [15:40:02] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.93, 3.11, 2.64 [15:43:40] PROBLEM - cp22 NTP time on cp22 is WARNING: NTP WARNING: Offset 0.1672259271 secs [15:55:08] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 1.09, 1.88, 3.94 [16:03:08] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.03, 2.08, 3.36 [16:09:40] RECOVERY - cp22 NTP time on cp22 is OK: NTP OK: Offset 0.06884148717 secs [16:11:08] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 6.59, 7.32, 4.51 [16:14:58] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.41, 3.99, 3.75 [16:16:54] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.06, 2.96, 3.39 [16:26:23] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 4.00, 3.45, 2.60 [16:28:18] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.77, 3.89, 2.86 [16:30:14] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.78, 3.26, 2.77 [16:32:38] PROBLEM - cp33 NTP time on cp33 is WARNING: NTP WARNING: Offset 0.2134219408 secs [16:39:59] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 7.67, 5.98, 3.93 [16:42:38] RECOVERY - cp33 NTP time on cp33 is OK: NTP OK: Offset -0.0220824182 secs [16:45:44] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.25, 3.44, 3.49 [16:47:39] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 0.87, 2.60, 3.17 [16:52:20] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.90, 3.43, 2.77 [16:54:16] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 8.75, 4.99, 3.39 [16:55:24] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 6.83, 7.63, 5.08 [16:58:06] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.71, 3.82, 3.29 [17:00:01] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.43, 3.25, 3.14 [17:03:19] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.27, 3.19, 3.86 [17:05:19] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.76, 3.62, 3.93 [17:20:49] PROBLEM - wiki.zulaclub.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.zulaclub.net All nameservers failed to answer the query. [17:27:19] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.50, 2.77, 3.96 [17:27:52] !log take db121 out of read only [17:27:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:31:19] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.43, 1.88, 3.30 [17:38:11] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.54, 3.54, 3.52 [17:40:06] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.57, 3.25, 3.43 [17:42:01] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.41, 2.61, 3.18 [17:50:08] PROBLEM - wiki.zulaclub.net - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query zulaclub.net. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [17:59:37] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.84, 3.98, 2.92 [18:01:37] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.89, 3.23, 2.78 [18:09:19] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.61, 3.07, 2.26 [18:09:40] PROBLEM - cp22 NTP time on cp22 is WARNING: NTP WARNING: Offset 0.2261553109 secs [18:11:19] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 13.73, 7.61, 4.05 [18:17:19] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.52, 3.14, 3.13 [18:19:29] RECOVERY - wiki.zulaclub.net - reverse DNS on sslhost is OK: SSL OK - wiki.zulaclub.net reverse DNS resolves to cp23.miraheze.org - CNAME OK [18:26:16] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.62, 3.87, 3.10 [18:26:54] !log [universalomega@mwtask141] INSERT INTO content_models (model_id, model_name) VALUES ( 5, 'json'); on hkrailwiki [18:26:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:29:48] !log [universalomega@mwtask141] sudo -u www-data php /srv/mediawiki/w/extensions/MirahezeMagic/maintenance/assignImportedEdits.php --wiki=hkrailwiki (END - exit=0) [18:29:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:30:06] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.42, 3.06, 2.98 [18:31:08] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 5.06, 3.72, 2.20 [18:32:53] !log [universalomega@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=hkrailwiki (START) [18:32:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:33:19] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.13, 3.65, 3.06 [18:37:40] RECOVERY - cp22 NTP time on cp22 is OK: NTP OK: Offset 0.09617963433 secs [18:39:19] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.06, 3.45, 3.22 [18:41:19] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.21, 3.09, 3.12 [18:49:08] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 0.79, 2.28, 3.85 [18:49:25] !log [universalomega@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=hkrailwiki (END - exit=0) [18:49:26] !log [universalomega@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildImages.php --wiki=hkrailwiki --missing (START) [18:49:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:49:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:49:37] !log [universalomega@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildImages.php --wiki=hkrailwiki --missing (END - exit=0) [18:49:38] !log [universalomega@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/initSiteStats.php --wiki=hkrailwiki --active --update (END - exit=0) [18:49:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:49:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:53:08] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 0.87, 1.52, 3.18 [19:06:34] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.61, 3.45, 2.79 [19:09:40] PROBLEM - cp22 NTP time on cp22 is WARNING: NTP WARNING: Offset -0.1117898822 secs [19:16:10] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.83, 3.99, 3.89 [19:19:40] RECOVERY - cp22 NTP time on cp22 is OK: NTP OK: Offset -0.08446493745 secs [19:20:00] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.19, 2.49, 3.28 [19:47:40] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.88, 3.18, 2.52 [19:49:37] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.82, 2.83, 2.46 [20:11:19] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.34, 3.80, 2.84 [20:13:19] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.19, 3.34, 2.79 [20:23:37] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 6.18, 3.86, 2.44 [20:25:37] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.93, 3.41, 2.45 [20:27:37] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.47, 3.83, 2.72 [20:29:08] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 4.70, 2.75, 1.65 [20:29:37] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.44, 3.48, 2.72 [20:31:08] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 2.24, 2.54, 1.72 [20:31:37] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.19, 2.62, 2.50 [20:35:08] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 4.75, 4.50, 2.77 [20:37:19] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 6.52, 3.77, 2.72 [20:41:08] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 1.60, 3.97, 3.26 [20:43:08] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 0.94, 2.93, 2.96 [20:47:19] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 0.88, 3.22, 3.36 [20:54:31] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 8.44, 5.20, 3.54 [21:04:07] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.17, 3.96, 3.71 [21:07:20] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.76, 3.53, 2.88 [21:07:58] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.59, 2.91, 3.31 [21:09:19] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.99, 2.91, 2.73 [21:18:39] PROBLEM - cp33 NTP time on cp33 is WARNING: NTP WARNING: Offset 0.2226652205 secs [21:25:37] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 6.08, 5.22, 3.86 [21:31:37] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.82, 3.18, 3.39 [21:35:37] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.96, 3.39, 3.43 [21:36:38] RECOVERY - cp33 NTP time on cp33 is OK: NTP OK: Offset -0.02097216249 secs [21:37:37] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.04, 2.86, 3.22 [21:38:47] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 9.50, 5.44, 3.85 [21:55:39] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 11.08, 4.97, 3.56 [21:56:05] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.74, 2.52, 3.74 [21:58:00] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.17, 3.00, 3.76 [22:03:37] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.31, 3.96, 3.72 [22:07:37] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.72, 2.95, 3.38 [22:15:37] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.58, 3.71, 3.66 [22:17:37] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.29, 2.82, 3.33 [22:24:31] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 16.47, 7.02, 4.70 [22:34:08] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 1.50, 3.13, 3.94 [22:36:03] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.16, 3.73, 4.04 [22:37:59] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.19, 3.16, 3.80 [22:45:40] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.55, 2.29, 3.18 [22:55:37] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.70, 3.81, 3.43 [22:57:37] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.94, 3.01, 3.17 [23:29:08] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 3.61, 2.43, 1.46 [23:31:08] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 5.14, 3.59, 2.02 [23:31:19] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.69, 2.58, 3.99 [23:35:19] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.72, 3.41, 3.98 [23:37:19] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.33, 2.86, 3.71 [23:39:19] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.78, 3.75, 3.91 [23:41:19] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.74, 3.58, 3.82 [23:47:19] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.05, 2.52, 3.26 [23:49:08] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 0.49, 1.89, 3.94 [23:53:08] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 0.69, 1.23, 3.19 [23:56:29] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.22, 4.06, 3.27