[00:01:40] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 25.39, 21.64, 16.80 [00:03:37] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 23.74, 22.18, 17.56 [00:05:34] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 18.28, 20.31, 17.42 [00:05:44] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 18.00, 23.13, 21.69 [00:11:44] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 26.10, 23.72, 22.30 [00:13:44] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 16.69, 21.46, 21.68 [00:15:44] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 11.62, 17.95, 20.36 [00:42:46] PROBLEM - cp51 Varnish Backends on cp51 is CRITICAL: 1 backends are down. mw152 [00:46:46] RECOVERY - cp51 Varnish Backends on cp51 is OK: All 19 backends are healthy [01:09:44] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 25.14, 20.17, 18.16 [01:11:44] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 19.34, 19.95, 18.34 [01:15:44] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 29.01, 23.92, 20.26 [01:19:44] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.07, 23.38, 20.92 [01:21:44] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.31, 23.35, 21.19 [01:23:44] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.83, 23.39, 21.48 [01:25:44] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.20, 23.95, 21.94 [01:29:44] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.99, 23.78, 22.36 [01:39:44] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 15.45, 18.07, 19.96 [01:47:44] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 23.65, 21.08, 20.58 [01:51:44] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 29.46, 24.58, 22.01 [01:53:44] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 19.94, 22.85, 21.71 [01:57:44] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 25.99, 23.04, 21.92 [01:59:44] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 16.02, 21.41, 21.54 [02:00:13] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 109.123.230.163/cpweb [02:01:44] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 8.84, 17.12, 19.96 [02:02:13] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [02:02:25] [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [02:34:32] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 23.57, 20.56, 18.27 [02:36:27] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 18.84, 20.02, 18.36 [02:55:44] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.84, 20.59, 17.78 [02:57:44] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 16.04, 19.30, 17.69 [03:09:44] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 25.39, 20.79, 18.52 [03:11:44] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 23.14, 21.28, 18.96 [03:13:44] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 26.13, 22.99, 19.87 [03:19:44] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 17.00, 22.30, 20.97 [03:25:44] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 15.43, 19.46, 20.31 [04:15:44] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.23, 18.61, 17.73 [04:17:44] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 19.16, 19.35, 18.15 [04:21:44] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.23, 21.84, 19.42 [04:25:44] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.53, 23.00, 20.48 [04:27:44] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 28.81, 25.06, 21.53 [05:05:27] PROBLEM - worldtriggerwiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'worldtriggerwiki.com' expires in 15 day(s) (Wed 17 Jul 2024 04:45:38 AM GMT +0000). [05:09:44] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 17.93, 21.62, 23.80 [05:27:44] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 26.40, 23.06, 22.45 [05:51:44] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 19.94, 22.18, 23.46 [05:53:44] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.56, 22.84, 23.51 [05:55:44] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.25, 21.95, 23.10 [06:02:25] [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [06:03:44] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 27.22, 23.98, 23.54 [06:55:44] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 17.17, 21.00, 23.55 [07:01:44] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.29, 20.50, 22.38 [07:03:44] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 17.04, 19.33, 21.75 [07:07:44] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 13.01, 15.81, 19.74 [07:17:44] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 29.49, 21.98, 19.82 [07:21:44] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 20.97, 22.46, 20.63 [07:31:44] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 17.69, 19.87, 20.29 [07:44:12] PROBLEM - franchise.franchising.org.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - franchise.franchising.org.ua All nameservers failed to answer the query. [07:46:18] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 28.87, 23.32, 20.63 [07:54:40] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 23.75, 20.39, 16.30 [07:56:40] RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 13.81, 17.89, 15.89 [08:03:44] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.47, 22.89, 23.05 [08:12:56] RECOVERY - franchise.franchising.org.ua - reverse DNS on sslhost is OK: SSL OK - franchise.franchising.org.ua reverse DNS resolves to cp36.wikitide.net - CNAME OK [08:23:28] PROBLEM - moddedminecraft.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'moddedminecraft.wiki' expires in 15 day(s) (Wed 17 Jul 2024 08:13:30 AM GMT +0000). [08:23:39] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/488d8577791c...9fda12d657b3 [08:23:42] [02ssl] 07WikiTideSSLBot 039fda12d - Bot: Update SSL cert for moddedminecraft.wiki [08:23:44] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 12.58, 16.43, 19.90 [08:28:50] PROBLEM - puppet181 APT on puppet181 is CRITICAL: APT CRITICAL: 67 packages available for upgrade (3 critical updates). [08:29:10] PROBLEM - swiftobject161 APT on swiftobject161 is CRITICAL: APT CRITICAL: 58 packages available for upgrade (3 critical updates). [08:29:14] PROBLEM - phorge171 APT on phorge171 is CRITICAL: APT CRITICAL: 60 packages available for upgrade (3 critical updates). [08:29:38] PROBLEM - os151 APT on os151 is CRITICAL: APT CRITICAL: 60 packages available for upgrade (3 critical updates). [08:29:44] PROBLEM - cloud18 APT on cloud18 is CRITICAL: APT CRITICAL: 108 packages available for upgrade (4 critical updates). [08:30:23] PROBLEM - mw161 APT on mw161 is CRITICAL: APT CRITICAL: 71 packages available for upgrade (3 critical updates). [08:30:29] PROBLEM - mwtask171 APT on mwtask171 is CRITICAL: APT CRITICAL: 69 packages available for upgrade (3 critical updates). [08:30:29] PROBLEM - db151 APT on db151 is CRITICAL: APT CRITICAL: 66 packages available for upgrade (3 critical updates). [08:30:48] RECOVERY - puppet181 APT on puppet181 is OK: APT OK: 62 packages available for upgrade (0 critical updates). [08:31:06] PROBLEM - os162 APT on os162 is CRITICAL: APT CRITICAL: 60 packages available for upgrade (3 critical updates). [08:31:07] PROBLEM - mw181 APT on mw181 is CRITICAL: APT CRITICAL: 72 packages available for upgrade (3 critical updates). [08:32:36] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 26.78, 22.83, 21.27 [08:32:42] PROBLEM - matomo151 APT on matomo151 is CRITICAL: APT CRITICAL: 69 packages available for upgrade (3 critical updates). [08:33:27] PROBLEM - mw172 APT on mw172 is CRITICAL: APT CRITICAL: 71 packages available for upgrade (3 critical updates). [08:33:36] RECOVERY - os151 APT on os151 is OK: APT OK: 55 packages available for upgrade (0 critical updates). [08:34:04] PROBLEM - mw171 APT on mw171 is CRITICAL: APT CRITICAL: 71 packages available for upgrade (3 critical updates). [08:34:32] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 15.72, 20.54, 20.64 [08:34:32] PROBLEM - mon181 APT on mon181 is CRITICAL: APT CRITICAL: 64 packages available for upgrade (3 critical updates). [08:34:35] PROBLEM - cp27 APT on cp27 is CRITICAL: APT CRITICAL: 58 packages available for upgrade (3 critical updates). [08:34:43] PROBLEM - swiftobject171 APT on swiftobject171 is CRITICAL: APT CRITICAL: 58 packages available for upgrade (3 critical updates). [08:35:07] PROBLEM - cloud17 APT on cloud17 is CRITICAL: APT CRITICAL: 107 packages available for upgrade (4 critical updates). [08:35:30] PROBLEM - swiftobject151 APT on swiftobject151 is CRITICAL: APT CRITICAL: 58 packages available for upgrade (3 critical updates). [08:36:26] PROBLEM - bots171 APT on bots171 is CRITICAL: APT CRITICAL: 69 packages available for upgrade (3 critical updates). [08:36:27] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 17.98, 19.84, 20.38 [08:36:34] RECOVERY - cp27 APT on cp27 is OK: APT OK: 53 packages available for upgrade (0 critical updates). [08:37:17] PROBLEM - swiftac171 APT on swiftac171 is CRITICAL: APT CRITICAL: 47 packages available for upgrade (3 critical updates). [08:37:18] PROBLEM - prometheus151 APT on prometheus151 is CRITICAL: APT CRITICAL: 58 packages available for upgrade (3 critical updates). [08:37:44] PROBLEM - cp26 APT on cp26 is CRITICAL: APT CRITICAL: 58 packages available for upgrade (3 critical updates). [08:37:56] PROBLEM - cp36 APT on cp36 is CRITICAL: APT CRITICAL: 58 packages available for upgrade (3 critical updates). [08:38:28] RECOVERY - mwtask171 APT on mwtask171 is OK: APT OK: 64 packages available for upgrade (0 critical updates). [08:39:06] RECOVERY - os162 APT on os162 is OK: APT OK: 55 packages available for upgrade (0 critical updates). [08:39:11] RECOVERY - phorge171 APT on phorge171 is OK: APT OK: 55 packages available for upgrade (0 critical updates). [08:39:36] PROBLEM - mw152 APT on mw152 is CRITICAL: APT CRITICAL: 71 packages available for upgrade (3 critical updates). [08:39:41] RECOVERY - cloud18 APT on cloud18 is OK: APT OK: 102 packages available for upgrade (0 critical updates). [08:40:13] PROBLEM - kafka181 APT on kafka181 is CRITICAL: APT CRITICAL: 44 packages available for upgrade (3 critical updates). [08:40:25] RECOVERY - mw161 APT on mw161 is OK: APT OK: 66 packages available for upgrade (0 critical updates). [08:41:08] PROBLEM - graphite151 APT on graphite151 is CRITICAL: APT CRITICAL: 46 packages available for upgrade (3 critical updates). [08:41:09] PROBLEM - changeprop151 APT on changeprop151 is CRITICAL: APT CRITICAL: 52 packages available for upgrade (3 critical updates). [08:41:40] PROBLEM - test151 APT on test151 is CRITICAL: APT CRITICAL: 81 packages available for upgrade (3 critical updates). [08:42:14] RECOVERY - kafka181 APT on kafka181 is OK: APT OK: 39 packages available for upgrade (0 critical updates). [08:42:43] RECOVERY - swiftobject171 APT on swiftobject171 is OK: APT OK: 53 packages available for upgrade (0 critical updates). [08:42:58] PROBLEM - ns1 APT on ns1 is CRITICAL: APT CRITICAL: 57 packages available for upgrade (3 critical updates). [08:45:07] RECOVERY - cloud17 APT on cloud17 is OK: APT OK: 101 packages available for upgrade (0 critical updates). [08:46:03] RECOVERY - mw171 APT on mw171 is OK: APT OK: 66 packages available for upgrade (0 critical updates). [08:47:18] RECOVERY - prometheus151 APT on prometheus151 is OK: APT OK: 53 packages available for upgrade (0 critical updates). [08:47:27] RECOVERY - mw172 APT on mw172 is OK: APT OK: 66 packages available for upgrade (0 critical updates). [08:47:30] RECOVERY - swiftobject151 APT on swiftobject151 is OK: APT OK: 53 packages available for upgrade (0 critical updates). [08:48:32] RECOVERY - mon181 APT on mon181 is OK: APT OK: 59 packages available for upgrade (0 critical updates). [08:49:30] PROBLEM - db151 Current Load on db151 is CRITICAL: LOAD CRITICAL - total load average: 57.08, 26.75, 10.88 [08:49:56] RECOVERY - cp36 APT on cp36 is OK: APT OK: 53 packages available for upgrade (0 critical updates). [08:50:31] PROBLEM - db151 PowerDNS Recursor on db151 is CRITICAL: CRITICAL - Plugin timed out while executing system call [08:51:19] RECOVERY - changeprop151 APT on changeprop151 is OK: APT OK: 46 packages available for upgrade (0 critical updates). [08:52:26] RECOVERY - moddedminecraft.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'moddedminecraft.wiki' will expire on Sun 29 Sep 2024 07:23:33 AM GMT +0000. [08:52:27] PROBLEM - db151 Puppet on db151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [08:52:57] RECOVERY - ns1 APT on ns1 is OK: APT OK: 52 packages available for upgrade (0 critical updates). [08:53:34] RECOVERY - mw152 APT on mw152 is OK: APT OK: 66 packages available for upgrade (0 critical updates). [08:53:35] RECOVERY - test151 APT on test151 is OK: APT OK: 76 packages available for upgrade (0 critical updates). [08:53:40] RECOVERY - cp26 APT on cp26 is OK: APT OK: 53 packages available for upgrade (0 critical updates). [08:54:26] RECOVERY - bots171 APT on bots171 is OK: APT OK: 64 packages available for upgrade (0 critical updates). [08:55:09] RECOVERY - swiftobject161 APT on swiftobject161 is OK: APT OK: 53 packages available for upgrade (0 critical updates). [08:55:16] RECOVERY - swiftac171 APT on swiftac171 is OK: APT OK: 42 packages available for upgrade (0 critical updates). [08:55:28] PROBLEM - db151 SSH on db151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:55:33] RECOVERY - mw181 APT on mw181 is OK: APT OK: 67 packages available for upgrade (0 critical updates). [08:56:42] RECOVERY - matomo151 APT on matomo151 is OK: APT OK: 64 packages available for upgrade (0 critical updates). [08:56:45] RECOVERY - db151 PowerDNS Recursor on db151 is OK: DNS OK: 0.048 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206 [08:57:08] RECOVERY - graphite151 APT on graphite151 is OK: APT OK: 41 packages available for upgrade (0 critical updates). [08:57:26] RECOVERY - db151 SSH on db151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0) [08:57:27] RECOVERY - db151 Puppet on db151 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [08:58:37] RECOVERY - db151 APT on db151 is OK: APT OK: 61 packages available for upgrade (0 critical updates). [09:07:25] [Grafana] !tech RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [09:21:30] PROBLEM - db151 Current Load on db151 is WARNING: LOAD WARNING - total load average: 0.27, 1.26, 11.16 [09:23:30] RECOVERY - db151 Current Load on db151 is OK: LOAD OK - total load average: 0.45, 1.01, 9.86 [09:33:44] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.18, 21.24, 16.81 [09:35:44] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 14.47, 18.80, 16.46 [10:01:48] !log [alex@ldap171] changed my LDAP password [10:01:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:06:27] !log [alex@ldap171] changed my LDAP password again [10:06:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:09:38] PROBLEM - wiki.alathramc.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.alathramc.com All nameservers failed to answer the query. [10:11:49] PROBLEM - wiki.mostly.vet - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mostly.vet All nameservers failed to answer the query. [10:14:03] PROBLEM - archive.stellurgists.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - archive.stellurgists.wiki All nameservers failed to answer the query. [10:15:30] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.26, 21.79, 18.38 [10:23:12] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.75, 23.40, 20.31 [10:25:08] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 18.14, 21.18, 19.85 [10:27:03] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 18.69, 20.28, 19.67 [10:32:25] [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [10:33:22] !log [alex@ldap171] restarted slapd [10:33:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:39:38] PROBLEM - wiki.alathramc.com - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['damon.ns.cloudflare.com.', 'gigi.ns.cloudflare.com.'], 'CNAME': 'wiki.alathra.com.'} [10:41:09] RECOVERY - wiki.mostly.vet - reverse DNS on sslhost is OK: SSL OK - wiki.mostly.vet reverse DNS resolves to cp36.wikitide.net - CNAME OK [10:42:36] RECOVERY - archive.stellurgists.wiki - reverse DNS on sslhost is OK: SSL OK - archive.stellurgists.wiki reverse DNS resolves to cp36.wikitide.net - CNAME OK [11:35:44] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.19, 18.82, 15.31 [11:37:44] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 17.77, 18.13, 15.47 [12:04:29] !log [oa@mwtask181] starting deploy of {'l10n': True, 'versions': '1.41', 'upgrade_extensions': 'Description2'} to all [12:04:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:06:08] !log [oa@mwtask181] finished deploy of {'l10n': True, 'versions': '1.41', 'upgrade_extensions': 'Description2'} to all - SUCCESS in 100s [12:06:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:09:19] [02mw-config] 07OAuthority commented on pull request 03#5597: global-sysop -> global-admin - 13https://github.com/miraheze/mw-config/pull/5597#issuecomment-2199976976 [12:12:31] [02GlobalNewFiles] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/GlobalNewFiles/compare/ab8f05b4b752...5067b3cc233c [12:12:33] [02GlobalNewFiles] 07translatewiki 035067b3c - Localisation updates from https://translatewiki.net. [12:16:55] miraheze/GlobalNewFiles - translatewiki the build passed. [12:27:44] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 23.60, 19.92, 16.92 [12:29:44] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 15.56, 18.19, 16.65 [12:57:44] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 28.32, 19.82, 16.91 [12:59:44] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 20.08, 19.88, 17.31 [13:17:42] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 28.27, 18.52, 12.58 [13:18:09] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 29.54, 22.52, 18.83 [13:19:42] RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 19.82, 19.46, 13.71 [13:20:05] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 23.78, 22.93, 19.43 [13:22:00] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 25.32, 23.92, 20.23 [13:23:55] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 18.74, 22.02, 19.97 [13:25:51] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 25.14, 22.85, 20.48 [13:29:44] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 20.53, 22.76, 21.05 [13:33:44] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 25.53, 23.28, 21.55 [13:37:44] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 15.02, 21.63, 21.54 [13:37:48] PROBLEM - neurologiahpm.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'neurologiahpm.com' expires in 15 day(s) (Wed 17 Jul 2024 01:06:13 PM GMT +0000). [13:38:00] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/9fda12d657b3...33699ab1c992 [13:38:01] [02ssl] 07WikiTideSSLBot 0333699ab - Bot: Update SSL cert for neurologiahpm.com [13:41:44] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 11.69, 16.62, 19.51 [13:51:44] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.10, 19.28, 18.85 [13:53:44] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 16.94, 18.38, 18.57 [13:59:44] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.74, 20.75, 19.46 [14:01:44] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 16.36, 18.60, 18.80 [14:07:36] RECOVERY - neurologiahpm.com - LetsEncrypt on sslhost is OK: OK - Certificate 'neurologiahpm.com' will expire on Sun 29 Sep 2024 12:37:53 PM GMT +0000. [14:11:44] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.83, 22.92, 20.63 [14:15:44] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.12, 23.09, 21.29 [14:25:44] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 17.32, 19.38, 20.31 [14:32:25] [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [14:46:11] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.402 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out. [15:08:05] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 23.62, 18.47, 17.08 [15:13:51] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 17.96, 19.56, 18.06 [15:16:43] PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:22:36] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 26.15, 22.82, 20.11 [15:32:14] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 17.63, 23.35, 22.38 [15:41:51] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.27, 21.70, 21.79 [15:43:47] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 23.25, 21.87, 21.83 [15:45:00] RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:57:44] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 15.66, 18.83, 20.18 [16:01:44] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.78, 20.67, 20.61 [16:03:44] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 26.25, 22.48, 21.26 [16:05:44] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 23.96, 22.94, 21.59 [16:11:44] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 13.39, 18.27, 20.14 [16:21:30] PROBLEM - wiki.lifesprogress.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.lifesprogress.com' expires in 15 day(s) (Wed 17 Jul 2024 04:02:55 PM GMT +0000). [16:21:42] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/33699ab1c992...472c6bfbf4ce [16:21:43] [02ssl] 07WikiTideSSLBot 03472c6bf - Bot: Update SSL cert for wiki.lifesprogress.com [16:21:52] PROBLEM - wiki.andreijiroh.uk.eu.org - LetsEncrypt on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket [16:31:09] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/472c6bfbf4ce...a596fab73687 [16:31:10] [02ssl] 07WikiTideSSLBot 03a596fab - Bot: Add SSL cert for meijinversus.com [16:32:11] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/a596fab73687...233a09f237b9 [16:32:13] [02ssl] 07WikiTideSSLBot 03233a09f - Bot: Add SSL cert for valsora.tep.wiki [16:33:23] PROBLEM - cp51 Varnish Backends on cp51 is CRITICAL: 1 backends are down. mw152 [16:35:20] RECOVERY - cp51 Varnish Backends on cp51 is OK: All 19 backends are healthy [16:50:00] RECOVERY - wiki.lifesprogress.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.lifesprogress.com' will expire on Sun 29 Sep 2024 03:21:35 PM GMT +0000. [17:18:18] [02MirahezeMagic] 07OAuthority pushed 031 commit to 03OAuthority-patch-1 [+0/-0/±1] 13https://github.com/miraheze/MirahezeMagic/commit/8d11d6cb8515 [17:18:19] [02MirahezeMagic] 07OAuthority 038d11d6c - Pretty Print JSON result [17:18:20] [02MirahezeMagic] 07OAuthority created branch 03OAuthority-patch-1 - 13https://github.com/miraheze/MirahezeMagic [17:18:25] [02MirahezeMagic] 07OAuthority opened pull request 03#496: Pretty Print JSON result - 13https://github.com/miraheze/MirahezeMagic/pull/496 [17:18:31] [02MirahezeMagic] 07coderabbitai[bot] commented on pull request 03#496: Pretty Print JSON result - 13https://github.com/miraheze/MirahezeMagic/pull/496#issuecomment-2200662805 [17:23:36] miraheze/MirahezeMagic - OAuthority the build has errored. [17:26:36] [02MirahezeMagic] 07OAuthority closed pull request 03#496: Pretty Print JSON result - 13https://github.com/miraheze/MirahezeMagic/pull/496 [17:26:37] [02MirahezeMagic] 07OAuthority pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/MirahezeMagic/compare/64597fddb1df...425be95acf25 [17:26:40] [02MirahezeMagic] 07OAuthority 03425be95 - Pretty Print JSON result (#496) [17:26:43] [02MirahezeMagic] 07OAuthority deleted branch 03OAuthority-patch-1 [17:26:44] [02MirahezeMagic] 07OAuthority deleted branch 03OAuthority-patch-1 - 13https://github.com/miraheze/MirahezeMagic [17:26:54] !log [oa@mwtask181] starting deploy of {'l10n': True, 'versions': '1.41', 'upgrade_extensions': 'MirahezeMagic'} to all [17:26:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:27:48] !log [oa@mwtask181] finished deploy of {'l10n': True, 'versions': '1.41', 'upgrade_extensions': 'MirahezeMagic'} to all - SUCCESS in 54s [17:27:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:30:16] miraheze/MirahezeMagic - OAuthority the build has errored. [17:32:45] !log [@mwtask181] starting deploy of {'folders': '1.42/extensions/MirahezeMagic'} to all [17:32:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:33:02] !log [@mwtask181] finished deploy of {'folders': '1.42/extensions/MirahezeMagic'} to all - SUCCESS in 16s [17:33:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:37:21] !log [@mwtask171] starting deploy of {'folders': '1.41/extensions/MirahezeMagic'} to all [17:37:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:37:37] !log [@mwtask171] finished deploy of {'folders': '1.41/extensions/MirahezeMagic'} to all - SUCCESS in 16s [17:37:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:37:49] !log [@mwtask171] starting deploy of {'folders': '1.42/extensions/MirahezeMagic'} to all [17:37:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:38:06] !log [@mwtask171] finished deploy of {'folders': '1.42/extensions/MirahezeMagic'} to all - SUCCESS in 17s [17:38:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:54:21] !log [@test151] starting deploy of {'folders': '1.41/extensions/MirahezeMagic'} to test151 [17:54:22] !log [@test151] finished deploy of {'folders': '1.41/extensions/MirahezeMagic'} to test151 - SUCCESS in 0s [17:54:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:54:33] !log [@test151] starting deploy of {'folders': '1.42/extensions/MirahezeMagic'} to test151 [17:54:34] !log [@test151] finished deploy of {'folders': '1.42/extensions/MirahezeMagic'} to test151 - SUCCESS in 0s [17:54:46] !log [@test151] starting deploy of {'folders': '1.43/extensions/MirahezeMagic'} to test151 [17:54:47] !log [@test151] finished deploy of {'folders': '1.43/extensions/MirahezeMagic'} to test151 - SUCCESS in 0s [17:54:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:54:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:54:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:55:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:55:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:32:25] [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [20:14:04] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.andreijiroh.uk.eu.org All nameservers failed to answer the query. [20:20:37] RECOVERY - wiki.andreijiroh.uk.eu.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.andreijiroh.uk.eu.org' will expire on Tue 23 Jul 2024 06:11:42 PM GMT +0000. [20:52:25] PROBLEM - wiki.wikimedia.cat - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.404 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out. [20:59:04] PROBLEM - wiki.wikimedia.cat - LetsEncrypt on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket [21:02:18] https://github.com/miraheze/MirahezeMagic/pull/491 [21:14:01] RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp36.wikitide.net - CNAME OK [21:52:05] PROBLEM - wiki.wikimedia.cat - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.wikimedia.cat All nameservers failed to answer the query. [21:58:47] RECOVERY - wiki.wikimedia.cat - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.wikimedia.cat' will expire on Sun 28 Jul 2024 08:30:13 AM GMT +0000. [22:06:46] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 109.123.230.163/cpweb [22:08:46] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [22:21:50] RECOVERY - wiki.wikimedia.cat - reverse DNS on sslhost is OK: SSL OK - wiki.wikimedia.cat reverse DNS resolves to cp36.wikitide.net - CNAME OK [22:22:46] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 109.123.230.163/cpweb [22:24:46] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [22:32:25] [Grafana] !tech RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [23:46:40] PROBLEM - wiki.aclevo.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.aclevo.com' expires in 15 day(s) (Wed 17 Jul 2024 11:23:29 PM GMT +0000). [23:46:54] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/233a09f237b9...4d2031089065 [23:46:57] [02ssl] 07WikiTideSSLBot 034d20310 - Bot: Update SSL cert for wiki.aclevo.com [23:54:46] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 109.123.230.163/cpweb [23:56:46] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online