[00:33:54] !log [void@db151] [xdjibiwiki]> UPDATE revision SET rev_comment_id=15 WHERE rev_comment_id=0; -- set invalid rev_comment_id to new empty comment [00:33:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:52:25] [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [02:42:25] [Grafana] !tech RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [03:14:11] PROBLEM - ping6 on ns2 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 148.37 ms [03:18:12] RECOVERY - ping6 on ns2 is OK: PING OK - Packet loss = 0%, RTA = 139.79 ms [03:46:32] PROBLEM - ping6 on ns2 is CRITICAL: PING CRITICAL - Packet loss = 44%, RTA = 148.03 ms [03:48:32] RECOVERY - ping6 on ns2 is OK: PING OK - Packet loss = 0%, RTA = 139.74 ms [04:20:48] PROBLEM - ping6 on ns2 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 148.24 ms [04:24:55] RECOVERY - ping6 on ns2 is OK: PING OK - Packet loss = 0%, RTA = 139.74 ms [07:31:13] PROBLEM - ns2 NTP time on ns2 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H [-4|-6] [-w ] [-c ] [-v verbose] [-o ] [07:33:10] RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset -0.0005624294281 secs [07:47:14] PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_dns] [08:15:00] RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [09:45:20] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 20.84, 20.21, 18.23 [09:49:20] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 27.34, 23.39, 19.93 [09:51:21] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.51, 22.67, 20.11 [09:59:20] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 19.71, 20.23, 20.02 [10:51:50] !log [alex@mwtask181] starting deploy of {'folders': '1.41/vendor'} to all [10:51:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:52:03] !log [alex@mwtask181] finished deploy of {'folders': '1.41/vendor'} to all - SUCCESS in 13s [10:52:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:54:22] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 25.65, 21.63, 19.39 [11:00:10] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 20.68, 22.11, 20.41 [11:02:05] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 16.36, 20.26, 19.95 [11:35:20] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 23.34, 21.32, 18.67 [11:39:20] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 19.16, 20.28, 18.85 [12:07:34] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.46, 20.08, 19.15 [12:11:25] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 18.21, 19.55, 19.17 [12:26:12] PROBLEM - wiki.moores.tech - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.moores.tech could not be found [12:27:19] PROBLEM - wiki.moores.tech - LetsEncrypt on sslhost is CRITICAL: Name or service not knownHTTP CRITICAL - Unable to open TCP socket [12:29:04] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 26.15, 22.11, 20.26 [12:31:00] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 20.00, 20.45, 19.83 [12:32:56] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 15.79, 18.52, 19.19 [12:43:51] RECOVERY - cp37 Nginx Backend for mw181 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8119 [12:47:52] PROBLEM - cp37 Nginx Backend for mw181 on cp37 is CRITICAL: connect to address localhost and port 8119: Connection refused [13:02:54] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.52, 20.70, 18.87 [13:04:49] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 25.48, 22.13, 19.59 [13:06:45] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 20.47, 21.52, 19.66 [13:10:37] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 14.12, 19.87, 19.62 [13:20:17] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 25.80, 21.94, 20.39 [13:28:01] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 19.23, 22.50, 21.72 [13:37:40] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 16.58, 18.12, 19.94 [13:45:25] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 28.04, 22.52, 20.78 [13:47:21] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 23.26, 22.56, 21.00 [13:49:20] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 25.12, 23.01, 21.32 [13:51:20] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.79, 22.67, 21.39 [14:05:20] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 15.39, 18.81, 20.23 [14:25:20] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 28.17, 22.24, 19.57 [14:27:20] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 17.55, 20.85, 19.44 [14:29:20] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 17.20, 19.51, 19.11 [14:47:20] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 23.85, 19.88, 18.70 [14:51:20] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.77, 21.26, 19.45 [14:53:20] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 18.70, 20.47, 19.41 [14:54:12] PROBLEM - wiki.moores.tech - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.moores.tech All nameservers failed to answer the query. [14:55:20] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 26.39, 22.90, 20.44 [14:57:20] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 23.52, 23.33, 20.93 [14:59:20] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 27.10, 24.37, 21.57 [15:07:20] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 23.24, 23.95, 22.65 [15:21:20] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 16.08, 17.90, 19.81 [15:27:20] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 23.62, 21.74, 20.86 [15:29:20] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 17.05, 19.74, 20.22 [15:37:20] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 18.56, 21.19, 20.65 [15:39:20] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 18.29, 20.27, 20.37 [15:43:20] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.74, 21.63, 20.85 [15:49:21] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 26.30, 22.64, 21.41 [15:53:20] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.41, 23.46, 22.10 [15:55:20] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.46, 24.17, 22.54 [15:57:20] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.68, 23.20, 22.37 [16:05:20] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.36, 23.02, 22.46 [16:09:20] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 20.17, 21.91, 22.24 [16:17:20] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 16.56, 18.34, 20.31 [16:23:20] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.77, 22.71, 21.64 [16:28:41] RECOVERY - cp37 Nginx Backend for mw171 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8117 [16:31:20] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 19.32, 18.93, 20.29 [16:32:41] PROBLEM - cp37 Nginx Backend for mw171 on cp37 is CRITICAL: connect to address localhost and port 8117: Connection refused [16:37:25] [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [16:39:20] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 23.66, 20.49, 20.21 [16:41:20] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 27.28, 22.62, 21.00 [16:43:20] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 17.13, 20.45, 20.41 [16:49:20] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 15.18, 19.52, 20.29 [16:55:20] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 26.40, 23.48, 21.55 [16:59:20] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 20.95, 23.97, 22.35 [17:17:20] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 19.13, 19.07, 20.27 [17:23:20] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.34, 20.18, 20.27 [17:27:20] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.35, 22.29, 21.08 [17:29:20] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 19.07, 21.50, 20.98 [17:31:20] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.13, 22.37, 21.33 [17:33:20] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 20.51, 21.62, 21.18 [17:35:17] PROBLEM - ns2 NTP time on ns2 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H [-4|-6] [-w ] [-c ] [-v verbose] [-o ] [17:37:14] RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset -0.0003668665886 secs [17:39:20] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 16.49, 19.16, 20.26 [17:45:20] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.55, 21.13, 20.64 [17:46:44] PROBLEM - ns2 NTP time on ns2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:47:20] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 16.99, 19.13, 19.94 [17:48:41] RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset -0.0003629624844 secs [18:24:08] PROBLEM - wiki.andreijiroh.uk.eu.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.andreijiroh.uk.eu.org' expires in 15 day(s) (Tue 23 Jul 2024 06:11:42 PM GMT +0000). [18:24:37] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/da30525491df...cb55a002846f [18:24:38] [02ssl] 07WikiTideSSLBot 03cb55a00 - Bot: Update SSL cert for wiki.andreijiroh.uk.eu.org [18:34:57] PROBLEM - ns2 NTP time on ns2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:37:04] PROBLEM - ns2 NTP time on ns2 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H [-4|-6] [-w ] [-c ] [-v verbose] [-o ] [18:39:01] RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset -0.0005208849907 secs [18:53:47] RECOVERY - wiki.andreijiroh.uk.eu.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.andreijiroh.uk.eu.org' will expire on Sat 05 Oct 2024 05:24:27 PM GMT +0000. [19:17:09] PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:24:41] RECOVERY - cp37 Nginx Backend for mw171 on cp37 is OK: TCP OK - 0.000 second response time on localhost port 8117 [19:28:41] PROBLEM - cp37 Nginx Backend for mw171 on cp37 is CRITICAL: connect to address localhost and port 8117: Connection refused [19:47:25] [Grafana] !tech RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [20:15:05] RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [21:25:02] PROBLEM - wiki.col6.de - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.col6.de' expires in 15 day(s) (Tue 23 Jul 2024 09:06:12 PM GMT +0000). [21:25:14] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/cb55a002846f...d51ecf38c2a8 [21:25:16] [02ssl] 07WikiTideSSLBot 03d51ecf3 - Bot: Update SSL cert for wiki.col6.de [21:35:05] PROBLEM - wiki.lewisgill.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.lewisgill.com' expires in 15 day(s) (Tue 23 Jul 2024 09:32:19 PM GMT +0000). [21:35:16] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/d51ecf38c2a8...d97e43e4d005 [21:35:17] [02ssl] 07WikiTideSSLBot 03d97e43e - Bot: Update SSL cert for wiki.lewisgill.com [21:49:42] PROBLEM - msmhacks.uk - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'msmhacks.uk' expires in 15 day(s) (Tue 23 Jul 2024 09:32:23 PM GMT +0000). [21:49:52] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/d97e43e4d005...bda73cc2183e [21:49:54] [02ssl] 07WikiTideSSLBot 03bda73cc - Bot: Update SSL cert for msmhacks.uk [21:53:47] PROBLEM - wiki.simplifiedcoding.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.simplifiedcoding.org' expires in 15 day(s) (Tue 23 Jul 2024 09:30:52 PM GMT +0000). [21:53:57] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/bda73cc2183e...4e0e4b94e032 [21:53:58] [02ssl] 07WikiTideSSLBot 034e0e4b9 - Bot: Update SSL cert for wiki.simplifiedcoding.org [21:54:04] RECOVERY - wiki.col6.de - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.col6.de' will expire on Sat 05 Oct 2024 08:25:08 PM GMT +0000. [21:56:38] PROBLEM - zomvilles.pixiemeta.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'zomvilles.pixiemeta.com' expires in 15 day(s) (Tue 23 Jul 2024 09:40:49 PM GMT +0000). [21:56:49] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/4e0e4b94e032...18158595a19b [21:56:50] [02ssl] 07WikiTideSSLBot 031815859 - Bot: Update SSL cert for zomvilles.pixiemeta.com [21:57:11] PROBLEM - eosdev.xyz - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'eosdev.xyz' expires in 15 day(s) (Tue 23 Jul 2024 09:30:04 PM GMT +0000). [21:57:21] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/18158595a19b...f13fc41651ec [21:57:23] [02ssl] 07WikiTideSSLBot 03f13fc41 - Bot: Update SSL cert for eosdev.xyz [21:59:01] PROBLEM - wiki.manos.pt - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.manos.pt' expires in 15 day(s) (Tue 23 Jul 2024 09:32:08 PM GMT +0000). [21:59:15] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/f13fc41651ec...325b3ae25b10 [21:59:17] [02ssl] 07WikiTideSSLBot 03325b3ae - Bot: Update SSL cert for wiki.manos.pt [22:03:40] RECOVERY - wiki.lewisgill.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.lewisgill.com' will expire on Sat 05 Oct 2024 08:35:11 PM GMT +0000. [22:12:49] PROBLEM - wiki.sheepservermc.net - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.sheepservermc.net' expires in 15 day(s) (Tue 23 Jul 2024 10:09:18 PM GMT +0000). [22:13:00] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/325b3ae25b10...f484ac657684 [22:13:02] [02ssl] 07WikiTideSSLBot 03f484ac6 - Bot: Update SSL cert for wiki.sheepservermc.net [22:15:20] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.72, 17.63, 14.16 [22:17:20] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 19.14, 18.66, 15.00 [22:19:23] RECOVERY - msmhacks.uk - LetsEncrypt on sslhost is OK: OK - Certificate 'msmhacks.uk' will expire on Sat 05 Oct 2024 08:49:46 PM GMT +0000. [22:23:03] RECOVERY - wiki.simplifiedcoding.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.simplifiedcoding.org' will expire on Sat 05 Oct 2024 08:53:51 PM GMT +0000. [22:25:20] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 20.49, 20.23, 17.00 [22:25:46] RECOVERY - zomvilles.pixiemeta.com - LetsEncrypt on sslhost is OK: OK - Certificate 'zomvilles.pixiemeta.com' will expire on Sat 05 Oct 2024 08:56:43 PM GMT +0000. [22:26:51] RECOVERY - eosdev.xyz - LetsEncrypt on sslhost is OK: OK - Certificate 'eosdev.xyz' will expire on Sat 05 Oct 2024 08:57:15 PM GMT +0000. [22:27:20] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 18.25, 20.26, 17.45 [22:27:32] RECOVERY - wiki.manos.pt - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.manos.pt' will expire on Sat 05 Oct 2024 08:59:10 PM GMT +0000. [23:10:50] RECOVERY - wiki.sheepservermc.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.sheepservermc.net' will expire on Sat 05 Oct 2024 09:12:55 PM GMT +0000. [23:37:20] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.01, 20.28, 17.98 [23:39:20] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 17.42, 19.53, 18.01