[00:06:55] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 18.16, 22.98, 22.04 [00:18:36] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 25.07, 22.00, 21.49 [00:20:33] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 23.39, 22.18, 21.60 [00:26:27] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 25.63, 22.62, 21.83 [00:30:27] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 20.58, 22.81, 22.18 [00:34:27] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 26.39, 23.64, 22.58 [00:38:27] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 19.18, 22.13, 22.23 [00:42:27] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 25.05, 23.29, 22.62 [00:46:27] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 18.14, 22.63, 22.69 [01:14:27] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 26.64, 23.70, 22.59 [01:20:27] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 18.77, 22.50, 22.62 [01:26:27] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 16.69, 17.84, 20.37 [01:32:10] [02MessageCachePerformance] 07colleirose opened pull request 03#2: Fix broken link and add author info in extension.json - 13https://github.com/miraheze/MessageCachePerformance/pull/2 [01:32:15] [02MessageCachePerformance] 07coderabbitai[bot] commented on pull request 03#2: Fix broken link and add author info in extension.json - 13https://github.com/miraheze/MessageCachePerformance/pull/2#issuecomment-2195936078 [01:40:22] [02mw-config] 07colleirose deleted a comment on pull request 03#5565: Merge duplicate entries together - 13https://github.com/miraheze/mw-config/pull/5565#issuecomment-2119362118 [02:23:46] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 23.81, 21.75, 20.17 [02:25:43] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 25.59, 22.97, 20.80 [02:31:33] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.93, 23.58, 21.96 [02:33:30] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 26.92, 24.58, 22.50 [02:35:27] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.00, 23.50, 22.38 [02:49:05] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.94, 23.51, 22.81 [03:08:33] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.15, 23.10, 23.94 [03:10:30] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 25.39, 23.90, 24.13 [03:12:18] PROBLEM - graylog161 HTTPS on graylog161 is CRITICAL: HTTP CRITICAL: HTTP/2 502 - 113 bytes in 0.024 second response time [03:12:27] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 20.60, 22.79, 23.72 [03:22:25] [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [03:26:27] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 17.38, 17.33, 20.14 [03:52:18] RECOVERY - graylog161 HTTPS on graylog161 is OK: HTTP OK: HTTP/2 200 - 646 bytes in 0.060 second response time [03:55:02] PROBLEM - graylog161 Current Load on graylog161 is WARNING: LOAD WARNING - total load average: 7.33, 4.79, 2.17 [03:57:27] PROBLEM - os151 Current Load on os151 is CRITICAL: LOAD CRITICAL - total load average: 4.58, 2.89, 1.29 [03:58:45] PROBLEM - os161 Current Load on os161 is CRITICAL: LOAD CRITICAL - total load average: 4.85, 3.30, 1.54 [03:59:02] RECOVERY - graylog161 Current Load on graylog161 is OK: LOAD OK - total load average: 6.56, 5.92, 3.25 [04:07:25] [Grafana] !tech RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [04:08:45] PROBLEM - os161 Current Load on os161 is WARNING: LOAD WARNING - total load average: 2.52, 3.60, 2.65 [04:09:27] PROBLEM - os151 Current Load on os151 is WARNING: LOAD WARNING - total load average: 2.16, 3.48, 2.73 [04:10:45] RECOVERY - os161 Current Load on os161 is OK: LOAD OK - total load average: 2.06, 3.07, 2.57 [04:11:27] RECOVERY - os151 Current Load on os151 is OK: LOAD OK - total load average: 1.64, 2.90, 2.61 [05:06:27] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.33, 18.02, 15.27 [05:08:27] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 20.24, 18.29, 15.68 [05:15:17] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.71, 22.04, 18.23 [05:15:40] PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [05:19:28] PROBLEM - wiki.lewisgill.com - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.403 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out. [05:21:08] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.77, 23.21, 20.02 [05:26:59] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 27.07, 24.69, 21.60 [05:28:55] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 19.80, 23.57, 21.62 [05:43:40] RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [05:46:27] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 15.59, 19.47, 20.37 [05:46:56] PROBLEM - db151 Current Load on db151 is CRITICAL: LOAD CRITICAL - total load average: 49.26, 24.45, 9.91 [05:48:16] PROBLEM - db151 APT on db151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [05:52:27] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 26.12, 22.96, 21.54 [05:53:12] RECOVERY - db151 APT on db151 is OK: APT OK: 33 packages available for upgrade (0 critical updates). [05:56:56] RECOVERY - db151 Current Load on db151 is OK: LOAD OK - total load average: 0.72, 9.00, 10.20 [06:40:27] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 18.14, 21.14, 23.46 [06:56:27] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 25.25, 20.85, 21.40 [07:00:27] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 20.11, 22.03, 21.89 [07:10:27] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 17.38, 18.25, 20.07 [07:17:13] RECOVERY - wiki.lewisgill.com - reverse DNS on sslhost is OK: SSL OK - wiki.lewisgill.com reverse DNS resolves to cp36.wikitide.net - CNAME OK [08:22:25] [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [09:05:14] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 26.48, 22.45, 19.61 [09:11:05] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.63, 23.97, 21.33 [09:18:52] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 15.25, 18.97, 20.21 [09:24:40] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 26.91, 22.93, 21.47 [09:26:36] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.57, 21.82, 21.22 [09:27:25] [Grafana] !tech RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [09:34:27] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 14.36, 18.41, 20.20 [09:40:27] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.83, 20.20, 20.36 [09:46:27] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 14.87, 18.22, 19.61 [09:58:27] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.62, 20.81, 19.56 [10:00:27] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 17.54, 19.85, 19.38 [10:02:25] [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [10:12:27] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.08, 21.58, 20.22 [10:14:27] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.84, 22.48, 20.76 [10:16:27] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.47, 23.09, 21.18 [10:18:27] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 18.28, 21.60, 20.89 [10:22:27] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 16.17, 19.79, 20.39 [10:30:27] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 25.04, 21.51, 20.66 [10:31:32] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 109.123.230.163/cpweb, 2400:d320:2161:9775::1/cpweb [10:31:43] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 109.123.230.163/cpweb, 2400:d320:2161:9775::1/cpweb [10:32:00] PROBLEM - cp41 SSH on cp41 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:32:19] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 100% [10:32:27] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.74, 21.70, 20.85 [10:32:56] PROBLEM - cp41 HTTPS on cp41 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Connection timed out after 10002 milliseconds [10:33:37] PROBLEM - cp41 conntrack_table_size on cp41 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:33:38] PROBLEM - cp41 Nginx Backend for reports171 on cp41 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:33:39] PROBLEM - cp41 HTTP 4xx/5xx ERROR Rate on cp41 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:33:42] PROBLEM - Host cp41 is DOWN: PING CRITICAL - Packet loss = 100% [10:34:27] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 27.57, 23.75, 21.70 [10:38:27] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 20.81, 22.76, 21.85 [10:52:27] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 19.31, 19.20, 20.23 [11:23:05] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.54, 20.66, 19.34 [11:30:52] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 15.87, 20.23, 19.97 [11:32:51] RECOVERY - Host cp41 is UP: PING OK - Packet loss = 0%, RTA = 102.67 ms [11:33:26] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 102.54 ms [11:33:32] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [11:33:43] RECOVERY - cp41 SSH on cp41 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u2 (protocol 2.0) [11:33:47] RECOVERY - cp41 conntrack_table_size on cp41 is OK: OK: nf_conntrack is 1 % full [11:33:54] RECOVERY - cp41 HTTPS on cp41 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3782 bytes in 0.693 second response time [11:33:58] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [11:34:03] RECOVERY - cp41 Nginx Backend for reports171 on cp41 is OK: TCP OK - 0.000 second response time on localhost port 8205 [11:34:41] RECOVERY - cp41 HTTP 4xx/5xx ERROR Rate on cp41 is OK: OK - NGINX Error Rate is 2% [11:36:39] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 23.88, 21.26, 20.30 [11:40:33] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.30, 21.99, 20.74 [11:44:27] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 19.97, 21.73, 20.97 [11:46:27] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 14.51, 19.03, 20.07 [11:58:27] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 23.64, 20.68, 19.93 [12:06:27] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.51, 22.87, 21.22 [12:08:27] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 17.58, 20.91, 20.72 [12:14:27] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.67, 22.44, 21.36 [12:14:27] PROBLEM - db171 Backups SQL on db171 is WARNING: FILE_AGE WARNING: /var/log/sql-backup.log is 864187 seconds old and 134735 bytes [12:16:27] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 18.15, 21.40, 21.16 [12:30:27] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 15.92, 18.75, 20.40 [13:35:24] [02mw-config] 07BlankEclair opened pull request 03#5595: T12263: Fix editing main pages of wikis that use /$1 and root as main page - 13https://github.com/miraheze/mw-config/pull/5595 [13:36:23] miraheze/mw-config - BlankEclair the build passed. [14:02:25] [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [14:24:27] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 26.46, 22.18, 18.43 [14:26:27] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 20.77, 21.28, 18.55 [14:28:27] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 26.39, 23.01, 19.49 [14:30:27] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.19, 22.42, 19.70 [14:37:19] BlankEclair: do you discord? [14:37:27] nope [14:38:24] She’s escaped [14:38:33] :xsob: [14:38:43] Run Claire. Escape for us all [14:39:17] i am concurrently holding two conversations whilst editing my wiki rn lol [14:39:43] LMAO [14:39:50] BlankEclair: there was a support thread about that PR [14:39:53] (the comic author just dropped a page with so much lore that i anticipate that my sleep is gonna be held back for an hour) [14:39:55] She’s multi threaded [14:40:03] single threaded, doing all three tasks poorly [14:40:08] You sleep? [14:40:16] supposedly [14:40:21] I never said they’re powerful threads [14:40:27] touche [14:40:34] Ro goes to sleep at 3 AM fairly consistently so [14:41:05] my sleep times cluster around 11 pm to 1 am [14:41:12] Wonder if we could relay threads [14:41:22] chaotic but would be an interesting idea [14:41:24] Same. Well this summer at least [14:41:39] Probably using ## channels [14:41:41] How? [14:41:49] Not sure how you'd discover them [14:41:59] Truuue [14:42:10] (oh, and another wiki editor noticed some funky stuff with embedding a page on discord, so i gotta debug that as well) [14:42:11] You’d need a discord person to tell ya [14:42:27] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 18.44, 20.38, 20.33 [14:42:32] Also how would perms be managed [14:42:44] Wikis are forcing you to use discord for troubleshooting? Lmao [14:42:56] Listen. I don’t think things through [14:42:58] We'd probably need a group account to be able to go and register them [14:43:11] It's a very good idea [14:43:12] oh, nah, it's just that the description passed for a page returns stuff in