[00:03:25] PROBLEM - db112 Current Load on db112 is CRITICAL: CRITICAL - load average: 14.94, 7.30, 3.36 [00:07:35] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 4.91, 3.92, 3.34 [00:29:45] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 8.00, 10.72, 9.65 [00:30:49] RECOVERY - mw132 Current Load on mw132 is OK: LOAD OK - total load average: 7.16, 9.64, 9.62 [00:31:41] RECOVERY - mw134 Current Load on mw134 is OK: LOAD OK - total load average: 8.37, 9.89, 9.46 [00:37:30] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 2.60, 3.34, 3.91 [00:39:10] RECOVERY - wows.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'wows.wiki' will expire on Sat 30 Dec 2023 06:14:45 GMT +0000. [00:39:28] PROBLEM - os131 Current Load on os131 is WARNING: LOAD WARNING - total load average: 3.55, 3.24, 3.94 [00:40:57] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.48, 3.87, 3.24 [00:42:53] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 2.58, 3.32, 3.11 [00:46:50] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.60, 4.11, 3.47 [00:48:53] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.75, 3.87, 3.45 [00:49:30] RECOVERY - os131 Current Load on os131 is OK: LOAD OK - total load average: 2.27, 2.72, 3.33 [00:50:48] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.83, 6.47, 6.00 [00:52:48] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.27, 6.32, 6.00 [00:52:48] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 2.66, 3.20, 3.27 [00:57:30] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 2.00, 2.98, 3.32 [00:58:18] PROBLEM - os131 Current Load on os131 is WARNING: LOAD WARNING - total load average: 3.52, 3.32, 3.38 [01:00:15] RECOVERY - os131 Current Load on os131 is OK: LOAD OK - total load average: 2.97, 3.29, 3.37 [01:06:13] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 5.58, 4.33, 3.74 [01:06:14] PROBLEM - os131 Current Load on os131 is CRITICAL: LOAD CRITICAL - total load average: 6.98, 4.58, 3.76 [01:06:48] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.51, 3.40, 3.30 [01:06:57] PROBLEM - mw134 Current Load on mw134 is CRITICAL: LOAD CRITICAL - total load average: 12.11, 10.90, 8.90 [01:08:00] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 11.62, 10.11, 8.54 [01:08:50] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.04, 3.57, 3.37 [01:08:57] RECOVERY - mw134 Current Load on mw134 is OK: LOAD OK - total load average: 8.52, 9.81, 8.73 [01:10:33] PROBLEM - mw131 Current Load on mw131 is CRITICAL: LOAD CRITICAL - total load average: 17.29, 11.65, 9.03 [01:11:23] PROBLEM - mw142 Current Load on mw142 is WARNING: LOAD WARNING - total load average: 10.94, 11.19, 8.42 [01:12:33] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 9.60, 10.79, 9.05 [01:12:53] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 2.36, 3.47, 3.43 [01:13:22] RECOVERY - mw142 Current Load on mw142 is OK: LOAD OK - total load average: 6.53, 9.42, 8.09 [01:14:33] RECOVERY - mw131 Current Load on mw131 is OK: LOAD OK - total load average: 8.48, 9.82, 8.90 [01:14:50] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 2.47, 3.11, 3.30 [01:17:49] PROBLEM - mw132 Current Load on mw132 is CRITICAL: LOAD CRITICAL - total load average: 14.71, 11.71, 9.88 [01:17:49] PROBLEM - mw134 Current Load on mw134 is CRITICAL: LOAD CRITICAL - total load average: 14.66, 11.30, 9.75 [01:19:02] PROBLEM - cp25 Varnish Backends on cp25 is CRITICAL: 3 backends are down. mw132 mw133 mw143 [01:19:12] PROBLEM - cp24 Varnish Backends on cp24 is CRITICAL: 2 backends are down. mw132 mw142 [01:19:17] PROBLEM - mw142 Current Load on mw142 is CRITICAL: LOAD CRITICAL - total load average: 16.11, 10.91, 8.86 [01:19:21] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 10.21, 10.86, 9.57 [01:19:25] PROBLEM - cp34 Varnish Backends on cp34 is CRITICAL: 2 backends are down. mw132 mw142 [01:19:36] PROBLEM - cp35 Varnish Backends on cp35 is CRITICAL: 1 backends are down. mw142 [01:20:33] PROBLEM - mw131 Current Load on mw131 is CRITICAL: LOAD CRITICAL - total load average: 14.96, 12.86, 10.45 [01:20:49] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.01, 4.19, 3.73 [01:20:58] RECOVERY - cp25 Varnish Backends on cp25 is OK: All 15 backends are healthy [01:21:07] RECOVERY - cp24 Varnish Backends on cp24 is OK: All 15 backends are healthy [01:21:15] PROBLEM - mw142 Current Load on mw142 is WARNING: LOAD WARNING - total load average: 10.34, 10.48, 8.97 [01:21:22] RECOVERY - cp34 Varnish Backends on cp34 is OK: All 15 backends are healthy [01:21:33] RECOVERY - cp35 Varnish Backends on cp35 is OK: All 15 backends are healthy [01:21:46] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 11.00, 11.92, 10.47 [01:22:34] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 9.29, 11.68, 10.33 [01:23:14] RECOVERY - mw142 Current Load on mw142 is OK: LOAD OK - total load average: 7.46, 9.45, 8.78 [01:23:14] RECOVERY - mw133 Current Load on mw133 is OK: LOAD OK - total load average: 6.48, 9.48, 9.37 [01:23:43] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 7.74, 10.97, 10.43 [01:25:39] PROBLEM - mw134 Current Load on mw134 is CRITICAL: LOAD CRITICAL - total load average: 13.48, 12.42, 11.04 [01:26:33] RECOVERY - mw131 Current Load on mw131 is OK: LOAD OK - total load average: 8.48, 9.84, 9.86 [01:26:50] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 2.81, 3.64, 3.66 [01:27:36] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 10.71, 11.57, 10.88 [01:27:46] PROBLEM - mw132 Current Load on mw132 is CRITICAL: LOAD CRITICAL - total load average: 14.97, 11.96, 10.71 [01:29:01] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 11.63, 11.05, 10.02 [01:29:34] PROBLEM - mw134 Current Load on mw134 is CRITICAL: LOAD CRITICAL - total load average: 15.94, 12.76, 11.37 [01:30:33] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 11.44, 11.81, 10.76 [01:30:59] PROBLEM - mw133 Current Load on mw133 is CRITICAL: LOAD CRITICAL - total load average: 17.28, 13.35, 10.98 [01:32:33] PROBLEM - mw131 Current Load on mw131 is CRITICAL: LOAD CRITICAL - total load average: 12.70, 11.63, 10.79 [01:32:51] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 6.32, 4.41, 3.89 [01:33:05] PROBLEM - mw142 Current Load on mw142 is CRITICAL: LOAD CRITICAL - total load average: 13.41, 11.83, 9.69 [01:33:54] PROBLEM - os141 PowerDNS Recursor on os141 is CRITICAL: CRITICAL - Plugin timed out while executing system call [01:34:27] PROBLEM - cp25 Varnish Backends on cp25 is CRITICAL: 2 backends are down. mw132 mw134 [01:34:36] PROBLEM - cp24 Varnish Backends on cp24 is CRITICAL: 2 backends are down. mw132 mw134 [01:34:41] PROBLEM - mw141 Current Load on mw141 is CRITICAL: LOAD CRITICAL - total load average: 17.17, 11.02, 8.65 [01:34:48] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.14, 3.94, 3.79 [01:34:53] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 9.82, 11.61, 10.83 [01:35:05] PROBLEM - cp34 Varnish Backends on cp34 is CRITICAL: 1 backends are down. mw141 [01:35:09] PROBLEM - cp35 Varnish Backends on cp35 is CRITICAL: 1 backends are down. mw141 [01:36:50] PROBLEM - mw133 Current Load on mw133 is CRITICAL: LOAD CRITICAL - total load average: 11.99, 12.51, 11.30 [01:36:51] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 6.80, 4.75, 4.08 [01:37:57] PROBLEM - mw143 Current Load on mw143 is CRITICAL: LOAD CRITICAL - total load average: 15.31, 11.87, 9.10 [01:38:01] RECOVERY - os141 PowerDNS Recursor on os141 is OK: DNS OK: 1.457 second response time. miraheze.org returns 2001:41d0:801:2000::3a18,2001:41d0:801:2000::5d68,51.195.201.140,51.89.139.24 [01:38:19] RECOVERY - cp25 Varnish Backends on cp25 is OK: All 15 backends are healthy [01:38:36] RECOVERY - cp24 Varnish Backends on cp24 is OK: All 15 backends are healthy [01:38:37] PROBLEM - mw141 Current Load on mw141 is WARNING: LOAD WARNING - total load average: 7.39, 10.89, 9.27 [01:38:59] RECOVERY - cp34 Varnish Backends on cp34 is OK: All 15 backends are healthy [01:39:02] RECOVERY - cp35 Varnish Backends on cp35 is OK: All 15 backends are healthy [01:39:56] PROBLEM - mw143 Current Load on mw143 is WARNING: LOAD WARNING - total load average: 8.38, 10.26, 8.84 [01:41:08] PROBLEM - wows.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address wows.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [01:41:55] RECOVERY - mw143 Current Load on mw143 is OK: LOAD OK - total load average: 4.95, 8.57, 8.41 [01:42:33] RECOVERY - mw141 Current Load on mw141 is OK: LOAD OK - total load average: 8.57, 9.29, 8.97 [01:42:34] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 6.45, 10.50, 11.04 [01:42:40] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 7.06, 10.51, 11.01 [01:42:58] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.14, 6.87, 6.50 [01:42:59] PROBLEM - mw142 Current Load on mw142 is WARNING: LOAD WARNING - total load average: 7.24, 10.59, 10.71 [01:44:52] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.07, 6.32, 6.34 [01:46:57] RECOVERY - mw142 Current Load on mw142 is OK: LOAD OK - total load average: 5.75, 7.96, 9.61 [01:47:46] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 10.70, 11.38, 11.91 [01:48:30] RECOVERY - mw133 Current Load on mw133 is OK: LOAD OK - total load average: 7.51, 8.46, 9.91 [01:48:33] RECOVERY - mw131 Current Load on mw131 is OK: LOAD OK - total load average: 8.63, 8.76, 10.03 [01:50:48] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.95, 6.70, 6.47 [01:51:46] PROBLEM - mw132 Current Load on mw132 is CRITICAL: LOAD CRITICAL - total load average: 20.58, 14.58, 12.94 [01:52:21] PROBLEM - mw133 Current Load on mw133 is CRITICAL: LOAD CRITICAL - total load average: 14.96, 11.95, 10.95 [01:52:33] PROBLEM - mw131 Current Load on mw131 is CRITICAL: LOAD CRITICAL - total load average: 12.28, 10.94, 10.62 [01:54:33] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 10.22, 11.02, 10.71 [01:54:48] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.05, 7.21, 6.69 [01:56:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.24, 7.00, 6.67 [01:58:21] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 8.03, 10.83, 11.03 [01:58:48] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.18, 6.34, 6.47 [02:02:33] RECOVERY - mw131 Current Load on mw131 is OK: LOAD OK - total load average: 8.23, 9.78, 10.18 [02:06:57] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 8.29, 9.91, 11.71 [02:08:21] RECOVERY - mw133 Current Load on mw133 is OK: LOAD OK - total load average: 7.32, 8.81, 10.12 [02:08:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.08, 6.96, 6.70 [02:09:46] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 7.57, 10.25, 11.82 [02:10:48] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.90, 6.55, 6.58 [02:10:57] PROBLEM - mw134 Current Load on mw134 is CRITICAL: LOAD CRITICAL - total load average: 13.29, 10.11, 11.26 [02:12:57] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 11.84, 10.85, 11.41 [02:13:46] PROBLEM - mw132 Current Load on mw132 is CRITICAL: LOAD CRITICAL - total load average: 12.11, 10.04, 11.26 [02:15:46] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 9.98, 9.94, 11.08 [02:16:48] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.55, 7.21, 6.89 [02:17:46] PROBLEM - mw132 Current Load on mw132 is CRITICAL: LOAD CRITICAL - total load average: 13.27, 10.37, 11.04 [02:19:46] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 7.77, 9.56, 10.69 [02:20:57] RECOVERY - mw134 Current Load on mw134 is OK: LOAD OK - total load average: 5.19, 8.24, 10.12 [02:21:40] PROBLEM - cloud11 Puppet on cloud11 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [02:21:59] PROBLEM - os131 Current Load on os131 is WARNING: LOAD WARNING - total load average: 3.28, 3.44, 3.93 [02:23:46] RECOVERY - mw132 Current Load on mw132 is OK: LOAD OK - total load average: 7.32, 8.83, 10.18 [02:24:48] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 4.75, 6.24, 6.61 [02:25:56] PROBLEM - os131 Current Load on os131 is CRITICAL: LOAD CRITICAL - total load average: 4.64, 3.94, 4.00 [02:27:53] PROBLEM - os131 Current Load on os131 is WARNING: LOAD WARNING - total load average: 2.26, 3.39, 3.80 [02:31:48] RECOVERY - os131 Current Load on os131 is OK: LOAD OK - total load average: 1.07, 2.17, 3.21 [02:37:46] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 10.84, 10.11, 9.72 [02:39:46] RECOVERY - mw132 Current Load on mw132 is OK: LOAD OK - total load average: 7.97, 9.31, 9.47 [02:47:22] PROBLEM - os141 Puppet on os141 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [02:48:20] PROBLEM - cp25 Varnish Backends on cp25 is CRITICAL: 1 backends are down. mw143 [02:48:41] PROBLEM - mw131 Current Load on mw131 is CRITICAL: LOAD CRITICAL - total load average: 18.01, 11.73, 9.62 [02:48:57] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 11.63, 8.30, 7.00 [02:49:40] RECOVERY - cloud11 Puppet on cloud11 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [02:49:46] PROBLEM - mw132 Current Load on mw132 is CRITICAL: LOAD CRITICAL - total load average: 15.36, 12.53, 10.61 [02:49:46] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 11.81, 11.42, 9.98 [02:50:19] RECOVERY - cp25 Varnish Backends on cp25 is OK: All 15 backends are healthy [02:50:21] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 10.63, 10.18, 9.08 [02:50:39] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 11.80, 11.37, 9.73 [02:51:10] PROBLEM - os141 PowerDNS Recursor on os141 is CRITICAL: CRITICAL - Plugin timed out while executing system call [02:51:46] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 8.13, 11.05, 10.33 [02:52:21] RECOVERY - mw133 Current Load on mw133 is OK: LOAD OK - total load average: 8.97, 9.83, 9.09 [02:52:36] RECOVERY - mw131 Current Load on mw131 is OK: LOAD OK - total load average: 7.36, 9.93, 9.40 [02:52:48] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.63, 7.50, 6.98 [02:52:58] RECOVERY - os141 Puppet on os141 is OK: OK: Puppet is currently enabled, last run 8 minutes ago with 0 failures [02:53:30] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.13, 3.67, 4.00 [02:53:46] RECOVERY - mw132 Current Load on mw132 is OK: LOAD OK - total load average: 7.93, 10.19, 10.11 [02:57:32] PROBLEM - mw134 Current Load on mw134 is CRITICAL: LOAD CRITICAL - total load average: 12.53, 12.31, 10.91 [02:59:29] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 9.85, 11.03, 10.59 [02:59:31] RECOVERY - os141 PowerDNS Recursor on os141 is OK: DNS OK: 0.512 seconds response time. miraheze.org returns 2001:41d0:801:2000::3a18,2001:41d0:801:2000::5d68,51.195.201.140,51.89.139.24 [03:00:48] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.95, 6.59, 6.79 [03:01:30] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 4.07, 3.41, 3.68 [03:03:30] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.36, 3.27, 3.59 [03:03:54] PROBLEM - os141 PowerDNS Recursor on os141 is CRITICAL: CRITICAL - Plugin timed out while executing system call [03:05:19] PROBLEM - mw134 Current Load on mw134 is CRITICAL: LOAD CRITICAL - total load average: 15.98, 13.53, 11.65 [03:05:30] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 5.19, 4.07, 3.85 [03:05:57] RECOVERY - os141 PowerDNS Recursor on os141 is OK: DNS OK: 5.396 seconds response time. miraheze.org returns 2001:41d0:801:2000::3a18,2001:41d0:801:2000::5d68,51.195.201.140,51.89.139.24 [03:06:39] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 10.96, 11.57, 10.64 [03:07:15] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 8.92, 11.63, 11.17 [03:09:30] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.50, 3.83, 3.80 [03:10:20] PROBLEM - os141 PowerDNS Recursor on os141 is CRITICAL: CRITICAL - Plugin timed out while executing system call [03:10:34] RECOVERY - mw132 Current Load on mw132 is OK: LOAD OK - total load average: 7.62, 9.88, 10.20 [03:11:26] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.30, 6.94, 6.75 [03:13:05] RECOVERY - mw134 Current Load on mw134 is OK: LOAD OK - total load average: 5.79, 8.34, 9.90 [03:13:20] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.25, 6.64, 6.66 [03:19:30] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 1.92, 2.98, 3.40 [03:20:59] RECOVERY - os141 PowerDNS Recursor on os141 is OK: DNS OK: 6.208 seconds response time. miraheze.org returns 2001:41d0:801:2000::3a18,2001:41d0:801:2000::5d68,51.195.201.140,51.89.139.24 [03:22:17] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 10.57, 9.62, 9.54 [03:25:23] PROBLEM - os141 PowerDNS Recursor on os141 is CRITICAL: CRITICAL - Plugin timed out while executing system call [03:25:46] PROBLEM - mw142 Current Load on mw142 is CRITICAL: LOAD CRITICAL - total load average: 13.78, 11.11, 9.34 [03:25:49] [Grafana] !sre FIRING: The mediawiki job queue has more than 2500 unclaimed jobs https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [03:27:03] PROBLEM - mw131 Current Load on mw131 is CRITICAL: LOAD CRITICAL - total load average: 13.59, 10.41, 9.03 [03:27:44] RECOVERY - mw142 Current Load on mw142 is OK: LOAD OK - total load average: 8.39, 10.16, 9.21 [03:28:57] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 9.41, 10.58, 10.00 [03:29:00] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 11.60, 10.69, 9.30 [03:29:10] PROBLEM - mw141 Current Load on mw141 is WARNING: LOAD WARNING - total load average: 9.00, 10.81, 9.56 [03:30:07] RECOVERY - mw132 Current Load on mw132 is OK: LOAD OK - total load average: 8.28, 9.70, 9.77 [03:30:57] RECOVERY - mw134 Current Load on mw134 is OK: LOAD OK - total load average: 9.67, 10.09, 9.87 [03:30:57] RECOVERY - mw131 Current Load on mw131 is OK: LOAD OK - total load average: 8.15, 9.54, 9.03 [03:31:36] RECOVERY - os141 PowerDNS Recursor on os141 is OK: DNS OK: 0.518 seconds response time. miraheze.org returns 2001:41d0:801:2000::3a18,2001:41d0:801:2000::5d68,51.195.201.140,51.89.139.24 [03:33:06] RECOVERY - mw141 Current Load on mw141 is OK: LOAD OK - total load average: 7.54, 9.48, 9.31 [03:39:08] RECOVERY - wows.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'wows.wiki' will expire on Sat 30 Dec 2023 06:14:45 GMT +0000. [03:42:00] PROBLEM - os141 PowerDNS Recursor on os141 is CRITICAL: CRITICAL - Plugin timed out while executing system call [03:45:13] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [03:46:10] RECOVERY - os141 PowerDNS Recursor on os141 is OK: DNS OK: 4.113 seconds response time. miraheze.org returns 2001:41d0:801:2000::3a18,2001:41d0:801:2000::5d68,51.195.201.140,51.89.139.24 [03:50:33] PROBLEM - os141 PowerDNS Recursor on os141 is CRITICAL: CRITICAL - Plugin timed out while executing system call [03:52:38] RECOVERY - os141 PowerDNS Recursor on os141 is OK: DNS OK: 6.423 seconds response time. miraheze.org returns 2001:41d0:801:2000::3a18,2001:41d0:801:2000::5d68,51.195.201.140,51.89.139.24 [04:00:31] PROBLEM - mw142 Current Load on mw142 is CRITICAL: LOAD CRITICAL - total load average: 13.16, 13.14, 9.81 [04:00:44] PROBLEM - mw141 Current Load on mw141 is WARNING: LOAD WARNING - total load average: 10.03, 10.90, 9.45 [04:01:05] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 11.95, 10.10, 8.32 [04:01:47] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 11.77, 10.53, 9.25 [04:02:30] PROBLEM - mw142 Current Load on mw142 is WARNING: LOAD WARNING - total load average: 8.68, 11.69, 9.68 [04:02:43] RECOVERY - mw141 Current Load on mw141 is OK: LOAD OK - total load average: 7.01, 9.32, 9.03 [04:03:46] RECOVERY - mw132 Current Load on mw132 is OK: LOAD OK - total load average: 7.62, 9.37, 8.98 [04:04:59] RECOVERY - mw133 Current Load on mw133 is OK: LOAD OK - total load average: 8.95, 9.60, 8.51 [04:06:29] PROBLEM - mw142 Current Load on mw142 is CRITICAL: LOAD CRITICAL - total load average: 22.83, 14.92, 11.27 [04:06:30] PROBLEM - os141 PowerDNS Recursor on os141 is CRITICAL: CRITICAL - Plugin timed out while executing system call [04:07:36] PROBLEM - cp24 Varnish Backends on cp24 is CRITICAL: 2 backends are down. mw132 mw134 [04:07:46] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 9.85, 10.60, 9.59 [04:07:47] PROBLEM - cp25 Varnish Backends on cp25 is CRITICAL: 1 backends are down. mw134 [04:08:35] PROBLEM - mw141 Current Load on mw141 is WARNING: LOAD WARNING - total load average: 9.11, 11.45, 10.16 [04:08:49] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 8.59, 10.37, 9.13 [04:09:30] RECOVERY - cp24 Varnish Backends on cp24 is OK: All 15 backends are healthy [04:09:42] RECOVERY - cp25 Varnish Backends on cp25 is OK: All 15 backends are healthy [04:09:46] RECOVERY - mw132 Current Load on mw132 is OK: LOAD OK - total load average: 6.84, 9.44, 9.30 [04:10:29] PROBLEM - mw142 Current Load on mw142 is WARNING: LOAD WARNING - total load average: 11.38, 11.92, 10.79 [04:10:33] RECOVERY - mw141 Current Load on mw141 is OK: LOAD OK - total load average: 6.14, 9.59, 9.63 [04:10:46] RECOVERY - mw133 Current Load on mw133 is OK: LOAD OK - total load average: 6.29, 8.96, 8.76 [04:11:10] PROBLEM - wows.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address wows.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [04:11:30] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 6.39, 4.13, 3.14 [04:13:46] PROBLEM - mw132 Current Load on mw132 is CRITICAL: LOAD CRITICAL - total load average: 12.93, 10.96, 9.89 [04:14:29] PROBLEM - mw142 Current Load on mw142 is CRITICAL: LOAD CRITICAL - total load average: 12.59, 11.47, 10.76 [04:15:14] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns.ankh.fr.eu.org.', 'ns1.eu.org.', 'ns1.eriomem.net.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [04:15:33] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 2.20, 3.60, 3.21 [04:17:04] RECOVERY - os141 PowerDNS Recursor on os141 is OK: DNS OK: 4.945 seconds response time. miraheze.org returns 2001:41d0:801:2000::3a18,2001:41d0:801:2000::5d68,51.195.201.140,51.89.139.24 [04:17:30] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 1.92, 3.12, 3.08 [04:18:24] PROBLEM - mw141 Current Load on mw141 is CRITICAL: LOAD CRITICAL - total load average: 12.32, 10.89, 9.97 [04:18:29] PROBLEM - mw142 Current Load on mw142 is WARNING: LOAD WARNING - total load average: 9.65, 11.29, 10.93 [04:20:22] RECOVERY - mw141 Current Load on mw141 is OK: LOAD OK - total load average: 7.46, 9.45, 9.55 [04:20:26] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 11.31, 10.19, 9.23 [04:22:23] RECOVERY - mw133 Current Load on mw133 is OK: LOAD OK - total load average: 6.83, 8.92, 8.88 [04:22:29] RECOVERY - mw142 Current Load on mw142 is OK: LOAD OK - total load average: 7.88, 9.00, 10.06 [04:23:46] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 5.92, 10.11, 10.57 [04:25:46] RECOVERY - mw132 Current Load on mw132 is OK: LOAD OK - total load average: 6.83, 8.94, 10.07 [04:58:48] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 1.02, 2.59, 3.79 [05:02:48] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 1.11, 1.81, 3.19 [05:03:28] PROBLEM - os131 Current Load on os131 is WARNING: LOAD WARNING - total load average: 3.59, 3.15, 2.23 [05:05:32] [Grafana] !sre RESOLVED: High Job Queue Backlog https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [05:07:33] PROBLEM - os131 Current Load on os131 is CRITICAL: LOAD CRITICAL - total load average: 4.56, 3.93, 2.76 [05:09:31] PROBLEM - os131 Current Load on os131 is WARNING: LOAD WARNING - total load average: 2.76, 3.47, 2.73 [05:11:28] RECOVERY - os131 Current Load on os131 is OK: LOAD OK - total load average: 1.58, 2.94, 2.63 [05:43:08] PROBLEM - alphabirdswiki.gq - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'alphabirdswiki.gq' expires in 15 day(s) (Sat 30 Dec 2023 05:39:12 GMT +0000). [06:13:48] PROBLEM - puppetdb121 Puppet on puppetdb121 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[puppetdb] [06:15:14] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [06:24:19] PROBLEM - db112 Disk Space on db112 is CRITICAL: DISK CRITICAL - free space: / 7915 MB (5% inode=99%); [06:25:25] PROBLEM - db112 Current Load on db112 is CRITICAL: CRITICAL - load average: 11.37, 8.02, 4.72 [06:27:25] PROBLEM - db112 Current Load on db112 is WARNING: WARNING - load average: 7.55, 7.43, 4.88 [06:29:25] PROBLEM - db112 Current Load on db112 is CRITICAL: CRITICAL - load average: 20.02, 13.33, 7.42 [06:35:43] PROBLEM - wizardia.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wizardia.wiki' expires in 15 day(s) (Sat 30 Dec 2023 06:22:18 GMT +0000). [06:35:59] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/436411db4748...72e44ef7b575 [06:36:02] [02miraheze/ssl] 07MirahezeSSLBot 0372e44ef - Bot: Update SSL cert for wizardia.wiki [06:38:44] PROBLEM - yoshipedia.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'yoshipedia.com' expires in 15 day(s) (Sat 30 Dec 2023 06:36:43 GMT +0000). [06:38:57] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/72e44ef7b575...32b62dc72899 [06:38:59] [02miraheze/ssl] 07MirahezeSSLBot 0332b62dc - Bot: Update SSL cert for yoshipedia.com [06:39:25] PROBLEM - db112 Current Load on db112 is WARNING: WARNING - load average: 3.78, 7.42, 7.23 [06:41:25] RECOVERY - db112 Current Load on db112 is OK: OK - load average: 3.20, 5.93, 6.70 [06:45:14] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.gab.pt.eu.org. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [06:48:20] PROBLEM - wiki.yunachannel.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.yunachannel.com' expires in 15 day(s) (Sat 30 Dec 2023 06:23:56 GMT +0000). [06:48:33] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/32b62dc72899...f7bb8ca0cf67 [06:48:34] [02miraheze/ssl] 07MirahezeSSLBot 03f7bb8ca - Bot: Update SSL cert for wiki.yunachannel.com [06:58:17] PROBLEM - db112 Current Load on db112 is CRITICAL: CRITICAL - load average: 42.00, 15.97, 9.28 [06:58:19] PROBLEM - db112 Disk Space on db112 is WARNING: DISK WARNING - free space: / 11367 MB (8% inode=99%); [07:03:31] PROBLEM - wiki.tmyt105.leyhp.com - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.tmyt105.leyhp.com. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [07:06:20] PROBLEM - db112 Disk Space on db112 is CRITICAL: DISK CRITICAL - free space: / 7651 MB (5% inode=99%); [07:07:46] RECOVERY - puppetdb121 Puppet on puppetdb121 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [07:07:57] RECOVERY - yoshipedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'yoshipedia.com' will expire on Wed 13 Mar 2024 05:38:50 GMT +0000. [07:16:20] PROBLEM - db112 Disk Space on db112 is WARNING: DISK WARNING - free space: / 13111 MB (9% inode=99%); [07:18:09] RECOVERY - wiki.yunachannel.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.yunachannel.com' will expire on Wed 13 Mar 2024 05:48:26 GMT +0000. [07:23:25] PROBLEM - db112 Current Load on db112 is WARNING: WARNING - load average: 0.53, 2.92, 7.24 [07:25:25] RECOVERY - db112 Current Load on db112 is OK: OK - load average: 1.13, 2.50, 6.57 [07:34:08] RECOVERY - wizardia.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'wizardia.wiki' will expire on Wed 13 Mar 2024 05:35:52 GMT +0000. [07:39:08] PROBLEM - wows.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wows.wiki' expires in 15 day(s) (Sat 30 Dec 2023 06:14:45 GMT +0000). [07:56:43] PROBLEM - wiki.kirbygang.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.kirbygang.com' expires in 15 day(s) (Sat 30 Dec 2023 07:51:20 GMT +0000). [07:56:56] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/f7bb8ca0cf67...dce56462dbd2 [07:56:57] [02miraheze/ssl] 07MirahezeSSLBot 03dce5646 - Bot: Update SSL cert for wiki.kirbygang.com [08:03:05] RECOVERY - wiki.tmyt105.leyhp.com - reverse DNS on sslhost is OK: SSL OK - wiki.tmyt105.leyhp.com reverse DNS resolves to cp25.miraheze.org - CNAME OK [08:09:08] PROBLEM - wows.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address wows.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [08:14:36] PROBLEM - null-cpu.emudev.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'null-cpu.emudev.org' expires in 15 day(s) (Sat 30 Dec 2023 07:47:31 GMT +0000). [08:14:48] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/dce56462dbd2...7991dd7e0145 [08:14:50] [02miraheze/ssl] 07MirahezeSSLBot 037991dd7 - Bot: Update SSL cert for null-cpu.emudev.org [08:34:48] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.51, 3.06, 2.19 [08:35:15] PROBLEM - wiki.coolstation.space - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.coolstation.space' expires in 15 day(s) (Sat 30 Dec 2023 08:15:29 GMT +0000). [08:35:30] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/7991dd7e0145...e007ef6ff6dd [08:35:33] [02miraheze/ssl] 07MirahezeSSLBot 03e007ef6 - Bot: Update SSL cert for wiki.coolstation.space [08:38:48] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 2.95, 3.19, 2.45 [08:43:41] RECOVERY - null-cpu.emudev.org - LetsEncrypt on sslhost is OK: OK - Certificate 'null-cpu.emudev.org' will expire on Wed 13 Mar 2024 07:14:41 GMT +0000. [08:50:22] PROBLEM - internalwiki.coolstation.space - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'internalwiki.coolstation.space' expires in 15 day(s) (Sat 30 Dec 2023 08:18:59 GMT +0000). [08:50:37] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/e007ef6ff6dd...073a44f214e8 [08:50:38] [02miraheze/ssl] 07MirahezeSSLBot 03073a44f - Bot: Update SSL cert for internalwiki.coolstation.space [08:51:51] PROBLEM - os141 PowerDNS Recursor on os141 is CRITICAL: CRITICAL - Plugin timed out while executing system call [08:53:50] RECOVERY - os141 PowerDNS Recursor on os141 is OK: DNS OK: 0.961 seconds response time. miraheze.org returns 2001:41d0:801:2000::3a18,2001:41d0:801:2000::5d68,51.195.201.140,51.89.139.24 [08:55:10] RECOVERY - wiki.kirbygang.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.kirbygang.com' will expire on Wed 13 Mar 2024 06:56:49 GMT +0000. [09:00:50] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.16, 3.65, 2.92 [09:02:49] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 2.86, 3.34, 2.90 [09:04:02] RECOVERY - wiki.coolstation.space - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.coolstation.space' will expire on Wed 13 Mar 2024 07:35:23 GMT +0000. [09:04:41] PROBLEM - mw141 Current Load on mw141 is CRITICAL: LOAD CRITICAL - total load average: 21.69, 12.26, 8.06 [09:06:20] PROBLEM - db131 Backups SQL on db131 is WARNING: FILE_AGE WARNING: /var/log/sql-backup.log is 864132 seconds old and 90365 bytes [09:06:39] PROBLEM - mw141 Current Load on mw141 is WARNING: LOAD WARNING - total load average: 9.76, 11.22, 8.21 [09:06:49] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 6.83, 5.70, 4.01 [09:08:37] RECOVERY - mw141 Current Load on mw141 is OK: LOAD OK - total load average: 4.95, 9.05, 7.78 [09:09:08] PROBLEM - wows.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wows.wiki' expires in 15 day(s) (Sat 30 Dec 2023 06:14:45 GMT +0000). [09:12:48] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 1.09, 3.64, 3.72 [09:14:48] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 0.52, 2.63, 3.34 [09:19:15] RECOVERY - internalwiki.coolstation.space - LetsEncrypt on sslhost is OK: OK - Certificate 'internalwiki.coolstation.space' will expire on Wed 13 Mar 2024 07:50:30 GMT +0000. [10:09:08] PROBLEM - wows.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address wows.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [10:13:48] PROBLEM - puppetdb121 Puppet on puppetdb121 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[puppetdb] [10:45:40] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 6.25, 4.49, 2.75 [10:46:57] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.86, 6.71, 5.63 [10:48:57] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 5.46, 6.24, 5.59 [10:51:07] PROBLEM - wiki.bloxotel.tk - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'wiki.bloxotel.tk' expires in 7 day(s) (Fri 22 Dec 2023 10:32:47 GMT +0000). [10:51:24] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 1.99, 3.59, 3.01 [10:53:21] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 1.28, 2.86, 2.81 [11:07:46] RECOVERY - puppetdb121 Puppet on puppetdb121 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [11:09:08] PROBLEM - wows.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wows.wiki' expires in 15 day(s) (Sat 30 Dec 2023 06:14:45 GMT +0000). [11:22:10] [02miraheze/mediawiki] 07paladox pushed 032 commits to 03REL1_40 [+0/-0/±2] 13https://github.com/miraheze/mediawiki/compare/cd7a1fbeda07...49067e39bb01 [11:22:13] [02miraheze/mediawiki] 07paladox 03f211047 - Update GlobalBlocking [11:22:14] [02miraheze/mediawiki] 07thiemowmde 0349067e3 - Make MagicWordArray not fail on old revs with broken UTF-8 [11:23:25] !log [paladox@mwtask141] starting deploy of {'world': True, 'l10n': True} to all [11:23:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:26:58] !log [paladox@mwtask141] finished deploy of {'world': True, 'l10n': True} to all - SUCCESS in 212s [11:27:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:29:07] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 10.64, 9.81, 7.92 [11:31:04] RECOVERY - mw131 Current Load on mw131 is OK: LOAD OK - total load average: 9.16, 9.30, 7.94 [11:36:20] [02miraheze/mediawiki] 07paladox pushed 03121 commits to 03REL1_41 [+25/-4/±697] 13https://github.com/miraheze/mediawiki/compare/950602755667...5c4f04369e38 [11:36:23] [02miraheze/mediawiki] 07cscott 03d0d7ae2 - Update wikimedia/parsoid to 0.18.0 [11:36:25] [02miraheze/mediawiki] 07reedy 030fab141 - Update git submodules [11:36:27] [02miraheze/mediawiki] 07umherirrender 035ba3788 - installer: Fix location of patch-pagelinks-target_id.sql for postgres [11:36:30] [02miraheze/mediawiki] ... and 118 more commits. [11:36:56] [02miraheze/mediawiki] 07paladox pushed 032 commits to 03REL1_41 [+0/-0/±4] 13https://github.com/miraheze/mediawiki/compare/5c4f04369e38...0525351f61c3 [11:36:59] [02miraheze/mediawiki] 07thiemowmde 033574909 - Make MagicWordArray not fail on old revs with broken UTF-8 [11:37:00] [02miraheze/mediawiki] 07paladox 030525351 - Merge remote-tracking branch 'upstream/heads/REL1_41' into REL1_41 [11:37:40] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_41 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/0525351f61c3...80fdfa4ddab0 [11:37:43] [02miraheze/mediawiki] 07paladox 0380fdfa4 - Update vendor [11:53:55] !log [paladox@test131] starting deploy of {'world': True, 'l10n': True} to all [11:54:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:55:30] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_40 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/49067e39bb01...d5073f68a222 [11:55:32] [02miraheze/mediawiki] 07paladox 03d5073f6 - SwiftFileBackend: Fix "PHP Deprecated: strlen(): Passing null to parameter #1 ($string) of type string is deprecated" [11:55:42] !log [paladox@mwtask141] starting deploy of {'world': True} to all [11:55:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:57:17] !log [paladox@mwtask141] finished deploy of {'world': True} to all - SUCCESS in 94s [11:57:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:05:27] !log [paladox@test131] finished deploy of {'world': True, 'l10n': True} to all - SUCCESS in 691s [12:05:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:09:08] PROBLEM - wows.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address wows.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [12:11:26] [02miraheze/MirahezeMagic] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/MirahezeMagic/compare/37a95db7250c...2f4bd04ed69a [12:11:29] [02miraheze/MirahezeMagic] 07translatewiki 032f4bd04 - Localisation updates from https://translatewiki.net. [12:11:30] [02miraheze/ManageWiki] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ManageWiki/compare/8c58f219c3e3...a20129abed75 [12:11:33] [02miraheze/ManageWiki] 07translatewiki 03a20129a - Localisation updates from https://translatewiki.net. [12:11:34] [02miraheze/SpriteSheet] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/SpriteSheet/compare/eefb3bbe4c78...44d95a0902c9 [12:11:37] [02miraheze/SpriteSheet] 07translatewiki 0344d95a0 - Localisation updates from https://translatewiki.net. [12:11:40] [02miraheze/landing] 07translatewiki pushed 031 commit to 03master [+1/-0/±0] 13https://github.com/miraheze/landing/compare/b45b88564ce8...dedcf786d911 [12:11:42] [02miraheze/landing] 07translatewiki 03dedcf78 - Localisation updates from https://translatewiki.net. [12:12:26] miraheze/landing - translatewiki the build passed. [12:16:51] miraheze/ManageWiki - translatewiki the build passed. [12:17:17] miraheze/MirahezeMagic - translatewiki the build has errored. [12:18:01] miraheze/SpriteSheet - translatewiki the build passed. [13:41:18] PROBLEM - cloud12 Puppet on cloud12 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [13:45:13] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [14:09:08] PROBLEM - wows.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wows.wiki' expires in 15 day(s) (Sat 30 Dec 2023 06:14:45 GMT +0000). [14:09:18] RECOVERY - cloud12 Puppet on cloud12 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [14:13:48] PROBLEM - puppetdb121 Puppet on puppetdb121 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[puppetdb] [14:15:14] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns.ankh.fr.eu.org.', 'ns1.eu.org.', 'ns1.eriomem.net.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [14:30:21] PROBLEM - mw133 Current Load on mw133 is CRITICAL: LOAD CRITICAL - total load average: 12.79, 10.46, 8.81 [14:32:21] RECOVERY - mw133 Current Load on mw133 is OK: LOAD OK - total load average: 8.56, 9.49, 8.65 [14:39:08] PROBLEM - wows.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address wows.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [14:48:48] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.98, 3.63, 2.51 [14:50:26] PROBLEM - os141 SSH on os141 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:50:57] PROBLEM - mw134 Current Load on mw134 is CRITICAL: LOAD CRITICAL - total load average: 16.42, 11.22, 9.27 [14:52:22] RECOVERY - os141 SSH on os141 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u1 (protocol 2.0) [14:52:39] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 9.53, 10.86, 9.61 [14:52:49] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.39, 3.84, 2.87 [14:52:57] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 9.69, 10.63, 9.31 [14:54:48] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 1.91, 3.17, 2.75 [14:56:35] RECOVERY - mw132 Current Load on mw132 is OK: LOAD OK - total load average: 8.88, 10.07, 9.58 [14:58:57] PROBLEM - mw134 Current Load on mw134 is CRITICAL: LOAD CRITICAL - total load average: 12.16, 11.01, 9.88 [15:02:57] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 7.26, 10.67, 10.16 [15:06:57] RECOVERY - mw134 Current Load on mw134 is OK: LOAD OK - total load average: 6.48, 8.98, 9.62 [15:07:46] RECOVERY - puppetdb121 Puppet on puppetdb121 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:51:40] PROBLEM - cloud11 Puppet on cloud11 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [15:57:38] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.58, 7.09, 6.27 [15:59:32] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.48, 6.42, 6.12 [16:13:53] PROBLEM - mw134 Current Load on mw134 is CRITICAL: LOAD CRITICAL - total load average: 14.08, 11.35, 9.14 [16:14:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.84, 6.98, 6.43 [16:15:49] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 11.33, 10.64, 9.10 [16:16:36] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 11.95, 10.65, 9.00 [16:16:48] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.39, 6.34, 6.25 [16:18:34] RECOVERY - mw132 Current Load on mw132 is OK: LOAD OK - total load average: 8.23, 9.86, 8.93 [16:19:43] RECOVERY - mw134 Current Load on mw134 is OK: LOAD OK - total load average: 8.88, 9.57, 9.05 [16:21:40] RECOVERY - cloud11 Puppet on cloud11 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:33:10] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 11.29, 9.74, 9.23 [16:37:05] RECOVERY - mw132 Current Load on mw132 is OK: LOAD OK - total load average: 9.30, 9.55, 9.27 [16:39:08] PROBLEM - wows.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wows.wiki' expires in 15 day(s) (Sat 30 Dec 2023 06:14:45 GMT +0000). [16:52:46] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 9.01, 10.40, 9.57 [16:52:57] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 11.07, 10.50, 9.00 [16:54:57] PROBLEM - mw134 Current Load on mw134 is CRITICAL: LOAD CRITICAL - total load average: 12.25, 10.94, 9.33 [16:56:37] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 10.35, 10.01, 8.66 [16:56:41] PROBLEM - mw132 Current Load on mw132 is CRITICAL: LOAD CRITICAL - total load average: 15.08, 11.59, 10.17 [16:58:34] PROBLEM - mw133 Current Load on mw133 is CRITICAL: LOAD CRITICAL - total load average: 12.48, 11.10, 9.23 [16:59:26] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 9.70, 10.82, 9.18 [17:00:30] RECOVERY - mw133 Current Load on mw133 is OK: LOAD OK - total load average: 8.12, 9.80, 8.97 [17:00:36] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 11.42, 11.97, 10.76 [17:00:57] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 9.03, 11.70, 10.48 [17:03:20] RECOVERY - mw131 Current Load on mw131 is OK: LOAD OK - total load average: 8.15, 9.75, 9.15 [17:04:11] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 4.44, 3.58, 2.93 [17:04:54] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.22, 2.78, 1.77 [17:04:57] PROBLEM - mw134 Current Load on mw134 is CRITICAL: LOAD CRITICAL - total load average: 14.39, 12.71, 11.08 [17:06:05] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 3.22, 3.28, 2.89 [17:06:29] PROBLEM - mw132 Current Load on mw132 is CRITICAL: LOAD CRITICAL - total load average: 17.10, 12.39, 11.09 [17:06:48] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.75, 3.20, 2.05 [17:08:21] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 10.11, 10.57, 9.57 [17:08:27] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 9.51, 11.57, 10.99 [17:08:49] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.44, 3.47, 2.28 [17:09:08]