[00:02:03] PROBLEM - cp24 Varnish Backends on cp24 is CRITICAL: 2 backends are down. mw131 mw142 [00:02:20] PROBLEM - cp25 Varnish Backends on cp25 is CRITICAL: 1 backends are down. mw131 [00:02:52] PROBLEM - cp35 Varnish Backends on cp35 is CRITICAL: 3 backends are down. mw131 mw133 mw134 [00:03:22] PROBLEM - cp34 Varnish Backends on cp34 is CRITICAL: 1 backends are down. mw134 [00:03:26] PROBLEM - db112 Current Load on db112 is CRITICAL: CRITICAL - load average: 13.25, 7.12, 3.32 [00:03:29] PROBLEM - mw143 Current Load on mw143 is CRITICAL: LOAD CRITICAL - total load average: 17.31, 12.94, 10.19 [00:04:02] PROBLEM - matomo121 Current Load on matomo121 is WARNING: LOAD WARNING - total load average: 3.78, 2.98, 1.86 [00:05:50] RECOVERY - cp24 Varnish Backends on cp24 is OK: All 15 backends are healthy [00:06:02] RECOVERY - matomo121 Current Load on matomo121 is OK: LOAD OK - total load average: 2.56, 2.67, 1.87 [00:06:20] RECOVERY - cp25 Varnish Backends on cp25 is OK: All 15 backends are healthy [00:06:44] RECOVERY - cp35 Varnish Backends on cp35 is OK: All 15 backends are healthy [00:07:16] RECOVERY - cp34 Varnish Backends on cp34 is OK: All 15 backends are healthy [00:07:25] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.01, 6.63, 5.94 [00:08:10] PROBLEM - mw141 Current Load on mw141 is WARNING: LOAD WARNING - total load average: 6.15, 11.00, 11.29 [00:09:20] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.87, 6.24, 5.87 [00:09:27] PROBLEM - os131 Current Load on os131 is CRITICAL: LOAD CRITICAL - total load average: 4.94, 3.79, 2.44 [00:09:29] RECOVERY - mw143 Current Load on mw143 is OK: LOAD OK - total load average: 4.73, 9.63, 9.99 [00:09:38] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [00:09:56] PROBLEM - mw142 Current Load on mw142 is WARNING: LOAD WARNING - total load average: 6.80, 10.29, 10.93 [00:11:25] PROBLEM - db112 Current Load on db112 is WARNING: WARNING - load average: 3.25, 7.20, 5.59 [00:11:28] PROBLEM - os131 Current Load on os131 is WARNING: LOAD WARNING - total load average: 3.75, 3.77, 2.61 [00:12:10] RECOVERY - mw141 Current Load on mw141 is OK: LOAD OK - total load average: 5.14, 8.04, 10.02 [00:13:25] RECOVERY - db112 Current Load on db112 is OK: OK - load average: 2.50, 5.64, 5.21 [00:13:27] RECOVERY - os131 Current Load on os131 is OK: LOAD OK - total load average: 2.36, 3.32, 2.59 [00:13:53] RECOVERY - mw142 Current Load on mw142 is OK: LOAD OK - total load average: 6.21, 8.22, 9.93 [00:14:37] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [00:16:50] PROBLEM - matomo121 Current Load on matomo121 is WARNING: LOAD WARNING - total load average: 3.66, 3.31, 2.58 [00:18:21] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 7.03, 8.32, 11.45 [00:18:33] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 6.97, 8.47, 11.58 [00:18:47] RECOVERY - matomo121 Current Load on matomo121 is OK: LOAD OK - total load average: 2.97, 3.26, 2.66 [00:19:46] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 6.34, 8.13, 11.74 [00:20:57] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 5.75, 8.22, 11.82 [00:24:21] RECOVERY - mw133 Current Load on mw133 is OK: LOAD OK - total load average: 7.54, 7.83, 10.20 [00:24:33] RECOVERY - mw131 Current Load on mw131 is OK: LOAD OK - total load average: 8.10, 7.56, 10.07 [00:26:34] PROBLEM - matomo121 Current Load on matomo121 is WARNING: LOAD WARNING - total load average: 3.42, 3.32, 2.87 [00:27:46] RECOVERY - mw132 Current Load on mw132 is OK: LOAD OK - total load average: 6.05, 7.29, 9.93 [00:28:30] RECOVERY - matomo121 Current Load on matomo121 is OK: LOAD OK - total load average: 3.37, 3.33, 2.93 [00:28:57] RECOVERY - mw134 Current Load on mw134 is OK: LOAD OK - total load average: 7.96, 7.41, 9.96 [00:33:30] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.95, 6.61, 5.86 [00:35:26] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 4.84, 6.02, 5.73 [00:42:18] PROBLEM - os131 Current Load on os131 is CRITICAL: LOAD CRITICAL - total load average: 6.00, 4.46, 3.23 [00:50:29] PROBLEM - os141 PowerDNS Recursor on os141 is CRITICAL: CRITICAL - Plugin timed out while executing system call [00:50:30] [Grafana] !sre FIRING: The mediawiki job queue has more than 2500 unclaimed jobs https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [00:50:58] PROBLEM - mw133 Current Load on mw133 is CRITICAL: LOAD CRITICAL - total load average: 12.46, 10.65, 9.66 [00:52:27] RECOVERY - os141 PowerDNS Recursor on os141 is OK: DNS OK: 0.530 seconds response time. miraheze.org returns 2001:41d0:801:2000::3a18,2001:41d0:801:2000::5d68,51.195.201.140,51.89.139.24 [00:52:54] RECOVERY - mw133 Current Load on mw133 is OK: LOAD OK - total load average: 8.46, 9.92, 9.53 [00:54:14] PROBLEM - os131 Current Load on os131 is WARNING: LOAD WARNING - total load average: 3.35, 3.90, 3.70 [01:00:07] RECOVERY - os131 Current Load on os131 is OK: LOAD OK - total load average: 2.32, 2.84, 3.28 [01:02:53] PROBLEM - os141 PowerDNS Recursor on os141 is CRITICAL: CRITICAL - Plugin timed out while executing system call [01:03:22] PROBLEM - mw142 Current Load on mw142 is WARNING: LOAD WARNING - total load average: 11.87, 11.87, 8.63 [01:03:39] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.22, 3.96, 3.29 [01:04:57] RECOVERY - os141 PowerDNS Recursor on os141 is OK: DNS OK: 5.002 seconds response time. miraheze.org returns 2001:41d0:801:2000::3a18,2001:41d0:801:2000::5d68,51.195.201.140,51.89.139.24 [01:05:21] RECOVERY - mw142 Current Load on mw142 is OK: LOAD OK - total load average: 6.66, 9.91, 8.29 [01:05:35] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 2.86, 3.55, 3.22 [01:06:57] PROBLEM - matomo121 Current Load on matomo121 is WARNING: LOAD WARNING - total load average: 3.42, 3.66, 3.26 [01:07:30] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.20, 3.75, 3.32 [01:12:46] RECOVERY - matomo121 Current Load on matomo121 is OK: LOAD OK - total load average: 2.40, 3.08, 3.15 [01:17:05] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.09, 3.82, 3.68 [01:19:03] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 3.82, 4.09, 3.81 [01:20:58] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 2.85, 3.76, 3.73 [01:24:53] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.45, 3.83, 3.74 [01:26:49] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.74, 3.82, 3.75 [01:28:48] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.09, 3.90, 3.78 [01:30:48] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.21, 3.62, 3.70 [01:38:49] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.83, 3.86, 3.70 [01:39:08] PROBLEM - wows.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address wows.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [01:40:05] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.53, 7.11, 6.58 [01:41:59] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.80, 7.21, 6.67 [01:42:49] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.21, 3.90, 3.79 [01:43:27] PROBLEM - mw134 Current Load on mw134 is CRITICAL: LOAD CRITICAL - total load average: 16.24, 11.67, 9.25 [01:43:53] PROBLEM - mw132 Current Load on mw132 is CRITICAL: LOAD CRITICAL - total load average: 12.80, 10.40, 8.67 [01:44:33] PROBLEM - mw131 Current Load on mw131 is CRITICAL: LOAD CRITICAL - total load average: 14.13, 10.31, 8.33 [01:45:48] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.00, 6.26, 6.42 [01:46:21] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 10.27, 9.37, 7.98 [01:48:21] PROBLEM - mw133 Current Load on mw133 is CRITICAL: LOAD CRITICAL - total load average: 17.69, 12.33, 9.23 [01:50:10] PROBLEM - mw141 Current Load on mw141 is WARNING: LOAD WARNING - total load average: 10.33, 9.85, 8.55 [01:52:10] PROBLEM - mw141 Current Load on mw141 is CRITICAL: LOAD CRITICAL - total load average: 14.44, 11.49, 9.30 [01:53:29] PROBLEM - mw143 Current Load on mw143 is CRITICAL: LOAD CRITICAL - total load average: 12.88, 10.92, 8.69 [01:53:49] PROBLEM - mw142 Current Load on mw142 is CRITICAL: LOAD CRITICAL - total load average: 12.51, 10.77, 8.56 [01:54:48] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.08, 3.52, 3.48 [01:55:48] PROBLEM - mw142 Current Load on mw142 is WARNING: LOAD WARNING - total load average: 10.38, 10.85, 8.87 [01:56:48] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 2.35, 3.13, 3.34 [01:57:46] PROBLEM - mw142 Current Load on mw142 is CRITICAL: LOAD CRITICAL - total load average: 16.00, 12.20, 9.56 [01:59:29] PROBLEM - mw143 Current Load on mw143 is WARNING: LOAD WARNING - total load average: 11.12, 11.65, 9.80 [02:02:15] PROBLEM - cp24 Varnish Backends on cp24 is CRITICAL: 4 backends are down. mw132 mw142 mw133 mw134 [02:02:49] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.49, 4.18, 3.70 [02:03:56] PROBLEM - cp35 Varnish Backends on cp35 is CRITICAL: 2 backends are down. mw142 mw133 [02:03:59] PROBLEM - cp34 Varnish Backends on cp34 is CRITICAL: 2 backends are down. mw142 mw133 [02:04:01] PROBLEM - cp25 Varnish Backends on cp25 is CRITICAL: 3 backends are down. mw142 mw133 mw134 [02:04:48] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 1.14, 3.03, 3.34 [02:05:29] PROBLEM - mw143 Current Load on mw143 is CRITICAL: LOAD CRITICAL - total load average: 18.77, 13.15, 10.77 [02:09:47] RECOVERY - cp25 Varnish Backends on cp25 is OK: All 15 backends are healthy [02:09:50] RECOVERY - cp24 Varnish Backends on cp24 is OK: All 15 backends are healthy [02:09:51] RECOVERY - cp34 Varnish Backends on cp34 is OK: All 15 backends are healthy [02:09:56] RECOVERY - cp35 Varnish Backends on cp35 is OK: All 15 backends are healthy [02:13:48] PROBLEM - puppetdb121 Puppet on puppetdb121 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[puppetdb] [02:15:14] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [02:15:30] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1[Grafana] !sre FIRING: The mediawiki job queue has more than 2500 unclaimed jobs https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [02:17:29] PROBLEM - mw143 Current Load on mw143 is WARNING: LOAD WARNING - total load average: 8.79, 11.50, 11.92 [02:21:29] PROBLEM - mw143 Current Load on mw143 is CRITICAL: LOAD CRITICAL - total load average: 13.15, 10.51, 11.30 [02:21:56] PROBLEM - cp35 Varnish Backends on cp35 is CRITICAL: 1 backends are down. mw133 [02:23:56] RECOVERY - cp35 Varnish Backends on cp35 is OK: All 15 backends are healthy [02:29:29] PROBLEM - mw143 Current Load on mw143 is WARNING: LOAD WARNING - total load average: 5.28, 9.88, 11.26 [02:30:30] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1[Grafana] !sre FIRING: The mediawiki job queue has more than 2500 unclaimed jobs https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [02:33:29] RECOVERY - mw143 Current Load on mw143 is OK: LOAD OK - total load average: 6.77, 7.88, 10.08 [02:34:10] PROBLEM - mw141 Current Load on mw141 is WARNING: LOAD WARNING - total load average: 6.85, 9.11, 11.81 [02:35:24] PROBLEM - mw142 Current Load on mw142 is WARNING: LOAD WARNING - total load average: 6.53, 8.28, 11.68 [02:41:19] RECOVERY - mw142 Current Load on mw142 is OK: LOAD OK - total load average: 5.60, 7.12, 10.11 [02:42:10] RECOVERY - mw141 Current Load on mw141 is OK: LOAD OK - total load average: 4.72, 6.63, 9.60 [02:45:14] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns.ankh.fr.eu.org.', 'ns1.eu.org.', 'ns1.eriomem.net.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [02:48:21] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 7.88, 8.85, 11.67 [02:48:33] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 7.72, 8.38, 11.38 [02:50:48] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.43, 6.53, 6.25 [02:51:46] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 6.91, 9.11, 11.57 [02:52:48] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.16, 6.82, 6.37 [02:52:57] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 7.22, 8.59, 11.71 [02:54:21] RECOVERY - mw133 Current Load on mw133 is OK: LOAD OK - total load average: 4.95, 6.91, 9.98 [02:54:33] RECOVERY - mw131 Current Load on mw131 is OK: LOAD OK - total load average: 6.63, 7.27, 9.96 [02:54:48] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.36, 6.59, 6.34 [02:57:46] RECOVERY - mw132 Current Load on mw132 is OK: LOAD OK - total load average: 7.37, 7.78, 10.16 [03:00:57] RECOVERY - mw134 Current Load on mw134 is OK: LOAD OK - total load average: 7.46, 7.44, 9.86 [03:07:46] RECOVERY - puppetdb121 Puppet on puppetdb121 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [03:20:30] [Grafana] !sre RESOLVED: High Job Queue Backlog https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [03:22:11] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 12.92, 8.33, 6.95 [03:23:20] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.49, 6.27, 5.80 [03:24:06] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.75, 7.60, 6.84 [03:25:17] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 6.07, 6.02, 5.76 [03:27:54] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.67, 6.68, 6.63 [03:34:33] PROBLEM - mw131 Current Load on mw131 is CRITICAL: LOAD CRITICAL - total load average: 18.89, 12.16, 9.34 [03:34:41] PROBLEM - mw142 Current Load on mw142 is CRITICAL: LOAD CRITICAL - total load average: 13.94, 9.49, 6.99 [03:35:36] PROBLEM - mw134 Current Load on mw134 is CRITICAL: LOAD CRITICAL - total load average: 22.72, 15.69, 11.18 [03:35:46] PROBLEM - mw132 Current Load on mw132 is CRITICAL: LOAD CRITICAL - total load average: 23.60, 15.24, 10.27 [03:36:21] PROBLEM - mw133 Current Load on mw133 is CRITICAL: LOAD CRITICAL - total load average: 20.35, 14.41, 9.89 [03:36:25] PROBLEM - mw141 Current Load on mw141 is CRITICAL: LOAD CRITICAL - total load average: 18.36, 11.74, 8.14 [03:36:47] PROBLEM - mw143 Current Load on mw143 is CRITICAL: LOAD CRITICAL - total load average: 13.64, 10.00, 7.39 [03:38:23] PROBLEM - mw141 Current Load on mw141 is WARNING: LOAD WARNING - total load average: 11.23, 11.77, 8.63 [03:38:38] PROBLEM - mw142 Current Load on mw142 is WARNING: LOAD WARNING - total load average: 10.52, 11.48, 8.47 [03:38:46] RECOVERY - mw143 Current Load on mw143 is OK: LOAD OK - total load average: 9.98, 9.87, 7.66 [03:42:19] PROBLEM - mw141 Current Load on mw141 is CRITICAL: LOAD CRITICAL - total load average: 13.88, 11.57, 9.17 [03:42:35] PROBLEM - mw142 Current Load on mw142 is CRITICAL: LOAD CRITICAL - total load average: 16.91, 12.46, 9.39 [03:43:09] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.23, 7.00, 6.72 [03:44:34] PROBLEM - mw142 Current Load on mw142 is WARNING: LOAD WARNING - total load average: 10.44, 11.51, 9.42 [03:44:40] PROBLEM - mw143 Current Load on mw143 is WARNING: LOAD WARNING - total load average: 9.67, 10.66, 8.74 [03:45:03] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.38, 6.43, 6.55 [03:46:15] RECOVERY - mw141 Current Load on mw141 is OK: LOAD OK - total load average: 6.88, 10.11, 9.24 [03:46:33] RECOVERY - mw142 Current Load on mw142 is OK: LOAD OK - total load average: 6.19, 9.50, 8.92 [03:46:39] RECOVERY - mw143 Current Load on mw143 is OK: LOAD OK - total load average: 6.01, 9.02, 8.38 [03:52:21] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 8.26, 10.99, 11.95 [03:54:21] PROBLEM - mw133 Current Load on mw133 is CRITICAL: LOAD CRITICAL - total load average: 19.60, 14.41, 13.09 [03:58:22] PROBLEM - mw142 Current Load on mw142 is CRITICAL: LOAD CRITICAL - total load average: 16.96, 13.74, 10.91 [03:59:59] PROBLEM - mw141 Current Load on mw141 is CRITICAL: LOAD CRITICAL - total load average: 13.05, 11.93, 10.33 [04:01:57] PROBLEM - mw141 Current Load on mw141 is WARNING: LOAD WARNING - total load average: 7.94, 10.99, 10.23 [04:02:19] PROBLEM - mw142 Current Load on mw142 is WARNING: LOAD WARNING - total load average: 7.69, 11.61, 10.75 [04:05:53] PROBLEM - mw141 Current Load on mw141 is CRITICAL: LOAD CRITICAL - total load average: 14.10, 12.13, 10.82 [04:07:51] PROBLEM - mw141 Current Load on mw141 is WARNING: LOAD WARNING - total load average: 8.30, 11.02, 10.59 [04:09:49] RECOVERY - mw141 Current Load on mw141 is OK: LOAD OK - total load average: 5.44, 9.09, 9.94 [04:10:13] RECOVERY - mw142 Current Load on mw142 is OK: LOAD OK - total load average: 7.02, 9.02, 9.94 [04:14:02] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.10, 6.63, 6.50 [04:15:56] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.19, 6.38, 6.42 [04:18:33] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 6.33, 9.17, 11.70 [04:20:21] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 9.18, 9.86, 11.66 [04:20:33] PROBLEM - mw131 Current Load on mw131 is CRITICAL: LOAD CRITICAL - total load average: 14.78, 10.48, 11.81 [04:20:35] [Grafana] !sre FIRING: The mediawiki job queue has more than 2500 unclaimed jobs https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [04:22:21] PROBLEM - mw133 Current Load on mw133 is CRITICAL: LOAD CRITICAL - total load average: 19.55, 13.45, 12.74 [04:23:29] PROBLEM - mw143 Current Load on mw143 is WARNING: LOAD WARNING - total load average: 11.52, 9.82, 8.23 [04:24:02] PROBLEM - mw142 Current Load on mw142 is WARNING: LOAD WARNING - total load average: 10.51, 9.52, 9.05 [04:24:32] PROBLEM - mw141 Current Load on mw141 is CRITICAL: LOAD CRITICAL - total load average: 12.53, 11.64, 10.09 [04:25:29] PROBLEM - mw143 Current Load on mw143 is CRITICAL: LOAD CRITICAL - total load average: 12.38, 10.41, 8.62 [04:26:01] PROBLEM - mw142 Current Load on mw142 is CRITICAL: LOAD CRITICAL - total load average: 14.92, 11.37, 9.77 [04:29:29] PROBLEM - mw143 Current Load on mw143 is WARNING: LOAD WARNING - total load average: 11.47, 10.81, 9.20 [04:33:29] PROBLEM - mw143 Current Load on mw143 is CRITICAL: LOAD CRITICAL - total load average: 12.18, 11.11, 9.71 [04:35:29] PROBLEM - mw143 Current Load on mw143 is WARNING: LOAD WARNING - total load average: 8.43, 10.49, 9.68 [04:37:39] PROBLEM - cp34 Varnish Backends on cp34 is CRITICAL: 3 backends are down. mw141 mw142 mw143 [04:37:56] PROBLEM - cp35 Varnish Backends on cp35 is CRITICAL: 1 backends are down. mw143 [04:39:29] RECOVERY - mw143 Current Load on mw143 is OK: LOAD OK - total load average: 9.06, 10.14, 9.79 [04:39:36] RECOVERY - cp34 Varnish Backends on cp34 is OK: All 15 backends are healthy [04:39:51] PROBLEM - mw142 Current Load on mw142 is WARNING: LOAD WARNING - total load average: 9.47, 11.56, 11.56 [04:39:56] RECOVERY - cp35 Varnish Backends on cp35 is OK: All 15 backends are healthy [04:43:48] PROBLEM - mw142 Current Load on mw142 is CRITICAL: LOAD CRITICAL - total load average: 13.78, 12.15, 11.75 [04:45:29] PROBLEM - mw143 Current Load on mw143 is WARNING: LOAD WARNING - total load average: 9.11, 10.56, 10.14 [04:45:47] PROBLEM - mw142 Current Load on mw142 is WARNING: LOAD WARNING - total load average: 9.90, 11.11, 11.41 [04:47:29] PROBLEM - mw143 Current Load on mw143 is CRITICAL: LOAD CRITICAL - total load average: 12.24, 10.85, 10.27 [04:49:29] PROBLEM - mw143 Current Load on mw143 is WARNING: LOAD WARNING - total load average: 11.52, 11.05, 10.42 [04:50:10] PROBLEM - mw141 Current Load on mw141 is WARNING: LOAD WARNING - total load average: 9.68, 11.25, 11.87 [04:50:48] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.36, 7.03, 6.53 [04:51:29] RECOVERY - mw143 Current Load on mw143 is OK: LOAD OK - total load average: 8.43, 9.98, 10.10 [04:51:43] PROBLEM - mw142 Current Load on mw142 is CRITICAL: LOAD CRITICAL - total load average: 13.13, 11.08, 11.20 [04:52:10] PROBLEM - mw141 Current Load on mw141 is CRITICAL: LOAD CRITICAL - total load average: 13.18, 12.37, 12.23 [04:52:48] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.81, 6.59, 6.43 [04:55:29] PROBLEM - mw143 Current Load on mw143 is WARNING: LOAD WARNING - total load average: 10.49, 10.60, 10.37 [04:55:35] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1[Grafana] !sre FIRING: The mediawiki job queue has more than 2500 unclaimed jobs https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [04:55:40] PROBLEM - mw142 Current Load on mw142 is WARNING: LOAD WARNING - total load average: 11.60, 11.34, 11.27 [04:57:29] RECOVERY - mw143 Current Load on mw143 is OK: LOAD OK - total load average: 7.86, 9.55, 10.01 [04:58:48] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.46, 6.90, 6.65 [04:59:37] PROBLEM - mw142 Current Load on mw142 is CRITICAL: LOAD CRITICAL - total load average: 12.52, 11.18, 11.13 [05:00:48] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.83, 6.55, 6.55 [05:01:36] PROBLEM - mw142 Current Load on mw142 is WARNING: LOAD WARNING - total load average: 11.70, 11.35, 11.20 [05:03:35] PROBLEM - mw142 Current Load on mw142 is CRITICAL: LOAD CRITICAL - total load average: 13.47, 12.62, 11.72 [05:07:37] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.92, 6.67, 6.56 [05:08:10] PROBLEM - mw141 Current Load on mw141 is WARNING: LOAD WARNING - total load average: 9.16, 11.31, 11.91 [05:09:31] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.26, 7.05, 6.70 [05:10:10] PROBLEM - mw141 Current Load on mw141 is CRITICAL: LOAD CRITICAL - total load average: 12.48, 12.10, 12.14 [05:11:25] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.73, 7.14, 6.79 [05:11:29] PROBLEM - mw142 Current Load on mw142 is WARNING: LOAD WARNING - total load average: 8.84, 11.64, 11.78 [05:12:10] PROBLEM - mw141 Current Load on mw141 is WARNING: LOAD WARNING - total load average: 9.99, 10.99, 11.72 [05:13:28] PROBLEM - mw142 Current Load on mw142 is CRITICAL: LOAD CRITICAL - total load average: 15.63, 13.34, 12.37 [05:15:14] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.31, 6.73, 6.70 [05:16:10] PROBLEM - mw141 Current Load on mw141 is CRITICAL: LOAD CRITICAL - total load average: 12.39, 11.41, 11.72 [05:18:10] PROBLEM - mw141 Current Load on mw141 is WARNING: LOAD WARNING - total load average: 8.92, 10.59, 11.40 [05:21:22] PROBLEM - mw142 Current Load on mw142 is WARNING: LOAD WARNING - total load average: 10.71, 11.50, 11.87 [05:22:51] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.69, 7.13, 6.79 [05:24:23] PROBLEM - mw143 Current Load on mw143 is WARNING: LOAD WARNING - total load average: 10.70, 10.66, 10.26 [05:24:48] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.30, 7.04, 6.79 [05:26:48] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.79, 6.79, 6.75 [05:28:21] RECOVERY - mw143 Current Load on mw143 is OK: LOAD OK - total load average: 9.28, 9.78, 10.00 [05:29:17] PROBLEM - mw142 Current Load on mw142 is CRITICAL: LOAD CRITICAL - total load average: 13.13, 10.92, 11.26 [05:33:14] PROBLEM - mw142 Current Load on mw142 is WARNING: LOAD WARNING - total load average: 11.16, 11.57, 11.50 [05:38:10] RECOVERY - mw141 Current Load on mw141 is OK: LOAD OK - total load average: 5.36, 8.53, 10.13 [05:39:10] RECOVERY - mw142 Current Load on mw142 is OK: LOAD OK - total load average: 5.78, 8.14, 10.09 [05:40:35] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1[Grafana] !sre FIRING: The mediawiki job queue has more than 2500 unclaimed jobs https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [05:46:21] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 5.00, 6.61, 11.38 [05:48:33] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 5.86, 6.76, 11.45 [05:48:57] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 5.41, 6.11, 11.79 [05:49:46] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 7.05, 6.82, 11.45 [05:50:21] RECOVERY - mw133 Current Load on mw133 is OK: LOAD OK - total load average: 4.50, 5.65, 9.90 [05:52:33] RECOVERY - mw131 Current Load on mw131 is OK: LOAD OK - total load average: 5.28, 5.95, 10.05 [05:52:57] RECOVERY - mw134 Current Load on mw134 is OK: LOAD OK - total load average: 4.46, 5.17, 10.10 [05:53:46] RECOVERY - mw132 Current Load on mw132 is OK: LOAD OK - total load average: 5.55, 6.21, 10.14 [06:13:48] PROBLEM - puppetdb121 Puppet on puppetdb121 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[puppetdb] [06:15:35] [Grafana] !sre RESOLVED: High Job Queue Backlog https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [06:43:46] PROBLEM - mw132 Current Load on mw132 is CRITICAL: LOAD CRITICAL - total load average: 13.65, 10.87, 8.91 [06:46:21] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 11.02, 9.97, 8.40 [06:46:45] PROBLEM - mw134 Current Load on mw134 is CRITICAL: LOAD CRITICAL - total load average: 12.10, 11.51, 9.54 [06:47:46] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 11.40, 11.67, 9.76 [06:48:21] RECOVERY - mw133 Current Load on mw133 is OK: LOAD OK - total load average: 9.55, 9.66, 8.47 [06:48:42] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 11.35, 11.75, 9.89 [06:49:46] PROBLEM - mw132 Current Load on mw132 is CRITICAL: LOAD CRITICAL - total load average: 12.69, 12.23, 10.22 [06:50:20] PROBLEM - mw131 Current Load on mw131 is CRITICAL: LOAD CRITICAL - total load average: 12.24, 10.83, 9.06 [06:50:39] PROBLEM - mw134 Current Load on mw134 is CRITICAL: LOAD CRITICAL - total load average: 14.91, 13.02, 10.57 [06:50:48] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.10, 7.07, 6.54 [06:52:21] PROBLEM - mw133 Current Load on mw133 is CRITICAL: LOAD CRITICAL - total load average: 13.19, 11.63, 9.58 [06:52:48] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.09, 6.49, 6.40 [06:54:29] PROBLEM - mw142 Current Load on mw142 is WARNING: LOAD WARNING - total load average: 10.22, 9.39, 7.59 [06:56:29] RECOVERY - mw142 Current Load on mw142 is OK: LOAD OK - total load average: 7.98, 8.90, 7.63 [07:07:46] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 8.68, 11.46, 11.85 [07:07:46] RECOVERY - puppetdb121 Puppet on puppetdb121 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:07:56] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 10.05, 11.60, 11.68 [07:08:21] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 8.85, 11.07, 11.24 [07:09:08] PROBLEM - wows.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wows.wiki' expires in 13 day(s) (Sat 30 Dec 2023 06:14:45 GMT +0000). [07:15:33] [Grafana] !sre FIRING: The mediawiki job queue has more than 2500 unclaimed jobs https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [07:16:21] RECOVERY - mw133 Current Load on mw133 is OK: LOAD OK - total load average: 7.45, 8.48, 10.00 [07:17:46] RECOVERY - mw132 Current Load on mw132 is OK: LOAD OK - total load average: 5.64, 8.03, 10.02 [07:19:48] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 5.41, 8.66, 11.45 [07:21:38] RECOVERY - mw131 Current Load on mw131 is OK: LOAD OK - total load average: 4.88, 7.49, 9.71 [07:22:02] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.11, 7.04, 6.50 [07:23:56] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.29, 6.66, 6.42 [07:27:34] RECOVERY - mw134 Current Load on mw134 is OK: LOAD OK - total load average: 6.73, 7.97, 10.02 [07:39:08] PROBLEM - wows.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address wows.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [07:43:32] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.00, 6.66, 6.48 [07:45:26] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.19, 6.44, 6.42 [07:50:33] [Grafana] !sre RESOLVED: High Job Queue Backlog https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [08:20:57] PROBLEM - mw134 Current Load on mw134 is CRITICAL: LOAD CRITICAL - total load average: 15.50, 10.92, 8.38 [08:21:46] PROBLEM - mw132 Current Load on mw132 is CRITICAL: LOAD CRITICAL - total load average: 12.36, 10.05, 7.73 [08:22:21] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 11.17, 9.16, 7.51 [08:24:21] PROBLEM - mw133 Current Load on mw133 is CRITICAL: LOAD CRITICAL - total load average: 15.46, 11.28, 8.47 [08:25:26] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 10.94, 11.04, 8.53 [08:25:46] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 10.16, 11.72, 9.06 [08:26:21] RECOVERY - mw133 Current Load on mw133 is OK: LOAD OK - total load average: 6.67, 9.38, 8.12 [08:26:57] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 6.92, 10.58, 9.31 [08:27:23] RECOVERY - mw131 Current Load on mw131 is OK: LOAD OK - total load average: 6.85, 9.29, 8.17 [08:27:46] RECOVERY - mw132 Current Load on mw132 is OK: LOAD OK - total load average: 5.34, 9.38, 8.53 [08:28:57] RECOVERY - mw134 Current Load on mw134 is OK: LOAD OK - total load average: 4.87, 8.68, 8.77 [09:02:21] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 9.81, 10.49, 9.36 [09:03:42] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 11.82, 11.06, 9.95 [09:03:46] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 10.09, 10.30, 9.28 [09:04:21] RECOVERY - mw133 Current Load on mw133 is OK: LOAD OK - total load average: 8.75, 9.69, 9.18 [09:05:38] PROBLEM - mw134 Current Load on mw134 is CRITICAL: LOAD CRITICAL - total load average: 13.76, 11.79, 10.33 [09:05:46] PROBLEM - mw132 Current Load on mw132 is CRITICAL: LOAD CRITICAL - total load average: 13.09, 11.42, 9.81 [09:11:11] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 10.86, 10.52, 9.80 [09:13:08] PROBLEM - mw133 Current Load on mw133 is CRITICAL: LOAD CRITICAL - total load average: 12.74, 11.57, 10.30 [09:13:15] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 11.58, 10.62, 9.58 [09:13:46] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 9.64, 12.00, 10.94 [09:15:04] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 9.52, 10.90, 10.21 [09:15:13] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [09:15:55] PROBLEM - mw132 Current Load on mw132 is CRITICAL: LOAD CRITICAL - total load average: 12.82, 12.41, 11.22 [09:17:01] PROBLEM - mw133 Current Load on mw133 is CRITICAL: LOAD CRITICAL - total load average: 13.24, 11.79, 10.62 [09:18:48] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.37, 6.58, 6.01 [09:19:07] PROBLEM - mw131 Current Load on mw131 is CRITICAL: LOAD CRITICAL - total load average: 12.07, 11.25, 10.12 [09:19:46] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 11.12, 11.95, 11.33 [09:21:04] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 11.54, 11.55, 10.38 [09:22:48] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.29, 6.22, 6.02 [09:23:02] PROBLEM - mw131 Current Load on mw131 is CRITICAL: LOAD CRITICAL - total load average: 13.58, 12.06, 10.68 [09:23:46] PROBLEM - mw132 Current Load on mw132 is CRITICAL: LOAD CRITICAL - total load average: 18.31, 14.01, 12.22 [09:26:48] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.83, 6.44, 6.14 [09:28:48] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.34, 6.18, 6.09 [09:31:18] PROBLEM - phab121 Current Load on phab121 is WARNING: LOAD WARNING - total load average: 1.90, 1.46, 0.84 [09:33:18] RECOVERY - phab121 Current Load on phab121 is OK: LOAD OK - total load average: 1.01, 1.21, 0.82 [09:33:39] PROBLEM - mw142 Current Load on mw142 is WARNING: LOAD WARNING - total load average: 11.92, 9.57, 8.54 [09:34:10] PROBLEM - mw141 Current Load on mw141 is CRITICAL: LOAD CRITICAL - total load average: 12.77, 10.51, 8.87 [09:35:37] RECOVERY - mw142 Current Load on mw142 is OK: LOAD OK - total load average: 9.49, 9.69, 8.72 [09:40:35] [Grafana] !sre FIRING: The mediawiki job queue has more than 2500 unclaimed jobs https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [09:42:25] PROBLEM - mw143 Current Load on mw143 is WARNING: LOAD WARNING - total load average: 10.38, 10.39, 8.77 [09:42:30] PROBLEM - mw142 Current Load on mw142 is WARNING: LOAD WARNING - total load average: 11.35, 11.15, 9.70 [09:46:10] PROBLEM - mw141 Current Load on mw141 is WARNING: LOAD WARNING - total load average: 8.65, 11.37, 10.78 [09:46:22] RECOVERY - mw143 Current Load on mw143 is OK: LOAD OK - total load average: 7.25, 9.51, 8.85 [09:46:29] RECOVERY - mw142 Current Load on mw142 is OK: LOAD OK - total load average: 8.49, 9.84, 9.52 [09:48:10] PROBLEM - mw141 Current Load on mw141 is CRITICAL: LOAD CRITICAL - total load average: 12.30, 11.89, 11.04 [09:50:10] PROBLEM - mw141 Current Load on mw141 is WARNING: LOAD WARNING - total load average: 11.98, 11.92, 11.16 [09:50:34] PROBLEM - mw142 Current Load on mw142 is CRITICAL: LOAD CRITICAL - total load average: 16.09, 12.47, 10.64 [09:51:27] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.44, 3.38, 2.32 [09:52:10] PROBLEM - mw141 Current Load on mw141 is CRITICAL: LOAD CRITICAL - total load average: 13.57, 12.47, 11.44 [09:52:33] PROBLEM - mw142 Current Load on mw142 is WARNING: LOAD WARNING - total load average: 11.40, 11.28, 10.40 [09:52:36] PROBLEM - cp34 Varnish Backends on cp34 is CRITICAL: 1 backends are down. mw131 [09:52:36] PROBLEM - cp24 Varnish Backends on cp24 is CRITICAL: 1 backends are down. mw131 [09:53:17] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 3.08, 3.13, 2.35 [09:54:16] PROBLEM - mw143 Current Load on mw143 is WARNING: LOAD WARNING - total load average: 9.54, 10.52, 9.65 [09:54:31] PROBLEM - mw142 Current Load on mw142 is CRITICAL: LOAD CRITICAL - total load average: 12.36, 11.95, 10.77 [09:54:33] RECOVERY - cp34 Varnish Backends on cp34 is OK: All 15 backends are healthy [09:54:36] RECOVERY - cp24 Varnish Backends on cp24 is OK: All 15 backends are healthy [09:55:35] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1[Grafana] !sre FIRING: The mediawiki job queue has more than 2500 unclaimed jobs https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [09:56:14] RECOVERY - mw143 Current Load on mw143 is OK: LOAD OK - total load average: 7.88, 9.67, 9.45 [09:56:30] PROBLEM - mw142 Current Load on mw142 is WARNING: LOAD WARNING - total load average: 9.13, 11.05, 10.59 [10:00:29] PROBLEM - mw142 Current Load on mw142 is CRITICAL: LOAD CRITICAL - total load average: 15.50, 13.01, 11.46 [10:05:08] PROBLEM - mw143 Current Load on mw143 is CRITICAL: LOAD CRITICAL - total load average: 22.91, 15.77, 11.94 [10:06:36] PROBLEM - cp35 Varnish Backends on cp35 is CRITICAL: 2 backends are down. mw132 mw142 [10:06:37] PROBLEM - cp24 Varnish Backends on cp24 is CRITICAL: 1 backends are down. mw132 [10:06:50] PROBLEM - cp25 Varnish Backends on cp25 is CRITICAL: 1 backends are down. mw132 [10:08:32] RECOVERY - cp35 Varnish Backends on cp35 is OK: All 15 backends are healthy [10:08:36] RECOVERY - cp24 Varnish Backends on cp24 is OK: All 15 backends are healthy [10:08:46] RECOVERY - cp25 Varnish Backends on cp25 is OK: All 15 backends are healthy [10:15:14] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns.ankh.fr.eu.org.', 'ns1.eu.org.', 'ns1.eriomem.net.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [10:16:59] PROBLEM - mw143 Current Load on mw143 is WARNING: LOAD WARNING - total load average: 7.60, 10.56, 11.23 [10:17:37] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.40, 7.05, 6.45 [10:19:31] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.60, 6.43, 6.28 [10:22:55] PROBLEM - mw143 Current Load on mw143 is CRITICAL: LOAD CRITICAL - total load average: 13.45, 10.96, 11.03 [10:26:53] PROBLEM - mw143 Current Load on mw143 is WARNING: LOAD WARNING - total load average: 8.51, 10.32, 10.81 [10:30:09] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 7.36, 5.48, 3.81 [10:32:09] RECOVERY - db121 Current Load on db121 is OK: OK - load average: 5.76, 5.49, 4.02 [10:39:08] PROBLEM - wows.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wows.wiki' expires in 13 day(s) (Sat 30 Dec 2023 06:14:45 GMT +0000). [10:40:43] PROBLEM - mw143 Current Load on mw143 is CRITICAL: LOAD CRITICAL - total load average: 13.32, 11.40, 11.05 [10:42:42] PROBLEM - mw143 Current Load on mw143 is WARNING: LOAD WARNING - total load average: 10.27, 10.77, 10.85 [10:44:41] PROBLEM - mw143 Current Load on mw143 is CRITICAL: LOAD CRITICAL - total load average: 12.37, 11.22, 11.00 [10:46:40] PROBLEM - mw143 Current Load on mw143 is WARNING: LOAD WARNING - total load average: 7.16, 9.79, 10.51 [10:48:09] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 7.93, 5.25, 3.93 [10:50:09] RECOVERY - db121 Current Load on db121 is OK: OK - load average: 3.01, 4.49, 3.83 [10:50:38] PROBLEM - mw143 Current Load on mw143 is CRITICAL: LOAD CRITICAL - total load average: 12.43, 10.37, 10.51 [10:52:33] RECOVERY - mw131 Current Load on mw131 is OK: LOAD OK - total load average: 0.99, 0.35, 0.12 [10:52:37] PROBLEM - cp35 Varnish Backends on cp35 is CRITICAL: 1 backends are down. mw131 [10:52:44] PROBLEM - cp25 Varnish Backends on cp25 is CRITICAL: 1 backends are down. mw131 [10:52:47] PROBLEM - cp24 Varnish Backends on cp24 is CRITICAL: 1 backends are down. mw131 [10:53:13] PROBLEM - cp34 Varnish Backends on cp34 is CRITICAL: 2 backends are down. mw131 mw141 [10:56:30] RECOVERY - cp35 Varnish Backends on cp35 is OK: All 15 backends are healthy [10:56:33] PROBLEM - mw131 Current Load on mw131 is CRITICAL: LOAD CRITICAL - total load average: 18.41, 8.79, 3.43 [10:56:35] RECOVERY - cp25 Varnish Backends on cp25 is OK: All 15 backends are healthy [10:56:36] RECOVERY - cp24 Varnish Backends on cp24 is OK: All 15 backends are healthy [10:59:05] RECOVERY - cp34 Varnish Backends on cp34 is OK: All 15 backends are healthy [11:04:17] PROBLEM - cp35 Varnish Backends on cp35 is CRITICAL: 1 backends are down. mw132 [11:06:14] RECOVERY - cp35 Varnish Backends on cp35 is OK: All 15 backends are healthy [11:15:48] PROBLEM - cp34 Varnish Backends on cp34 is CRITICAL: 1 backends are down. mw134 [11:15:58] PROBLEM - cp35 Varnish Backends on cp35 is CRITICAL: 1 backends are down. mw133 [11:17:45] RECOVERY - cp34 Varnish Backends on cp34 is OK: All 15 backends are healthy [11:17:56] RECOVERY - cp35 Varnish Backends on cp35 is OK: All 15 backends are healthy [11:18:19] PROBLEM - mw143 Current Load on mw143 is WARNING: LOAD WARNING - total load average: 9.41, 11.11, 11.95 [11:20:18] PROBLEM - mw143 Current Load on mw143 is CRITICAL: LOAD CRITICAL - total load average: 13.50, 12.51, 12.37 [11:26:13] PROBLEM - mw143 Current Load on mw143 is WARNING: LOAD WARNING - total load average: 7.54, 11.32, 12.00 [11:28:12] PROBLEM - mw143 Current Load on mw143 is CRITICAL: LOAD CRITICAL - total load average: 15.25, 13.30, 12.66 [11:34:34] PROBLEM - cp34 Varnish Backends on cp34 is CRITICAL: 1 backends are down. mw132 [11:36:33] RECOVERY - cp34 Varnish Backends on cp34 is OK: All 15 backends are healthy [11:39:08] PROBLEM - wows.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address wows.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [11:42:58] PROBLEM - cp25 Varnish Backends on cp25 is CRITICAL: 1 backends are down. mw134 [11:44:53] RECOVERY - cp25 Varnish Backends on cp25 is OK: All 15 backends are healthy [11:52:09] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 7.59, 5.24, 3.69 [11:54:09] RECOVERY - db121 Current Load on db121 is OK: OK - load average: 3.97, 5.01, 3.82 [12:08:50] PROBLEM - cp35 Varnish Backends on cp35 is CRITICAL: 1 backends are down. mw131 [12:10:57] PROBLEM - cp34 Varnish Backends on cp34 is CRITICAL: 1 backends are down. mw142 [12:12:43] RECOVERY - cp35 Varnish Backends on cp35 is OK: All 15 backends are healthy [12:12:54] RECOVERY - cp34 Varnish Backends on cp34 is OK: All 15 backends are healthy [12:16:38] PROBLEM - cp35 Varnish Backends on cp35 is CRITICAL: 1 backends are down. mw133 [12:16:44] PROBLEM - cp25 Varnish Backends on cp25 is CRITICAL: 1 backends are down. mw133 [12:16:46] PROBLEM - cp24 Varnish Backends on cp24 is CRITICAL: 1 backends are down. mw133 [12:16:51] PROBLEM - cp34 Varnish Backends on cp34 is CRITICAL: 1 backends are down. mw133 [12:18:48] RECOVERY - cp34 Varnish Backends on cp34 is OK: All 15 backends are healthy [12:20:30] RECOVERY - cp35 Varnish Backends on cp35 is OK: All 15 backends are healthy [12:20:35] RECOVERY - cp25 Varnish Backends on cp25 is OK: All 15 backends are healthy [12:20:36] RECOVERY - cp24 Varnish Backends on cp24 is OK: All 15 backends are healthy [12:22:50] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 6.34, 4.13, 2.57 [12:24:48] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.57, 6.94, 6.24 [12:26:23] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.69, 3.03, 2.59 [12:28:17] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 2.18, 2.71, 2.53 [12:30:48] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.24, 6.70, 6.37 [12:50:48] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 1.66, 3.14, 3.73 [12:51:14] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.75, 6.71, 6.31 [12:51:29] PROBLEM - mw143 Current Load on mw143 is WARNING: LOAD WARNING - total load average: 5.18, 9.63, 11.98 [12:52:14] PROBLEM - cp34 Varnish Backends on cp34 is CRITICAL: 8 backends are down. mw131 mw132 mw141 mw142 mw133 mw134 mw143 mediawiki [12:52:20] PROBLEM - cp25 Varnish Backends on cp25 is CRITICAL: 8 backends are down. mw131 mw132 mw141 mw142 mw133 mw134 mw143 mediawiki [12:52:36] PROBLEM - cp24 Varnish Backends on cp24 is CRITICAL: 8 backends are down. mw131 mw132 mw141 mw142 mw133 mw134 mw143 mediawiki [12:53:08] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.11, 6.06, 6.11 [12:54:11] RECOVERY - cp34 Varnish Backends on cp34 is OK: All 15 backends are healthy [12:54:20] RECOVERY - cp25 Varnish Backends on cp25 is OK: All 15 backends are healthy [12:54:36] RECOVERY - cp24 Varnish Backends on cp24 is OK: All 15 backends are healthy [12:54:50] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 1.84, 2.33, 3.25 [12:55:35] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1[Grafana] !sre FIRING: The mediawiki job queue has more than 2500 unclaimed jobs https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [12:57:04] !log [@test131] starting deploy of {'config': True} to all [12:57:06] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 2s [12:57:29] PROBLEM - mw143 Current Load on mw143 is CRITICAL: LOAD CRITICAL - total load average: 14.76, 11.16, 11.65 [12:57:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:58:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:58:49] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.91, 3.74, 3.61 [13:01:36] PROBLEM - cp35 Varnish Backends on cp35 is CRITICAL: 1 backends are down. mw141 [13:02:36] PROBLEM - cp24 Varnish Backends on cp24 is CRITICAL: 1 backends are down. mw134 [13:03:29] PROBLEM - mw143 Current Load on mw143 is WARNING: LOAD WARNING - total load average: 11.85, 11.69, 11.97 [13:04:36] RECOVERY - cp24 Varnish Backends on cp24 is OK: All 15 backends are healthy [13:05:29] PROBLEM - mw143 Current Load on mw143 is CRITICAL: LOAD CRITICAL - total load average: 9.99, 11.79, 12.03 [13:05:30] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 5.59, 4.24, 3.12 [13:06:10] PROBLEM - mw141 Current Load on mw141 is WARNING: LOAD WARNING - total load average: 5.97, 9.18, 11.64 [13:07:26] RECOVERY - cp35 Varnish Backends on cp35 is OK: All 15 backends are healthy [13:07:29] PROBLEM - mw143 Current Load on mw143 is WARNING: LOAD WARNING - total load average: 11.86, 11.54, 11.89 [13:07:30] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 2.46, 3.60, 3.03 [13:08:11] PROBLEM - mw141 Current Load on mw141 is CRITICAL: LOAD CRITICAL - total load average: 16.68, 11.54, 12.16 [13:09:29] PROBLEM - mw143 Current Load on mw143 is CRITICAL: LOAD CRITICAL - total load average: 11.06, 12.22, 12.15 [13:09:30] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 2.62, 3.21, 2.94 [13:11:12] PROBLEM - cp25 Varnish Backends on cp25 is CRITICAL: 1 backends are down. mw143 [13:11:23] PROBLEM - cp24 Puppet on cp24 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [13:16:58] RECOVERY - cp25 Varnish Backends on cp25 is OK: All 15 backends are healthy [13:19:31] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 7.10, 4.69, 3.55 [13:19:46] PROBLEM - cp34 Varnish Backends on cp34 is CRITICAL: 1 backends are down. mw142 [13:20:51] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 2.85, 3.18, 3.91 [13:22:09] PROBLEM - cp24 Varnish Backends on cp24 is CRITICAL: 1 backends are down. mw143 [13:24:03] RECOVERY - cp24 Varnish Backends on cp24 is OK: All 15 backends are healthy [13:27:36] PROBLEM - cp25 Varnish Backends on cp25 is CRITICAL: 2 backends are down. mw131 mw133 [13:27:49] PROBLEM - mw142 MediaWiki Rendering on mw142 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:27:50] PROBLEM - cp24 Varnish Backends on cp24 is CRITICAL: 2 backends are down. mw131 mw133 [13:27:56] PROBLEM - cp35 Varnish Backends on cp35 is CRITICAL: 3 backends are down. mw131 mw142 mw133 [13:30:35] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1[Grafana] !sre FIRING: The mediawiki job queue has more than 2500 unclaimed jobs https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [13:30:51] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 5.47, 3.89, 3.83 [13:33:58] RECOVERY - mw142 MediaWiki Rendering on mw142 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 2.780 second response time [13:36:48] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 1.98, 3.62, 3.87 [13:37:30] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 2.79, 3.47, 3.92 [13:39:08] PROBLEM - wows.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wows.wiki' expires in 13 day(s) (Sat 30 Dec 2023 06:14:45 GMT +0000). [13:39:17] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query uk.eu.org. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [13:39:23] RECOVERY - cp24 Puppet on cp24 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:40:48] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 0.71, 2.08, 3.17 [13:43:30] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 1.72, 2.35, 3.27 [13:47:56] RECOVERY - cp35 Varnish Backends on cp35 is OK: All 15 backends are healthy [13:51:56] PROBLEM - cp35 Varnish Backends on cp35 is CRITICAL: 1 backends are down. mw132 [13:52:58] RECOVERY - cp34 Varnish Backends on cp34 is OK: All 15 backends are healthy [13:56:55] PROBLEM - cp34 Varnish Backends on cp34 is CRITICAL: 2 backends are down. mw132 mw134 [14:01:56] RECOVERY - cp35 Varnish Backends on cp35 is OK: All 15 backends are healthy [14:02:20] RECOVERY - cp25 Varnish Backends on cp25 is OK: All 15 backends are healthy [14:02:36] RECOVERY - cp24 Varnish Backends on cp24 is OK: All 15 backends are healthy [14:06:48] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.36, 7.22, 6.54 [14:07:56] PROBLEM - cp35 Varnish Backends on cp35 is CRITICAL: 2 backends are down. mw141 mw133 [14:08:48] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.22, 7.09, 6.57 [14:09:08] PROBLEM - wows.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address wows.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [14:10:49] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 6.14, 3.31, 2.01 [14:12:20] PROBLEM - cp25 Varnish Backends on cp25 is CRITICAL: 2 backends are down. mw131 mw133 [14:12:36] PROBLEM - cp24 Varnish Backends on cp24 is CRITICAL: 1 backends are down. mw131 [14:13:49] PROBLEM - puppetdb121 Puppet on puppetdb121 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[puppetdb] [14:14:33] RECOVERY - cp34 Varnish Backends on cp34 is OK: All 15 backends are healthy [14:14:36] RECOVERY - cp24 Varnish Backends on cp24 is OK: All 15 backends are healthy [14:14:48] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.49, 3.96, 2.62 [14:16:48] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.59, 6.75, 6.64 [14:16:48] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 0.80, 2.77, 2.34 [14:17:56] RECOVERY - cp35 Varnish Backends on cp35 is OK: All 15 backends are healthy [14:21:24] PROBLEM - cp24 Varnish Backends on cp24 is CRITICAL: 1 backends are down. mw132 [14:21:31] PROBLEM - cp34 Varnish Backends on cp34 is CRITICAL: 3 backends are down. mw132 mw133 mw134 [14:23:18] RECOVERY - cp24 Varnish Backends on cp24 is OK: All 15 backends are healthy [14:23:56] PROBLEM - cp35 Varnish Backends on cp35 is CRITICAL: 2 backends are down. mw132 mw134 [14:24:20] RECOVERY - cp25 Varnish Backends on cp25 is OK: All 15 backends are healthy [14:27:06] PROBLEM - cp24 Varnish Backends on cp24 is CRITICAL: 1 backends are down. mw132 [14:27:22] RECOVERY - cp34 Varnish Backends on cp34 is OK: All 15 backends are healthy [14:27:56] RECOVERY - cp35 Varnish Backends on cp35 is OK: All 15 backends are healthy [14:28:20] PROBLEM - cp25 Varnish Backends on cp25 is CRITICAL: 1 backends are down. mw131 [14:29:00] RECOVERY - cp24 Varnish Backends on cp24 is OK: All 15 backends are healthy [14:31:56] PROBLEM - cp35 Varnish Backends on cp35 is CRITICAL: 2 backends are down. mw141 mw134 [14:32:47] PROBLEM - cp24 Varnish Backends on cp24 is CRITICAL: 3 backends are down. mw131 mw132 mw133 [14:33:16] PROBLEM - cp34 Varnish Backends on cp34 is CRITICAL: 2 backends are down. mw131 mw133 [14:35:56] RECOVERY - cp35 Varnish Backends on cp35 is OK: All 15 backends are healthy [14:36:20] RECOVERY - cp25 Varnish Backends on cp25 is OK: All 15 backends are healthy [14:36:36] RECOVERY - cp24 Varnish Backends on cp24 is OK: All 15 backends are healthy [14:37:03] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.andreijiroh.uk.eu.org All nameservers failed to answer the query. [14:39:08] RECOVERY - cp34 Varnish Backends on cp34 is OK: All 15 backends are healthy [14:40:49] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.67, 2.70, 1.93 [14:42:48] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 1.80, 2.14, 1.81 [14:42:52] PROBLEM - cp35 Varnish Backends on cp35 is CRITICAL: 2 backends are down. mw142 mw133 [14:43:05] PROBLEM - cp34 Varnish Backends on cp34 is CRITICAL: 2 backends are down. mw142 mw133 [14:44:20] PROBLEM - cp25 Varnish Backends on cp25 is CRITICAL: 2 backends are down. mw131 mw134 [14:44:36] PROBLEM - cp24 Varnish Backends on cp24 is CRITICAL: 2 backends are down. mw131 mw134 [14:44:48] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.74, 7.03, 6.57 [14:45:02] RECOVERY - cp34 Varnish Backends on cp34 is OK: All 15 backends are healthy [14:48:19] RECOVERY - cp25 Varnish Backends on cp25 is OK: All 15 backends are healthy [14:48:36] RECOVERY - cp24 Varnish Backends on cp24 is OK: All 15 backends are healthy [14:48:41] RECOVERY - cp35 Varnish Backends on cp35 is OK: All 15 backends are healthy [14:48:48] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.23, 6.78, 6.57 [14:51:57] PROBLEM - cp34 Varnish Backends on cp34 is CRITICAL: 3 backends are down. mw132 mw133 mw134 [14:52:20] PROBLEM - cp25 Varnish Backends on cp25 is CRITICAL: 2 backends are down. mw132 mw133 [14:52:36] PROBLEM - cp35 Varnish Backends on cp35 is CRITICAL: 1 backends are down. mw133 [14:55:51] RECOVERY - cp34 Varnish Backends on cp34 is OK: All 15 backends are healthy [14:56:36] PROBLEM - cp24 Varnish Backends on cp24 is CRITICAL: 3 backends are down. mw132 mw133 mw134 [14:59:49] PROBLEM - cp34 Varnish Backends on cp34 is CRITICAL: 2 backends are down. mw141 mw142 [15:07:46] RECOVERY - puppetdb121 Puppet on puppetdb121 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:09:08] PROBLEM - wows.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wows.wiki' expires in 13 day(s) (Sat 30 Dec 2023 06:14:45 GMT +0000). [15:15:14] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [15:20:35] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1[Grafana] !sre FIRING: The mediawiki job queue has more than 2500 unclaimed jobs https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [15:29:13] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.26, 7.17, 6.22 [15:30:12] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.86, 7.43, 6.82 [15:31:10] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 5.89, 6.49, 6.07 [15:34:01] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 3.84, 5.96, 6.38 [15:36:24] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query uk.eu.org. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [15:39:08] PROBLEM - wows.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address wows.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [15:45:14] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['NS.ANKH.FR.eu.org.', 'NS1.eu.org.', 'NS1.ERIOMEM.NET.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [15:46:20] RECOVERY - cp25 Varnish Backends on cp25 is OK: All 15 backends are healthy [15:48:36] RECOVERY - cp24 Varnish Backends on cp24 is OK: All 15 backends are healthy [15:48:37] RECOVERY - cp34 Varnish Backends on cp34 is OK: All 15 backends are healthy [15:50:20] PROBLEM - cp25 Varnish Backends on cp25 is CRITICAL: 2 backends are down. mw133 mw134 [15:50:35] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1[Grafana] !sre FIRING: The mediawiki job queue has more than 2500 unclaimed jobs https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [15:52:34] PROBLEM - cp34 Varnish Backends on cp34 is CRITICAL: 2 backends are down. mw132 mw134 [15:52:36] PROBLEM - cp24 Varnish Backends on cp24 is CRITICAL: 1 backends are down. mw132 [15:56:33] RECOVERY - cp34 Varnish Backends on cp34 is OK: All 15 backends are healthy [15:56:36] RECOVERY - cp24 Varnish Backends on cp24 is OK: All 15 backends are healthy [15:59:56] RECOVERY - cp35 Varnish Backends on cp35 is OK: All 15 backends are healthy [16:00:33] PROBLEM - cp34 Varnish Backends on cp34 is CRITICAL: 2 backends are down. mw132 mw134 [16:00:36] PROBLEM - cp24 Varnish Backends on cp24 is CRITICAL: 2 backends are down. mw132 mw134 [16:03:56] PROBLEM - cp35 Varnish Backends on cp35 is CRITICAL: 2 backends are down. mw132 mw133 [16:10:20] RECOVERY - cp25 Varnish Backends on cp25 is OK: All 15 backends are healthy [16:12:33] RECOVERY - cp34 Varnish Backends on cp34 is OK: All 15 backends are healthy [16:12:36] RECOVERY - cp24 Varnish Backends on cp24 is OK: All 15 backends are healthy [16:14:20] PROBLEM - cp25 Varnish Backends on cp25 is CRITICAL: 1 backends are down. mw132 [16:16:36] PROBLEM - cp24 Varnish Backends on cp24 is CRITICAL: 2 backends are down. mw132 mw134 [16:16:48] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.59, 6.60, 6.20 [16:19:31] PROBLEM - cp34 Varnish Backends on cp34 is CRITICAL: 1 backends are down. mw132 [16:19:56] RECOVERY - cp35 Varnish Backends on cp35 is OK: All 15 backends are healthy [16:20:20] RECOVERY - cp25 Varnish Backends on cp25 is OK: All 15 backends are healthy [16:20:36] RECOVERY - cp24 Varnish Backends on cp24 is OK: All 15 backends are healthy [16:20:49] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.89, 6.63, 6.32 [16:21:28] RECOVERY - cp34 Varnish Backends on cp34 is OK: All 15 backends are healthy [16:25:25] PROBLEM - cp34 Varnish Backends on cp34 is CRITICAL: 1 backends are down. mw141 [16:25:56] PROBLEM - cp35 Varnish Backends on cp35 is CRITICAL: 2 backends are down. mw132 mw134 [16:26:20] PROBLEM - cp25 Varnish Backends on cp25 is CRITICAL: 2 backends are down. mw132 mw141 [16:28:36] PROBLEM - cp24 Varnish Backends on cp24 is CRITICAL: 2 backends are down. mw132 mw141 [16:33:14] RECOVERY - cp34 Varnish Backends on cp34 is OK: All 15 backends are healthy [16:34:19] RECOVERY - cp25 Varnish Backends on cp25 is OK: All 15 backends are healthy [16:34:36] RECOVERY - cp24 Varnish Backends on cp24 is OK: All 15 backends are healthy [16:35:15] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.81, 6.95, 6.75 [16:36:23] RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp24.miraheze.org - CNAME OK [16:38:02] PROBLEM - matomo121 Current Load on matomo121 is WARNING: LOAD WARNING - total load average: 3.55, 3.16, 2.63 [16:38:19] PROBLEM - cp25 Varnish Backends on cp25 is CRITICAL: 2 backends are down. mw141 mw142 [16:38:36] PROBLEM - cp24 Varnish Backends on cp24 is CRITICAL: 2 backends are down. mw141 mw142 [16:39:42] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/dfce84be15d3...f4c6e83cce09 [16:39:44] [02miraheze/puppet] 07paladox 03f4c6e83 - Revert "mcrouter: set num_proxies to 2" [16:40:02] RECOVERY - matomo121 Current Load on matomo121 is OK: LOAD OK - total load average: 3.27, 3.13, 2.68 [16:40:09] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/f4c6e83cce09...0b3109da3781 [16:40:10] [02miraheze/puppet] 07paladox 030b3109d - mcrouter: increase num_proxies to 2 [16:42:15] !log [paladox@mwtask141] /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/w/maintenance/updateSpecialPages.php [16:42:36] RECOVERY - cp24 Varnish Backends on cp24 is OK: All 15 backends are healthy [16:43:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:44:05] PROBLEM - mw132 MediaWiki Rendering on mw132 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:44:48] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.39, 6.55, 6.72 [16:45:34] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.25, 2.98, 2.14 [16:46:08] RECOVERY - mw132 MediaWiki Rendering on mw132 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 8.423 second response time [16:47:29] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 2.72, 2.82, 2.18 [16:47:56] RECOVERY - cp35 Varnish Backends on cp35 is OK: All 15 backends are healthy [16:48:19] RECOVERY - cp25 Varnish Backends on cp25 is OK: All 15 backends are healthy [16:52:01] PROBLEM - cp34 Varnish Backends on cp34 is CRITICAL: 2 backends are down. mw132 mw142 [16:54:09] PROBLEM - cp24 Varnish Backends on cp24 is CRITICAL: 5 backends are down. mw131 mw132 mw141 mw142 mw133 [16:54:19] PROBLEM - cp25 Varnish Backends on cp25 is CRITICAL: 3 backends are down. mw131 mw132 mw133 [16:54:51] PROBLEM - cp35 Varnish Backends on cp35 is CRITICAL: 3 backends are down. mw142 mw134 mw143 [16:56:48] RECOVERY - cp35 Varnish Backends on cp35 is OK: All 15 backends are healthy [16:57:53] RECOVERY - cp34 Varnish Backends on cp34 is OK: All 15 backends are healthy [16:59:50] RECOVERY - cp24 Varnish Backends on cp24 is OK: All 15 backends are healthy [17:00:43] PROBLEM - cp35 Varnish Backends on cp35 is CRITICAL: 2 backends are down. mw131 mw132 [17:02:19] RECOVERY - cp25 Varnish Backends on cp25 is OK: All 15 backends are healthy [17:03:08] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.54, 7.13, 6.81 [17:06:57] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.65, 6.61, 6.71 [17:07:45] PROBLEM - cp34 Varnish Backends on cp34 is CRITICAL: 1 backends are down. mw131 [17:08:24] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.andreijiroh.uk.eu.org. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [17:12:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.16, 6.99, 6.82 [17:13:08] PROBLEM - cp24 Varnish Backends on cp24 is CRITICAL: 1 backends are down. mw141 [17:13:12] PROBLEM - cp25 Varnish Backends on cp25 is CRITICAL: 1 backends are down. mw141 [17:14:49] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.08, 6.37, 6.62 [17:16:56] RECOVERY - cp24 Varnish Backends on cp24 is OK: All 15 backends are healthy [17:17:03] RECOVERY - cp25 Varnish Backends on cp25 is OK: All 15 backends are healthy [17:28:48] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.79, 6.96, 6.84 [17:29:36] PROBLEM - cp25 Varnish Backends on cp25 is CRITICAL: 1 backends are down. mw134 [17:30:15] PROBLEM - cp24 Varnish Backends on cp24 is CRITICAL: 1 backends are down. mw131 [17:31:10] RECOVERY - cp34 Varnish Backends on cp34 is OK: All 15 backends are healthy [17:31:29] PROBLEM - mw132 MediaWiki Rendering on mw132 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:32:09] RECOVERY - cp24 Varnish Backends on cp24 is OK: All 15 backends are healthy [17:33:27] RECOVERY - cp25 Varnish Backends on cp25 is OK: All 15 backends are healthy [17:33:56] RECOVERY - cp35 Varnish Backends on cp35 is OK: All 15 backends are healthy [17:34:48] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 4.28, 6.03, 6.54 [17:35:30] RECOVERY - mw132 MediaWiki Rendering on mw132 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 1.228 second response time [17:44:52] PROBLEM - cp35 Varnish Backends on cp35 is CRITICAL: 1 backends are down. mw131 [17:44:57] PROBLEM - cp34 Varnish Backends on cp34 is CRITICAL: 1 backends are down. mw131 [17:46:48] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.39, 6.98, 6.57 [17:50:48] RECOVERY - cp34 Varnish Backends on cp34 is OK: All 15 backends are healthy [17:50:48] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.76, 7.66, 6.96 [17:54:33] RECOVERY - cp35 Varnish Backends on cp35 is OK: All 15 backends are healthy [17:54:45] PROBLEM - cp34 Varnish Backends on cp34 is CRITICAL: 1 backends are down. mw132 [17:56:43] RECOVERY - cp34 Varnish Backends on cp34 is OK: All 15 backends are healthy [18:02:48] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.39, 7.23, 6.96 [18:04:49] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.05, 6.39, 6.69 [18:05:35] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1[Grafana] !sre FIRING: The mediawiki job queue has more than 2500 unclaimed jobs https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [18:07:29] PROBLEM - mw143 Current Load on mw143 is WARNING: LOAD WARNING - total load average: 3.83, 6.66, 11.37 [18:08:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.30, 6.91, 6.87 [18:10:10] PROBLEM - mw141 Current Load on mw141 is WARNING: LOAD WARNING - total load average: 4.18, 6.22, 11.89 [18:10:29] PROBLEM - mw142 Current Load on mw142 is WARNING: LOAD WARNING - total load average: 4.81, 6.54, 11.64 [18:10:48] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.88, 6.64, 6.78 [18:11:29] RECOVERY - mw143 Current Load on mw143 is OK: LOAD OK - total load average: 4.59, 5.57, 9.86 [18:12:21] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 5.56, 6.79, 11.56 [18:12:34] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 4.67, 6.41, 11.31 [18:12:57] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 7.45, 7.69, 11.67 [18:13:46] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 6.49, 7.33, 11.53 [18:14:11] PROBLEM - mw141 Current Load on mw141 is CRITICAL: LOAD CRITICAL - total load average: 13.30, 7.80, 11.08 [18:14:29] RECOVERY - mw142 Current Load on mw142 is OK: LOAD OK - total load average: 5.29, 5.58, 10.05 [18:16:10] PROBLEM - mw141 Current Load on mw141 is WARNING: LOAD WARNING - total load average: 5.88, 6.94, 10.38 [18:16:33] RECOVERY - mw131 Current Load on mw131 is OK: LOAD OK - total load average: 4.46, 5.72, 9.95 [18:18:10] RECOVERY - mw141 Current Load on mw141 is OK: LOAD OK - total load average: 4.27, 6.07, 9.64 [18:18:21] RECOVERY - mw133 Current Load on mw133 is OK: LOAD OK - total load average: 5.88, 6.54, 9.92 [18:18:48] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.47, 7.06, 6.94 [18:18:57] RECOVERY - mw134 Current Load on mw134 is OK: LOAD OK - total load average: 5.07, 6.69, 10.01 [18:21:46] RECOVERY - mw132 Current Load on mw132 is OK: LOAD OK - total load average: 7.11, 7.35, 9.96 [18:26:48] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.52, 6.33, 6.72 [18:36:23] RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp25.miraheze.org - CNAME OK [19:09:30] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.91, 7.03, 6.36 [19:11:25] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.51, 6.72, 6.32 [19:19:04] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 10.64, 8.06, 6.95 [19:22:52] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.36, 7.00, 6.80 [19:24:48] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.16, 6.47, 6.63 [19:28:48] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.96, 6.63, 6.64 [19:30:48] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.05, 6.68, 6.68 [19:50:59] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.32, 7.35, 6.94 [19:52:53] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.35, 7.48, 7.02 [19:54:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.55, 6.79, 6.83 [20:00:48] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.95, 6.52, 6.73 [20:05:43] PROBLEM - wiki.kingmingshrimp.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.kingmingshrimp.com' expires in 15 day(s) (Mon 01 Jan 2024 19:57:20 GMT +0000). [20:06:04] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/534a48e8aa2f...9354432268a3 [20:06:05] [02miraheze/ssl] 07MirahezeSSLBot 039354432 - Bot: Update SSL cert for wiki.kingmingshrimp.com [20:30:02] PROBLEM - wiki.oasismp.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.oasismp.com' expires in 15 day(s) (Mon 01 Jan 2024 20:19:00 GMT +0000). [20:30:18] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/9354432268a3...5e5fb620e11c [20:30:19] [02miraheze/ssl] 07MirahezeSSLBot 035e5fb62 - Bot: Update SSL cert for wiki.oasismp.com [20:31:32] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.65, 6.85, 6.69 [20:34:55] RECOVERY - wiki.kingmingshrimp.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.kingmingshrimp.com' will expire on Fri 15 Mar 2024 19:05:57 GMT +0000. [20:39:08] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.26, 6.64, 6.64 [20:45:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.24, 6.90, 6.79 [20:47:43] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.39, 6.80, 6.77 [20:51:32] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.86, 6.78, 6.81 [20:53:26] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.78, 6.40, 6.66 [21:08:24] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.andreijiroh.uk.eu.org. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [21:20:06] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.26, 6.85, 6.50 [21:22:01] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.48, 7.42, 6.75 [21:22:11] PROBLEM - mw141 Current Load on mw141 is WARNING: LOAD WARNING - total load average: 10.55, 8.19, 6.49 [21:23:55] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.85, 6.76, 6.58 [21:24:10] RECOVERY - mw141 Current Load on mw141 is OK: LOAD OK - total load average: 7.44, 8.37, 6.81 [21:27:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.66, 6.89, 6.70 [21:29:06] RECOVERY - wiki.oasismp.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.oasismp.com' will expire on Fri 15 Mar 2024 19:30:11 GMT +0000. [21:31:38] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.15, 7.64, 7.02 [21:33:33] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.30, 7.34, 6.99 [21:37:17] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.andreijiroh.uk.eu.org All nameservers failed to answer the query. [21:48:48] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.64, 7.03, 6.92 [21:50:48] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.82, 6.96, 6.92 [21:54:48] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.67, 6.50, 6.77 [22:02:57] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.98, 6.91, 6.24 [22:04:57] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 4.98, 6.26, 6.09 [22:05:37] [Grafana] !sre FIRING: The mediawiki job queue has more than 2500 unclaimed jobs https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [22:06:23] RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp25.miraheze.org - CNAME OK [22:08:48] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.89, 2.84, 1.66 [22:10:48] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 3.01, 2.99, 1.87 [22:12:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.18, 7.72, 7.21 [22:13:50] PROBLEM - puppetdb121 Puppet on puppetdb121 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[puppetdb] [22:15:15] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [22:18:57] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.50, 6.78, 6.37 [22:20:48] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.20, 7.34, 7.14 [22:20:57] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 5.15, 6.10, 6.17 [22:24:48] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.25, 7.11, 7.11 [22:30:48] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.79, 6.22, 6.70 [22:44:48] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.51, 7.08, 6.69 [22:45:14] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.gab.pt.eu.org. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [22:46:48] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.54, 6.54, 6.55 [22:50:11] [02RottenLinks] 07paladox synchronize pull request 03#73: Redesign RottenLinks to not depend on a maintenance script - 13https://github.com/miraheze/RottenLinks/pull/73 [22:50:13] [02miraheze/RottenLinks] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/RottenLinks/compare/76d44097ebf9...cc06f4bbe70f [22:50:14] [02miraheze/RottenLinks] 07paladox 03cc06f4b - Update updateExternalLinks.php [22:51:20] [02miraheze/RottenLinks] 07github-actions[bot] pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/RottenLinks/compare/cc06f4bbe70f...a3c57a51a18d [22:51:22] [02miraheze/RottenLinks] 07github-actions 03a3c57a5 - CI: lint code to MediaWiki standards [22:51:24] [02RottenLinks] 07github-actions[bot] synchronize pull request 03#73: Redesign RottenLinks to not depend on a maintenance script - 13https://github.com/miraheze/RottenLinks/pull/73 [22:54:26] miraheze/RottenLinks - paladox the build has errored. [23:06:49] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/47048e5898c9...14db92c40189 [23:06:50] [02miraheze/mw-config] 07paladox 0314db92c - Set wgExternalLinksSchemaMigrationStage to "SCHEMA_COMPAT_WRITE_BOTH | SCHEMA_COMPAT_READ_OLD" [23:07:09] !log [paladox@mwtask141] starting deploy of {'pull': 'config', 'config': True} to all [23:07:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:07:35] !log [paladox@mwtask141] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 26s [23:07:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:07:42] !log [paladox@mwtask141] starting deploy of {'config': True} to all [23:07:46] RECOVERY - puppetdb121 Puppet on puppetdb121 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:07:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:07:50] miraheze/mw-config - paladox the build passed. [23:07:54] !log [paladox@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 11s [23:08:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:11:40] [02RottenLinks] 07paladox synchronize pull request 03#73: Redesign RottenLinks to not depend on a maintenance script - 13https://github.com/miraheze/RottenLinks/pull/73 [23:11:41] [02miraheze/RottenLinks] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/RottenLinks/compare/a3c57a51a18d...19f3cf8df11b [23:11:44] [02miraheze/RottenLinks] 07paladox 0319f3cf8 - Update updateExternalLinks.php [23:12:54] [02RottenLinks] 07paladox synchronize pull request 03#73: Redesign RottenLinks to not depend on a maintenance script - 13https://github.com/miraheze/RottenLinks/pull/73 [23:12:57] [02miraheze/RottenLinks] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/RottenLinks/compare/19f3cf8df11b...ae927a0b2070 [23:12:59] [02miraheze/RottenLinks] 07paladox 03ae927a0 - Update CHANGELOG.md [23:14:14] [02miraheze/RottenLinks] 07github-actions[bot] pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/RottenLinks/compare/ae927a0b2070...43c8641be5bb [23:14:15] [02miraheze/RottenLinks] 07github-actions 0343c8641 - CI: lint code to MediaWiki standards [23:14:18] [02RottenLinks] 07github-actions[bot] synchronize pull request 03#73: Redesign RottenLinks to not depend on a maintenance script - 13https://github.com/miraheze/RottenLinks/pull/73 [23:14:47] [02RottenLinks] 07paladox synchronize pull request 03#73: Redesign RottenLinks to not depend on a maintenance script - 13https://github.com/miraheze/RottenLinks/pull/73 [23:14:48] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.46, 6.72, 6.36 [23:14:50] [02miraheze/RottenLinks] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/RottenLinks/compare/43c8641be5bb...7e5e4a3f44e3 [23:14:52] [02miraheze/RottenLinks] 07paladox 037e5e4a3 - Update updateExternalLinks.php [23:15:34] miraheze/RottenLinks - paladox the build has errored. [23:16:26] [02RottenLinks] 07paladox synchronize pull request 03#73: Redesign RottenLinks to not depend on a maintenance script - 13https://github.com/miraheze/RottenLinks/pull/73 [23:16:29] [02miraheze/RottenLinks] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/RottenLinks/compare/7e5e4a3f44e3...6af07e0fc8cf [23:16:32] [02miraheze/RottenLinks] 07paladox 036af07e0 - Update updateExternalLinks.php [23:16:49] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.08, 6.48, 6.32 [23:17:12] miraheze/RottenLinks - paladox the build has errored. [23:19:01] miraheze/RottenLinks - paladox the build has errored. [23:20:33] [02RottenLinks] 07paladox synchronize pull request 03#73: Redesign RottenLinks to not depend on a maintenance script - 13https://github.com/miraheze/RottenLinks/pull/73 [23:20:36] [02miraheze/RottenLinks] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/RottenLinks/compare/6af07e0fc8cf...1df20af865e6 [23:20:38] [02miraheze/RottenLinks] 07paladox 031df20af - Update updateExternalLinks.php [23:22:53] miraheze/RottenLinks - paladox the build has errored. [23:23:49] [02RottenLinks] 07paladox synchronize pull request 03#73: Redesign RottenLinks to not depend on a maintenance script - 13https://github.com/miraheze/RottenLinks/pull/73 [23:23:52] [02miraheze/RottenLinks] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/RottenLinks/compare/1df20af865e6...7bc6590458c6 [23:23:53] [02miraheze/RottenLinks] 07paladox 037bc6590 - Update updateExternalLinks.php [23:26:47] miraheze/RottenLinks - paladox the build has errored. [23:26:52] !log [@test131] starting deploy of {'config': True} to all [23:26:53] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 1s [23:26:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:27:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:29:43] [02RottenLinks] 07paladox synchronize pull request 03#73: Redesign RottenLinks to not depend on a maintenance script - 13https://github.com/miraheze/RottenLinks/pull/73 [23:29:45] [02miraheze/RottenLinks] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/RottenLinks/compare/7bc6590458c6...e7b0000176ad [23:29:48] [02miraheze/RottenLinks] 07paladox 03e7b0000 - Update updateExternalLinks.php [23:32:10] miraheze/RottenLinks - paladox the build has errored. [23:33:26] [02RottenLinks] 07paladox synchronize pull request 03#73: Redesign RottenLinks to not depend on a maintenance script - 13https://github.com/miraheze/RottenLinks/pull/73 [23:33:29] [02miraheze/RottenLinks] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/RottenLinks/compare/e7b0000176ad...e8bbd986a167 [23:33:30] [02miraheze/RottenLinks] 07paladox 03e8bbd98 - Update updateExternalLinks.php [23:33:43] [02miraheze/RottenLinks] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/RottenLinks/compare/e8bbd986a167...4726f498e655 [23:33:45] [02miraheze/RottenLinks] 07paladox 034726f49 - Update config.php [23:33:46] [02RottenLinks] 07paladox synchronize pull request 03#73: Redesign RottenLinks to not depend on a maintenance script - 13https://github.com/miraheze/RottenLinks/pull/73 [23:38:55] miraheze/RottenLinks - paladox the build passed. [23:39:08] PROBLEM - wows.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wows.wiki' expires in 13 day(s) (Sat 30 Dec 2023 06:14:45 GMT +0000). [23:47:59] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.43, 6.81, 6.46 [23:49:54] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.45, 6.79, 6.50