[00:00:51] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.49, 7.73, 7.72 [00:02:51] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.71, 7.37, 7.60 [00:04:30] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 8.22, 4.34, 2.17 [00:04:40] PROBLEM - db112 Current Load on db112 is CRITICAL: CRITICAL - load average: 14.91, 9.05, 4.24 [00:05:22] PROBLEM - mw142 Current Load on mw142 is CRITICAL: LOAD CRITICAL - total load average: 23.10, 12.79, 7.59 [00:05:54] PROBLEM - cp35 Varnish Backends on cp35 is CRITICAL: 3 backends are down. mw141 mw142 mw143 [00:05:59] PROBLEM - mw143 Current Load on mw143 is CRITICAL: LOAD CRITICAL - total load average: 24.56, 14.13, 7.76 [00:06:09] PROBLEM - mw141 Current Load on mw141 is CRITICAL: LOAD CRITICAL - total load average: 13.97, 11.47, 7.19 [00:06:12] PROBLEM - mw143 MediaWiki Rendering on mw143 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:06:27] PROBLEM - os141 PowerDNS Recursor on os141 is CRITICAL: CRITICAL - Plugin timed out while executing system call [00:06:33] PROBLEM - cp24 Varnish Backends on cp24 is CRITICAL: 2 backends are down. mw142 mw143 [00:06:56] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 5.92, 6.29, 6.73 [00:06:56] PROBLEM - cp34 Varnish Backends on cp34 is CRITICAL: 2 backends are down. mw141 mw143 [00:07:13] PROBLEM - mw142 MediaWiki Rendering on mw142 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:07:37] PROBLEM - cp25 Varnish Backends on cp25 is CRITICAL: 2 backends are down. mw141 mw143 [00:08:09] RECOVERY - mw143 MediaWiki Rendering on mw143 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.353 second response time [00:08:24] RECOVERY - os141 PowerDNS Recursor on os141 is OK: DNS OK: 0.426 seconds response time. miraheze.org returns 2001:41d0:801:2000::3a18,2001:41d0:801:2000::5d68,51.195.201.140,51.89.139.24 [00:08:33] RECOVERY - cp24 Varnish Backends on cp24 is OK: All 15 backends are healthy [00:08:56] RECOVERY - cp34 Varnish Backends on cp34 is OK: All 15 backends are healthy [00:11:18] PROBLEM - mw142 Current Load on mw142 is WARNING: LOAD WARNING - total load average: 6.10, 10.81, 8.79 [00:11:59] RECOVERY - mw143 Current Load on mw143 is OK: LOAD OK - total load average: 2.72, 9.88, 8.48 [00:12:02] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 6.68, 3.73, 2.54 [00:12:08] RECOVERY - mw141 Current Load on mw141 is OK: LOAD OK - total load average: 3.45, 9.53, 8.18 [00:12:33] PROBLEM - cp24 Varnish Backends on cp24 is CRITICAL: 6 backends are down. mw131 mw132 mw141 mw142 mw134 mw143 [00:12:43] PROBLEM - os131 Current Load on os131 is CRITICAL: LOAD CRITICAL - total load average: 9.25, 5.80, 3.15 [00:12:51] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.32, 5.76, 6.71 [00:12:52] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 10.80, 9.23, 6.93 [00:12:56] PROBLEM - cp34 Varnish Backends on cp34 is CRITICAL: 8 backends are down. mw131 mw132 mw141 mw142 mw133 mw134 mw143 mediawiki [00:13:17] RECOVERY - mw142 Current Load on mw142 is OK: LOAD OK - total load average: 8.86, 9.58, 8.53 [00:13:29] RECOVERY - mw142 MediaWiki Rendering on mw142 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.365 second response time [00:13:54] RECOVERY - cp35 Varnish Backends on cp35 is OK: All 15 backends are healthy [00:14:02] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.24, 3.53, 2.63 [00:14:33] RECOVERY - cp24 Varnish Backends on cp24 is OK: All 15 backends are healthy [00:14:40] PROBLEM - db112 Current Load on db112 is WARNING: WARNING - load average: 2.33, 7.39, 6.62 [00:14:52] RECOVERY - mw132 Current Load on mw132 is OK: LOAD OK - total load average: 7.94, 8.25, 6.82 [00:14:56] RECOVERY - cp34 Varnish Backends on cp34 is OK: All 15 backends are healthy [00:15:37] RECOVERY - cp25 Varnish Backends on cp25 is OK: All 15 backends are healthy [00:16:02] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 2.22, 3.15, 2.60 [00:16:05] PROBLEM - mail121 Current Load on mail121 is CRITICAL: LOAD CRITICAL - total load average: 2.01, 1.40, 0.78 [00:16:40] RECOVERY - db112 Current Load on db112 is OK: OK - load average: 2.21, 5.61, 6.05 [00:16:42] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 15.70, 12.49, 8.40 [00:16:51] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 22.56, 19.20, 12.11 [00:18:05] RECOVERY - mail121 Current Load on mail121 is OK: LOAD OK - total load average: 0.40, 1.04, 0.73 [00:18:29] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 2.30, 3.53, 3.45 [00:20:29] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 2.70, 3.21, 3.33 [00:20:29] PROBLEM - os131 Current Load on os131 is WARNING: LOAD WARNING - total load average: 2.95, 3.88, 3.29 [00:22:25] RECOVERY - os131 Current Load on os131 is OK: LOAD OK - total load average: 2.31, 3.33, 3.15 [00:24:38] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 5.36, 7.43, 7.53 [00:26:30] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.28, 3.84, 3.58 [00:27:47] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.46, 7.09, 6.72 [00:28:18] PROBLEM - os131 Current Load on os131 is CRITICAL: LOAD CRITICAL - total load average: 4.77, 3.83, 3.38 [00:29:43] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 6.42, 6.61, 6.57 [00:30:31] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 2.21, 3.05, 3.32 [00:32:33] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 4.62, 5.73, 6.73 [00:32:56] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 4.78, 3.76, 2.99 [00:33:36] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.76, 7.48, 6.96 [00:34:09] PROBLEM - os131 Current Load on os131 is WARNING: LOAD WARNING - total load average: 3.30, 3.93, 3.61 [00:34:53] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 2.32, 3.17, 2.86 [00:35:32] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.01, 7.36, 6.96 [00:37:28] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.09, 6.85, 6.82 [00:40:02] PROBLEM - mw131 APT on mw131 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [00:40:08] RECOVERY - os131 Current Load on os131 is OK: LOAD OK - total load average: 2.43, 2.98, 3.29 [00:41:20] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 6.33, 6.60, 6.72 [00:41:30] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.67, 3.76, 3.54 [00:42:03] PROBLEM - mw131 APT on mw131 is CRITICAL: APT CRITICAL: 16 packages available for upgrade (15 critical updates). [00:43:29] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 2.77, 3.30, 3.39 [00:47:35] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.72, 3.74, 3.52 [00:48:51] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.95, 7.35, 7.97 [00:49:35] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 2.92, 3.58, 3.51 [00:50:51] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.83, 7.87, 8.06 [00:51:36] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 5.48, 4.42, 3.83 [00:52:46] PROBLEM - os141 PowerDNS Recursor on os141 is CRITICAL: CRITICAL - Plugin timed out while executing system call [00:54:44] RECOVERY - os141 PowerDNS Recursor on os141 is OK: DNS OK: 1.417 second response time. miraheze.org returns 2001:41d0:801:2000::3a18,2001:41d0:801:2000::5d68,51.195.201.140,51.89.139.24 [00:54:51] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.87, 7.56, 7.91 [00:55:25] [02mw-config] 07brandon-wm opened pull request 03#5421: T11535: Enable wgPageLanguageUseDB on sona.pona.la - 13https://github.com/miraheze/mw-config/pull/5421 [00:56:23] miraheze/mw-config - brandon-wm the build passed. [00:56:51] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 10.42, 8.21, 8.08 [00:57:44] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.49, 4.00, 3.89 [00:58:56] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.80, 7.22, 6.91 [00:59:49] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.09, 4.03, 3.91 [01:00:56] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.35, 7.53, 7.05 [01:02:51] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.95, 7.76, 8.00 [01:03:49] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.37, 3.84, 3.88 [01:05:49] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.77, 4.06, 3.95 [01:06:56] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.07, 7.56, 7.30 [01:13:51] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 2.93, 3.85, 3.97 [01:14:51] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.80, 7.66, 7.60 [01:16:57] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 11.11, 7.86, 7.32 [01:17:52] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.06, 3.48, 3.76 [01:18:56] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.62, 7.29, 7.18 [01:19:52] [02mw-config] 07Universal-Omega closed pull request 03#5421: T11535: Enable wgPageLanguageUseDB on sona.pona.la - 13https://github.com/miraheze/mw-config/pull/5421 [01:19:53] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 2.67, 3.25, 3.65 [01:19:54] [02mw-config] 07Universal-Omega commented on pull request 03#5421: T11535: Enable wgPageLanguageUseDB on sona.pona.la - 13https://github.com/miraheze/mw-config/pull/5421#issuecomment-1877981617 [01:21:04] [02mw-config] 07Universal-Omega commented on pull request 03#5380: Add $wgRSSUserAgent (T11425) - 13https://github.com/miraheze/mw-config/pull/5380#issuecomment-1877982277 [01:21:17] [02mw-config] 07Universal-Omega closed pull request 03#5380: Add $wgRSSUserAgent (T11425) - 13https://github.com/miraheze/mw-config/pull/5380 [01:21:19] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/9095095a1f76...50223da45c83 [01:21:20] [02miraheze/mw-config] 07OAuthority 0350223da - Add $wgRSSUserAgent (T11425) (#5380) [01:21:52] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 7.39, 4.73, 4.14 [01:22:16] miraheze/mw-config - Universal-Omega the build passed. [01:25:50] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.06, 3.91, 3.95 [01:26:27] !log [@test131] starting deploy of {'config': True} to all [01:26:28] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [01:26:51] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.19, 7.80, 7.90 [01:26:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:26:58] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 6.62, 6.23, 6.70 [01:27:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:30:56] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.70, 7.10, 6.98 [01:32:51] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.65, 8.11, 7.96 [01:32:56] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.08, 7.70, 7.23 [01:33:52] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 1.69, 2.66, 3.30 [01:34:51] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.80, 7.82, 7.86 [01:34:56] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 5.73, 7.00, 7.03 [01:38:52] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.32, 8.16, 7.97 [01:38:56] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.19, 7.41, 7.17 [01:38:58] !log [@mwtask141] starting deploy of {'config': True} to all [01:39:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:39:11] !log [@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 13s [01:39:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:40:51] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.10, 7.58, 7.77 [01:42:51] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.23, 7.71, 7.80 [01:42:56] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.47, 7.15, 7.13 [01:44:51] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.65, 7.70, 7.79 [01:46:05] PROBLEM - mail121 Current Load on mail121 is WARNING: LOAD WARNING - total load average: 1.86, 1.77, 1.11 [01:46:51] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.42, 8.07, 7.92 [01:48:05] RECOVERY - mail121 Current Load on mail121 is OK: LOAD OK - total load average: 1.41, 1.64, 1.14 [01:54:51] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.24, 7.69, 7.91 [01:58:57] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.47, 7.14, 7.06 [02:00:52] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.78, 8.08, 7.95 [02:00:56] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.68, 6.99, 7.01 [02:06:51] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.17, 7.76, 7.90 [02:08:51] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.83, 8.26, 8.05 [02:12:56] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 12.04, 8.33, 7.52 [02:13:32] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.37, 6.58, 6.19 [02:14:56] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.72, 7.98, 7.49 [02:15:31] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 5.41, 5.94, 5.99 [02:16:56] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.25, 8.11, 7.60 [02:18:56] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.59, 7.81, 7.54 [02:23:10] PROBLEM - puppetdb121 Puppet on puppetdb121 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[puppetdb] [02:24:57] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.75, 8.13, 7.72 [02:26:56] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.58, 7.58, 7.57 [02:28:56] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 9.07, 7.85, 7.65 [02:30:56] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 5.84, 6.86, 7.29 [02:32:56] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.33, 7.56, 7.50 [02:34:57] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.27, 7.34, 7.43 [02:36:51] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.20, 6.93, 7.85 [02:46:56] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 5.16, 6.16, 6.78 [02:50:51] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 10.15, 7.91, 7.58 [02:54:51] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.35, 7.24, 7.46 [03:00:10] RECOVERY - db121 Backups SQL on db121 is OK: FILE_AGE OK: /var/log/sql-backup.log is 9 seconds old and 0 bytes [03:01:50] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.19, 6.92, 6.54 [03:02:41] RECOVERY - ns1 NTP time on ns1 is OK: NTP OK: Offset 0.003386855125 secs [03:03:46] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 6.79, 6.69, 6.49 [03:09:09] RECOVERY - puppetdb121 Puppet on puppetdb121 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [03:18:51] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.07, 7.38, 7.18 [03:20:19] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.92, 6.82, 6.03 [03:22:19] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 5.11, 6.12, 5.87 [03:24:52] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.87, 7.26, 7.31 [03:48:51] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 12.02, 8.37, 7.56 [03:54:51] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.18, 7.94, 7.72 [04:00:51] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.18, 8.05, 7.83 [04:02:51] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.22, 7.24, 7.55 [04:10:51] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.99, 7.17, 7.30 [04:12:51] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.76, 6.58, 7.07 [04:14:51] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.20, 5.92, 6.76 [04:18:51] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.45, 6.96, 7.02 [04:20:51] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.67, 7.98, 7.38 [04:21:37] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 10.12, 7.32, 6.30 [04:23:36] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 6.33, 6.79, 6.23 [04:24:15] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.57, 7.36, 6.69 [04:28:51] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.84, 7.54, 7.70 [04:30:37] [Grafana] !sre FIRING: The mediawiki job queue has more than 2500 unclaimed jobs https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [04:35:50] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 5.66, 6.35, 6.54 [04:40:52] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.51, 7.19, 7.30 [04:42:51] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.68, 7.31, 7.34 [04:46:34] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [04:55:37] [Grafana] !sre RESOLVED: High Job Queue Backlog https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [04:56:10] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.52, 6.86, 6.50 [04:58:05] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 6.52, 6.66, 6.46 [05:10:51] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 10.47, 7.90, 7.45 [05:12:51] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.26, 7.75, 7.46 [05:14:51] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.38, 7.81, 7.50 [05:16:33] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.gab.pt.eu.org. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [05:16:51] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.77, 7.16, 7.31 [05:18:51] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.24, 7.35, 7.34 [05:19:26] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.50, 6.97, 6.63 [05:23:18] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 6.36, 6.55, 6.54 [05:24:51] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.59, 7.66, 7.59 [05:28:51] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.21, 7.84, 7.64 [05:30:06] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.47, 6.87, 6.67 [05:30:51] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.51, 7.26, 7.44 [05:32:02] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 6.46, 6.57, 6.57 [05:38:56] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.72, 6.81, 6.67 [05:40:51] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.13, 7.89, 7.52 [05:42:51] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.98, 7.81, 7.54 [05:44:56] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 6.16, 6.46, 6.61 [05:50:51] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 10.12, 7.94, 7.49 [05:52:51] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.76, 7.75, 7.48 [06:00:52] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.01, 7.57, 7.40 [06:02:51] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.23, 7.06, 7.24 [06:12:51] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 4.31, 5.86, 6.70 [06:18:51] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.82, 6.98, 6.89 [06:24:52] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.26, 7.05, 6.93 [06:28:48] PROBLEM - barengreza.my.id - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - barengreza.my.id All nameservers failed to answer the query. [06:28:51] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.34, 6.65, 6.84 [06:30:51] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.95, 6.43, 6.74 [06:33:08] RECOVERY - mwtask141 Disk Space on mwtask141 is OK: DISK OK - free space: / 26723MiB (37% inode=89%); [06:37:28] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: LOAD WARNING - total load average: 10.28, 9.77, 9.42 [06:39:27] RECOVERY - mwtask141 Current Load on mwtask141 is OK: LOAD OK - total load average: 9.22, 9.55, 9.38 [06:42:41] PROBLEM - ns1 NTP time on ns1 is WARNING: NTP WARNING: Offset 0.1656634808 secs [06:44:51] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.17, 6.81, 6.68 [06:48:51] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.11, 7.42, 6.91 [06:52:21] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: LOAD WARNING - total load average: 10.88, 10.16, 9.80 [06:52:51] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.23, 7.29, 6.98 [06:53:23] RECOVERY - cloud11 IPMI Sensors on cloud11 is OK: IPMI Status: OK [06:54:51] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 4.53, 6.36, 6.69 [06:56:20] RECOVERY - mwtask141 Current Load on mwtask141 is OK: LOAD OK - total load average: 9.57, 9.86, 9.77 [06:57:16] PROBLEM - cloud11 IPMI Sensors on cloud11 is CRITICAL: IPMI Status: Critical [Cntlr 2 Bay 8 = Critical] [06:58:50] PROBLEM - barengreza.my.id - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for barengreza.my.id could not be found [06:58:51] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.33, 8.25, 7.37 [06:58:57] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 9.41, 8.10, 6.71 [07:00:51] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.41, 7.78, 7.30 [07:00:56] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 5.58, 7.17, 6.54 [07:02:56] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.60, 7.69, 6.80 [07:04:56] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 5.39, 6.68, 6.53 [07:05:03] PROBLEM - mw142 Puppet on mw142 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[libboost-regex1.74.0-icu72] [07:05:35] [Grafana] !sre FIRING: The mediawiki job queue has more than 2500 unclaimed jobs https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [07:12:14] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: LOAD WARNING - total load average: 11.38, 9.86, 9.18 [07:14:51] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 4.80, 6.28, 6.80 [07:18:11] RECOVERY - mwtask141 Current Load on mwtask141 is OK: LOAD OK - total load average: 10.13, 10.16, 9.58 [07:18:52] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.91, 7.11, 7.01 [07:22:56] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.71, 6.96, 6.52 [07:24:56] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 6.10, 6.47, 6.39 [07:29:06] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: LOAD WARNING - total load average: 11.05, 10.62, 10.02 [07:31:02] RECOVERY - mw142 Puppet on mw142 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [07:32:51] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.09, 7.36, 7.55 [07:32:58] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.20, 6.50, 6.32 [07:34:56] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 5.99, 6.24, 6.25 [07:45:01] RECOVERY - mwtask141 Current Load on mwtask141 is OK: LOAD OK - total load average: 8.69, 9.94, 10.10 [07:46:32] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [07:50:51] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.38, 7.76, 7.29 [07:54:51] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.85, 7.20, 7.23 [08:02:51] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.04, 6.99, 7.03 [08:04:07] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.45, 6.94, 6.33 [08:04:51] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.64, 6.86, 6.98 [08:06:02] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 5.07, 6.16, 6.11 [08:08:51] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.35, 6.43, 6.76 [08:16:34] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns.ankh.fr.eu.org.', 'ns1.eu.org.', 'ns1.eriomem.net.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [08:19:44] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.22, 7.33, 6.94 [08:23:39] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.88, 7.82, 7.30 [08:25:36] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 7.94, 8.03, 7.44 [08:27:33] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.34, 7.41, 7.29 [08:31:29] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.24, 7.38, 7.27 [08:33:26] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.57, 6.74, 7.05 [08:39:17] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.46, 7.30, 7.10 [08:41:15] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.78, 7.00, 7.01 [08:42:41] PROBLEM - ns1 NTP time on ns1 is CRITICAL: NTP CRITICAL: Offset 0.5011704266 secs [08:47:07] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.14, 7.06, 6.97 [08:49:05] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.13, 6.89, 6.91 [08:51:03] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.37, 7.34, 7.07 [08:52:59] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.18, 7.48, 7.17 [08:56:54] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.47, 6.42, 6.79 [09:00:51] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.35, 7.76, 7.33 [09:06:51] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.06, 6.12, 6.73 [09:20:51] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 15.80, 10.26, 7.86 [09:23:39] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.71, 6.98, 6.06 [09:25:38] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 5.42, 6.34, 5.93 [09:28:51] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.20, 7.32, 7.51 [09:44:51] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.66, 6.19, 6.69 [09:52:51] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.51, 6.90, 6.82 [09:58:51] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.65, 6.65, 6.72 [10:20:24] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.53, 7.51, 6.74 [10:22:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.65, 7.58, 6.86 [10:23:29] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.61, 6.90, 6.07 [10:26:15] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.44, 8.03, 7.15 [10:27:29] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 6.16, 6.60, 6.14 [10:28:12] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.53, 7.40, 7.02 [10:36:01] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.40, 6.54, 6.78 [10:45:52] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.32, 6.82, 6.85 [10:47:49] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.20, 6.17, 6.61 [11:05:18] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03Reception123-patch-1 [+1/-0/±0] 13https://github.com/miraheze/puppet/commit/8ebdd50e1b87 [11:05:19] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.34, 7.08, 6.59 [11:05:20] [02miraheze/puppet] 07Reception123 038ebdd50 - Create rename-wiki.py [11:05:23] [02puppet] 07Reception123 created branch 03Reception123-patch-1 - 13https://github.com/miraheze/puppet [11:05:26] [02puppet] 07Reception123 opened pull request 03#3584: Create rename-wiki.py - 13https://github.com/miraheze/puppet/pull/3584 [11:07:16] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.51, 6.43, 6.41 [11:10:35] [Grafana] !sre FIRING: The mediawiki job queue has more than 2500 unclaimed jobs https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [11:11:51] [02puppet] 07RhinosF1 opened pull request 03#3585: rename-wiki: make less yucky - 13https://github.com/miraheze/puppet/pull/3585 [11:13:24] [02puppet] 07RhinosF1 synchronize pull request 03#3585: rename-wiki: make less yucky - 13https://github.com/miraheze/puppet/pull/3585 [11:14:36] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 7.00, 6.19, 5.54 [11:15:59] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.59, 7.23, 6.68 [11:16:36] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 4.61, 5.62, 5.40 [11:17:56] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.26, 6.69, 6.54 [11:21:51] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 14.12, 9.09, 7.42 [11:22:22] [02puppet] 07Reception123 synchronize pull request 03#3584: Create rename-wiki.py - 13https://github.com/miraheze/puppet/pull/3584 [11:22:24] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03Reception123-patch-1 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/8ebdd50e1b87...c6c5bdb48a98 [11:22:26] [02miraheze/puppet] 07Reception123 03c6c5bdb - merge @RhinosF1's changes [11:25:07] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.91, 7.79, 6.80 [11:25:14] [02puppet] 07RhinosF1 closed pull request 03#3585: rename-wiki: make less yucky - 13https://github.com/miraheze/puppet/pull/3585 [11:28:59] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 4.71, 6.48, 6.52 [11:29:36] [02puppet] 07RhinosF1 reviewed pull request 03#3584 commit - 13https://github.com/miraheze/puppet/pull/3584#discussion_r1442764424 [11:29:38] [02puppet] 07RhinosF1 reviewed pull request 03#3584 commit - 13https://github.com/miraheze/puppet/pull/3584#discussion_r1442767040 [11:29:40] [02puppet] 07RhinosF1 reviewed pull request 03#3584 commit - 13https://github.com/miraheze/puppet/pull/3584#discussion_r1442766841 [11:29:41] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.57, 7.40, 7.37 [11:29:41] [02puppet] 07RhinosF1 reviewed pull request 03#3584 commit - 13https://github.com/miraheze/puppet/pull/3584#discussion_r1442765895 [11:29:43] [02puppet] 07RhinosF1 reviewed pull request 03#3584 commit - 13https://github.com/miraheze/puppet/pull/3584#discussion_r1442765428 [11:29:46] [02puppet] 07RhinosF1 reviewed pull request 03#3584 commit - 13https://github.com/miraheze/puppet/pull/3584#discussion_r1442767380 [11:31:45] [02puppet] 07Reception123 reviewed pull request 03#3584 commit - 13https://github.com/miraheze/puppet/pull/3584#discussion_r1442769114 [11:34:27] [02puppet] 07RhinosF1 reviewed pull request 03#3584 commit - 13https://github.com/miraheze/puppet/pull/3584#discussion_r1442771294 [11:36:57] [02puppet] 07Reception123 synchronize pull request 03#3584: Create rename-wiki.py - 13https://github.com/miraheze/puppet/pull/3584 [11:36:59] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03Reception123-patch-1 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/c6c5bdb48a98...9c5c2d23e5b5 [11:37:00] [02miraheze/puppet] 07Reception123 039c5c2d2 - missing mysql -e [11:41:25] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.04, 7.06, 7.07 [11:43:22] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.07, 7.06, 7.07 [11:51:12] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 10.78, 8.49, 7.59 [11:53:09] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.54, 7.71, 7.41 [11:53:29] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.14, 7.13, 6.50 [11:55:06] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.21, 7.66, 7.41 [11:57:03] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.16, 6.70, 7.09 [11:57:29] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 5.78, 6.51, 6.41 [12:03:00] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.20, 6.77, 6.32 [12:04:56] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 6.04, 6.45, 6.25 [12:06:52] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.58, 6.44, 6.77 [12:15:24] PROBLEM - db101 Backups SQL on db101 is WARNING: FILE_AGE WARNING: /var/log/sql-backup.log is 864230 seconds old and 138761 bytes [12:18:51] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.02, 7.20, 6.83 [12:20:51] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.54, 8.28, 7.28 [12:22:37] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: LOAD WARNING - total load average: 11.93, 10.11, 8.87 [12:24:51] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.25, 7.88, 7.41 [12:26:36] RECOVERY - mwtask141 Current Load on mwtask141 is OK: LOAD OK - total load average: 10.16, 10.17, 9.18 [12:33:42] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 9.73, 8.15, 6.97 [12:36:36] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: LOAD CRITICAL - total load average: 12.53, 11.33, 9.97 [12:37:34] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.53, 7.46, 6.98 [12:37:59] PROBLEM - mw143 Current Load on mw143 is CRITICAL: LOAD CRITICAL - total load average: 12.78, 10.47, 6.84 [12:38:08] PROBLEM - mw141 Current Load on mw141 is CRITICAL: LOAD CRITICAL - total load average: 19.26, 11.54, 7.35 [12:39:24] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.65, 3.09, 1.86 [12:39:59] PROBLEM - mw143 Current Load on mw143 is WARNING: LOAD WARNING - total load average: 10.29, 10.60, 7.33 [12:40:08] PROBLEM - mw141 Current Load on mw141 is WARNING: LOAD WARNING - total load average: 8.78, 10.26, 7.39 [12:41:23] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 1.31, 2.47, 1.79 [12:41:26] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 5.03, 6.21, 6.58 [12:41:59] RECOVERY - mw143 Current Load on mw143 is OK: LOAD OK - total load average: 3.86, 8.01, 6.77 [12:42:08] RECOVERY - mw141 Current Load on mw141 is OK: LOAD OK - total load average: 4.15, 8.05, 6.93 [12:42:35] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: LOAD WARNING - total load average: 11.31, 11.52, 10.52 [12:44:51] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.27, 7.46, 7.17 [12:46:51] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.16, 7.23, 7.12 [12:48:41] RECOVERY - ns1 NTP time on ns1 is OK: NTP OK: Offset 0.002170860767 secs [12:50:51] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.76, 7.98, 7.44 [12:52:35] RECOVERY - mwtask141 Current Load on mwtask141 is OK: LOAD OK - total load average: 8.15, 9.46, 9.95 [12:54:51] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.06, 7.24, 7.30 [13:04:51] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.23, 7.17, 7.15 [13:06:35] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: LOAD WARNING - total load average: 10.49, 10.31, 10.11 [13:06:51] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.59, 7.57, 7.32 [13:07:53] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.83, 6.90, 6.62 [13:08:35] RECOVERY - mwtask141 Current Load on mwtask141 is OK: LOAD OK - total load average: 8.79, 9.67, 9.89 [13:09:49] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 5.03, 6.22, 6.41 [13:18:51] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 10.43, 7.67, 7.20 [13:27:31] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: LOAD WARNING - total load average: 11.55, 10.29, 9.70 [13:28:51] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.46, 7.88, 7.71 [13:29:30] RECOVERY - mwtask141 Current Load on mwtask141 is OK: LOAD OK - total load average: 9.85, 10.11, 9.71 [13:30:52] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.25, 8.17, 7.83 [13:35:27] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: LOAD WARNING - total load average: 11.20, 10.24, 9.82 [13:37:27] RECOVERY - mwtask141 Current Load on mwtask141 is OK: LOAD OK - total load average: 9.83, 10.03, 9.80 [13:38:51] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.16, 7.85, 7.86 [13:41:25] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: LOAD WARNING - total load average: 10.09, 11.02, 10.34 [13:42:51] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.36, 7.81, 7.81 [13:44:05] PROBLEM - mail121 Current Load on mail121 is CRITICAL: LOAD CRITICAL - total load average: 2.27, 1.57, 0.83 [13:44:34] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.49, 6.62, 6.10 [13:44:51] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.88, 8.00, 7.89 [13:45:24] RECOVERY - mwtask141 Current Load on mwtask141 is OK: LOAD OK - total load average: 9.15, 10.20, 10.17 [13:46:05] RECOVERY - mail121 Current Load on mail121 is OK: LOAD OK - total load average: 0.80, 1.27, 0.81 [13:46:33] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [13:46:51] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 7.74, 8.03, 7.93 [13:48:31] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 5.93, 6.47, 6.17 [13:48:51] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.67, 7.40, 7.70 [13:50:51] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.41, 7.69, 7.76 [13:52:51] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.54, 7.79, 7.81 [13:54:51] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.02, 7.93, 7.83 [13:56:51] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.50, 7.43, 7.67 [14:01:32] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.89, 6.85, 6.51 [14:03:30] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.14, 7.24, 6.69 [14:05:26] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.15, 7.09, 6.70 [14:05:59] RECOVERY - barengreza.my.id - LetsEncrypt on sslhost is OK: OK - Certificate 'barengreza.my.id' will expire on Sun 17 Mar 2024 17:33:19 GMT +0000. [14:10:51] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.40, 8.09, 7.54 [14:11:15] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.17, 8.43, 7.41 [14:12:51] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.97, 7.77, 7.47 [14:13:10] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.06, 7.88, 7.32 [14:13:26] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 5.65, 3.91, 2.11 [14:15:26] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 1.56, 2.94, 1.97 [14:16:32] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns.ankh.fr.eu.org.', 'ns1.eu.org.', 'ns1.eriomem.net.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [14:20:51] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.72, 8.09, 7.66 [14:23:10] PROBLEM - puppetdb121 Puppet on puppetdb121 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[puppetdb] [14:25:29] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.03, 6.60, 6.06 [14:28:51] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.42, 7.37, 7.69 [14:29:29] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 5.02, 6.09, 6.00 [14:31:12] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: LOAD CRITICAL - total load average: 13.30, 10.63, 9.25 [14:31:20] PROBLEM - mwtask141 MediaWiki Rendering on mwtask141 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:32:56] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 6.00, 6.43, 6.79 [14:33:12] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: LOAD WARNING - total load average: 10.78, 10.62, 9.42 [14:35:13] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: LOAD CRITICAL - total load average: 12.34, 11.13, 9.75 [14:37:16] PROBLEM - barengreza.my.id - LetsEncrypt on sslhost is CRITICAL: connect to address barengreza.my.id and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [14:37:45] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/importImages.php --wiki=shaperaversewiki /home/reception/shaperaverse --summary=Imported from https://shaperaverse.fandom.com --search-recursively (START) [14:37:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:38:51]