[00:01:44] PROBLEM - os141 Current Load on os141 is CRITICAL: CRITICAL - load average: 4.60, 4.09, 3.12 [00:01:59] PROBLEM - mw141 Current Load on mw141 is CRITICAL: CRITICAL - load average: 12.13, 10.11, 8.34 [00:02:15] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 11.50, 8.48, 4.63 [00:02:53] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 10.28, 12.02, 7.00 [00:02:56] PROBLEM - mw142 Current Load on mw142 is WARNING: WARNING - load average: 10.98, 9.06, 7.66 [00:03:13] RECOVERY - cp23 Disk Space on cp23 is OK: DISK OK - free space: / 11509 MB (15% inode=98%); [00:03:31] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 6.25, 6.86, 4.14 [00:03:48] RECOVERY - cp32 Disk Space on cp32 is OK: DISK OK - free space: / 11564 MB (15% inode=98%); [00:03:50] PROBLEM - cp22 Current Load on cp22 is CRITICAL: CRITICAL - load average: 6.08, 12.82, 6.99 [00:03:57] RECOVERY - mw141 Current Load on mw141 is OK: OK - load average: 9.21, 9.61, 8.37 [00:03:59] RECOVERY - cp22 Disk Space on cp22 is OK: DISK OK - free space: / 11601 MB (15% inode=98%); [00:04:40] RECOVERY - cp33 Disk Space on cp33 is OK: DISK OK - free space: / 11495 MB (15% inode=98%); [00:05:39] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 9.55, 7.67, 5.73 [00:06:49] RECOVERY - mw142 Current Load on mw142 is OK: OK - load average: 9.11, 9.49, 8.18 [00:07:39] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 7.53, 6.95, 5.67 [00:08:08] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 0.96, 3.94, 3.86 [00:09:21] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 1.92, 3.99, 3.72 [00:09:39] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 11.20, 8.46, 6.38 [00:12:08] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 0.97, 2.66, 3.37 [00:12:53] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:13:15] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.18, 2.58, 3.20 [00:14:41] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 0.84, 1.85, 3.68 [00:15:38] PROBLEM - cp22 Current Load on cp22 is WARNING: WARNING - load average: 0.94, 2.12, 3.77 [00:16:39] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 0.34, 1.35, 3.28 [00:17:38] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 0.70, 1.64, 3.39 [00:19:58] [02miraheze/puppet] 07The-Voidwalker pushed 031 commit to 03The-Voidwalker-patch-1 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/8497f01fb477 [00:19:59] [02miraheze/puppet] 07The-Voidwalker 038497f01 - Update robots.php [00:20:00] [02puppet] 07The-Voidwalker created branch 03The-Voidwalker-patch-1 - 13https://github.com/miraheze/puppet [00:20:01] [02puppet] 07The-Voidwalker opened pull request 03#3226: Update robots.php - 13https://github.com/miraheze/puppet/pull/3226 [00:21:20] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 7.671 second response time [00:29:56] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:31:58] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 4.835 second response time [00:54:21] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 4.71, 6.68, 7.87 [01:01:13] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 7.17, 7.42, 7.92 [01:04:22] PROBLEM - os141 PowerDNS Recursor on os141 is CRITICAL: CRITICAL - Plugin timed out while executing system call [01:06:23] RECOVERY - os141 PowerDNS Recursor on os141 is OK: DNS OK: 3.242 seconds response time. miraheze.org returns 109.228.51.216,217.174.247.33,2a00:da00:1800:326::1,2a00:da00:1800:328::1 [01:12:27] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:14:21] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 5.91, 5.99, 6.69 [01:14:33] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 9.120 second response time [01:15:53] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.47, 7.54, 8.00 [01:18:56] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:19:13] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 6.08, 6.22, 6.79 [01:25:10] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 2.186 second response time [01:29:53] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 9.21, 7.96, 7.82 [01:31:53] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.30, 7.46, 7.64 [01:37:53] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 8.81, 7.49, 7.52 [01:41:54] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 6.97, 7.51, 7.55 [01:43:10] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:45:09] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 1.657 second response time [01:46:07] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.57, 3.36, 2.64 [01:46:21] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 7.99, 7.07, 6.53 [01:47:38] PROBLEM - cp22 Current Load on cp22 is WARNING: WARNING - load average: 2.26, 3.74, 2.25 [01:48:21] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 5.41, 6.36, 6.34 [01:49:38] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 0.60, 2.64, 2.03 [01:50:00] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [01:50:07] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.73, 3.78, 2.97 [01:52:07] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 2.78, 3.43, 2.95 [01:53:38] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.andreijiroh.uk.eu.org. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [01:54:07] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.41, 3.11, 2.89 [02:06:40] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:08:42] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 6.553 second response time [02:14:58] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.63, 3.27, 2.95 [02:16:53] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 3.13, 3.23, 2.98 [02:21:53] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 6.12, 6.61, 6.79 [02:22:44] RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp23.miraheze.org - CNAME OK [02:26:08] PROBLEM - swiftobject112 Disk Space on swiftobject112 is CRITICAL: DISK CRITICAL - free space: / 29410 MB (5% inode=83%); [02:27:22] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 7.88, 7.22, 6.22 [02:27:39] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 5.09, 5.98, 7.97 [02:31:39] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 9.35, 6.55, 7.64 [02:31:47] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 6.92, 7.24, 7.03 [02:33:14] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 4.64, 6.13, 6.13 [02:33:39] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 5.60, 6.03, 7.31 [02:33:44] PROBLEM - os141 Current Load on os141 is WARNING: WARNING - load average: 3.30, 3.31, 3.96 [02:35:39] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 3.56, 4.90, 6.73 [02:35:44] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 8.57, 7.74, 7.25 [02:37:47] PROBLEM - os141 Current Load on os141 is CRITICAL: CRITICAL - load average: 6.84, 4.20, 4.09 [02:39:42] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 6.14, 7.23, 7.18 [02:39:44] PROBLEM - os141 Current Load on os141 is WARNING: WARNING - load average: 2.30, 3.67, 3.93 [02:43:44] RECOVERY - os141 Current Load on os141 is OK: OK - load average: 0.40, 1.90, 3.14 [02:45:14] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 7.07, 6.64, 6.03 [02:47:13] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 4.93, 6.08, 5.90 [02:57:31] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 6.01, 6.37, 6.76 [03:01:57] RECOVERY - db121 Backups SQL on db121 is OK: FILE_AGE OK: /var/log/sql-backup.log is 114 seconds old and 275 bytes [03:03:15] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.79, 3.57, 2.29 [03:05:15] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.82, 2.92, 2.22 [03:05:25] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.45, 7.07, 6.88 [03:07:24] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 6.54, 6.72, 6.77 [03:11:27] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query uk.religiononfire.mar.in.ua. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [03:14:08] PROBLEM - swiftobject112 Disk Space on swiftobject112 is WARNING: DISK WARNING - free space: / 32515 MB (6% inode=83%); [03:15:38] PROBLEM - cp22 Current Load on cp22 is CRITICAL: CRITICAL - load average: 7.70, 4.66, 2.36 [03:17:12] PROBLEM - cp22 APT on cp22 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [03:18:16] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 6.53, 6.90, 6.83 [03:19:12] RECOVERY - cp22 APT on cp22 is OK: APT OK: 35 packages available for upgrade (0 critical updates). [03:19:23] PROBLEM - swiftobject121 APT on swiftobject121 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [03:19:38] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 0.85, 2.94, 2.22 [03:21:24] PROBLEM - swiftobject121 APT on swiftobject121 is CRITICAL: APT CRITICAL: 52 packages available for upgrade (7 critical updates). [03:22:14] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 5.12, 6.58, 6.76 [03:27:28] PROBLEM - phab121 Current Load on phab121 is CRITICAL: CRITICAL - load average: 3.46, 2.54, 1.29 [03:29:28] PROBLEM - phab121 Current Load on phab121 is WARNING: WARNING - load average: 1.14, 1.90, 1.20 [03:31:28] RECOVERY - phab121 Current Load on phab121 is OK: OK - load average: 1.31, 1.64, 1.18 [03:36:05] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 8.19, 7.12, 6.80 [03:40:02] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.58, 7.58, 7.08 [03:41:24] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - uk.religiononfire.mar.in.ua All nameservers failed to answer the query. [03:44:04] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:45:59] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 6.04, 6.33, 6.67 [03:50:00] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [03:50:23] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 7.356 second response time [04:03:05] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:05:07] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 5.278 second response time [04:11:21] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [04:11:34] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:17:55] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 8.578 second response time [04:44:49] PROBLEM - db121 Current Load on db121 is CRITICAL: CRITICAL - load average: 8.41, 6.41, 5.03 [04:47:00] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [04:48:49] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 6.45, 6.91, 5.60 [04:50:49] RECOVERY - db121 Current Load on db121 is OK: OK - load average: 6.11, 6.66, 5.67 [04:53:34] PROBLEM - cp32 NTP time on cp32 is WARNING: NTP WARNING: Offset 0.1358028352 secs [05:00:49] PROBLEM - db121 Current Load on db121 is CRITICAL: CRITICAL - load average: 8.56, 7.13, 6.24 [05:02:14] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 8.18, 6.75, 6.19 [05:03:34] RECOVERY - cp32 NTP time on cp32 is OK: NTP OK: Offset 0.08653405309 secs [05:04:13] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 6.16, 6.48, 6.16 [05:10:49] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 6.00, 7.63, 7.08 [05:18:49] RECOVERY - db121 Current Load on db121 is OK: OK - load average: 5.25, 6.10, 6.62 [05:22:49] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 7.73, 7.07, 6.90 [05:22:58] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:25:03] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 7.630 second response time [05:32:49] PROBLEM - db121 Current Load on db121 is CRITICAL: CRITICAL - load average: 8.22, 7.73, 7.35 [05:35:53] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.16, 6.97, 6.25 [05:37:53] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 6.26, 6.71, 6.24 [05:38:49] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 6.44, 7.46, 7.41 [05:42:00] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [05:48:49] RECOVERY - db121 Current Load on db121 is OK: OK - load average: 6.20, 6.36, 6.80 [05:53:37] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.andreijiroh.uk.eu.org. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [05:59:31] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 7.33, 6.81, 6.74 [06:01:26] RECOVERY - db121 Current Load on db121 is OK: OK - load average: 4.94, 6.08, 6.48 [06:05:38] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.27, 6.92, 6.39 [06:07:37] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 5.47, 6.32, 6.23 [06:16:48] PROBLEM - cp22 APT on cp22 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:17:39] PROBLEM - cp22 Current Load on cp22 is CRITICAL: CRITICAL - load average: 9.28, 6.15, 3.13 [06:18:49] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 7.88, 6.59, 6.17 [06:18:53] RECOVERY - cp22 APT on cp22 is OK: APT OK: 35 packages available for upgrade (0 critical updates). [06:20:49] RECOVERY - db121 Current Load on db121 is OK: OK - load average: 5.34, 6.10, 6.04 [06:21:38] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 0.71, 3.39, 2.71 [06:22:43] RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp23.miraheze.org - CNAME OK [06:23:40] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 4.04, 6.12, 7.56 [06:25:40] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.28, 7.08, 7.73 [06:26:49] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 7.21, 6.78, 6.34 [06:27:39] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.17, 7.29, 7.74 [06:28:49] PROBLEM - db121 Current Load on db121 is CRITICAL: CRITICAL - load average: 8.65, 7.39, 6.61 [06:30:30] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [06:30:49] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 6.16, 6.95, 6.55 [06:31:39] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.19, 7.51, 7.71 [06:32:49] PROBLEM - db121 Current Load on db121 is CRITICAL: CRITICAL - load average: 8.39, 7.44, 6.77 [06:33:40] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.26, 7.33, 7.62 [06:34:49] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 7.74, 7.43, 6.85 [06:35:40] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.67, 8.37, 7.97 [06:40:49] RECOVERY - db121 Current Load on db121 is OK: OK - load average: 5.29, 6.22, 6.54 [06:49:48] PROBLEM - swiftobject121 APT on swiftobject121 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [06:51:48] PROBLEM - swiftobject121 APT on swiftobject121 is CRITICAL: APT CRITICAL: 52 packages available for upgrade (7 critical updates). [07:05:00] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 6.99, 6.87, 6.38 [07:05:40] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.49, 7.25, 7.94 [07:06:58] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 6.29, 6.71, 6.39 [07:07:40] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 7.90, 7.72, 8.03 [07:10:10] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 6.89, 7.03, 6.66 [07:12:06] PROBLEM - db121 Current Load on db121 is CRITICAL: CRITICAL - load average: 10.05, 8.13, 7.10 [07:17:51] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 6.81, 7.52, 7.22 [07:25:30] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [07:27:27] RECOVERY - db121 Current Load on db121 is OK: OK - load average: 5.69, 6.34, 6.80 [07:31:20] PROBLEM - db121 Current Load on db121 is CRITICAL: CRITICAL - load average: 8.08, 7.18, 7.03 [07:32:24] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:33:15] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 7.23, 7.16, 7.04 [07:34:29] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 7.157 second response time [07:44:49] PROBLEM - db121 Current Load on db121 is CRITICAL: CRITICAL - load average: 9.66, 7.84, 7.37 [07:46:49] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 6.21, 7.15, 7.17 [07:48:49] RECOVERY - db121 Current Load on db121 is OK: OK - load average: 4.51, 6.17, 6.80 [07:49:39] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.59, 7.52, 7.95 [07:51:40] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 7.93, 7.84, 8.03 [07:52:49] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 6.47, 6.64, 6.87 [07:54:49] RECOVERY - db121 Current Load on db121 is OK: OK - load average: 6.39, 6.52, 6.79 [08:07:40] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.24, 7.35, 7.89 [08:15:30] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [08:19:39] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 10.47, 8.19, 7.84 [08:21:40] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.02, 7.99, 7.84 [08:33:39] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 10.01, 7.74, 7.47 [08:41:04] [02dns] 07jchristin opened pull request 03#417: Update MX fields for arru.xyz - 13https://github.com/miraheze/dns/pull/417 [08:47:38] PROBLEM - cp22 Current Load on cp22 is CRITICAL: CRITICAL - load average: 3.64, 5.09, 2.98 [08:49:30] PROBLEM - swiftobject121 APT on swiftobject121 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [08:49:38] PROBLEM - cp22 Current Load on cp22 is WARNING: WARNING - load average: 1.47, 3.73, 2.73 [08:51:30] PROBLEM - swiftobject121 APT on swiftobject121 is CRITICAL: APT CRITICAL: 52 packages available for upgrade (7 critical updates). [08:51:38] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 0.83, 2.68, 2.46 [09:01:44] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:03:48] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 6.984 second response time [09:12:23] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:14:26] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 5.862 second response time [09:21:47] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - en.religiononfire.mar.in.ua All nameservers failed to answer the query. [09:24:59] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query cn.eu.org. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [09:30:30] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [09:43:40] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.67, 7.30, 7.92 [09:47:40] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 10.28, 8.05, 8.02 [09:49:49] PROBLEM - swiftobject121 APT on swiftobject121 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [09:51:31] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [09:51:49] PROBLEM - swiftobject121 APT on swiftobject121 is CRITICAL: APT CRITICAL: 52 packages available for upgrade (7 critical updates). [09:54:37] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.nj.cn.eu.org All nameservers failed to answer the query. [10:14:45] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mar.in.ua. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [10:19:50] PROBLEM - swiftobject121 APT on swiftobject121 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [10:21:50] PROBLEM - swiftobject121 APT on swiftobject121 is CRITICAL: APT CRITICAL: 52 packages available for upgrade (7 critical updates). [10:23:17] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query en.religiononfire.mar.in.ua. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [10:24:17] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query cn.eu.org. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [10:37:39] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.41, 7.06, 7.89 [10:37:53] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 6.14, 6.81, 6.42 [10:39:53] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 5.70, 6.30, 6.27 [10:44:42] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [10:47:40] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 10.12, 8.37, 7.97 [10:53:03] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [10:53:54] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.nj.cn.eu.org All nameservers failed to answer the query. [11:09:43] [02miraheze/RemovePII] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/RemovePII/compare/84f03c952aa3...544945d3eff9 [11:09:46] [02miraheze/RemovePII] 07translatewiki 03544945d - Localisation updates from https://translatewiki.net. [11:09:49] [02miraheze/ManageWiki] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ManageWiki/compare/bacf944e6c0f...6f33a28904e1 [11:09:50] [02miraheze/ManageWiki] 07translatewiki 036f33a28 - Localisation updates from https://translatewiki.net. [11:09:51] [02miraheze/MirahezeMagic] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/MirahezeMagic/compare/b3116cc9f5ae...387bc748324d [11:09:53] [02miraheze/MirahezeMagic] 07translatewiki 03387bc74 - Localisation updates from https://translatewiki.net. [11:15:38] PROBLEM - cp22 Current Load on cp22 is CRITICAL: CRITICAL - load average: 7.02, 4.20, 2.59 [11:16:35] miraheze/ManageWiki - translatewiki the build passed. [11:16:47] PROBLEM - wiki.nj.cn.eu.org - LetsEncrypt on sslhost is CRITICAL: connect to address wiki.nj.cn.eu.org and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [11:17:38] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 1.90, 3.25, 2.43 [11:23:33] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.nj.cn.eu.org. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [11:32:47] miraheze/RemovePII - translatewiki the build passed. [11:32:50] miraheze/MirahezeMagic - translatewiki the build passed. [11:36:10] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 6.63, 4.26, 2.21 [11:38:08] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.98, 3.80, 2.28 [11:40:08] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.46, 3.32, 2.29 [11:46:29] RECOVERY - wiki.nj.cn.eu.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.nj.cn.eu.org' will expire on Tue 25 Jul 2023 15:32:36 GMT +0000. [11:49:57] PROBLEM - swiftobject121 APT on swiftobject121 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [11:51:56] PROBLEM - swiftobject121 APT on swiftobject121 is CRITICAL: APT CRITICAL: 52 packages available for upgrade (7 critical updates). [11:53:11] RECOVERY - wiki.nj.cn.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.nj.cn.eu.org reverse DNS resolves to cp23.miraheze.org - CNAME OK [11:59:01] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [12:03:40] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 4.96, 6.76, 7.94 [12:07:40] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.64, 8.10, 8.18 [12:07:57] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.59, 7.14, 6.60 [12:09:56] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 5.90, 6.77, 6.53 [12:13:16] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:15:23] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 9.192 second response time [12:37:39] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 8.67, 5.84, 3.22 [12:37:44] PROBLEM - os141 Current Load on os141 is CRITICAL: CRITICAL - load average: 6.01, 3.26, 1.61 [12:41:39] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 1.86, 5.21, 3.71 [12:41:44] RECOVERY - os141 Current Load on os141 is OK: OK - load average: 0.98, 2.85, 1.91 [12:45:39] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.15, 7.16, 7.88 [12:47:40] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.22, 7.65, 7.96 [12:49:40] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.71, 7.67, 7.93 [12:51:40] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 7.93, 7.94, 8.01 [12:53:39] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.68, 7.74, 7.92 [12:57:40] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 7.77, 8.22, 8.08 [13:01:15] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:03:19] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 6.066 second response time [13:04:43] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:06:41] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 1.616 second response time [13:07:41] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:09:44] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 5.678 second response time [13:15:38] PROBLEM - cp22 Current Load on cp22 is CRITICAL: CRITICAL - load average: 5.46, 3.41, 1.98 [13:16:58] PROBLEM - cp22 APT on cp22 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:19:04] RECOVERY - cp22 APT on cp22 is OK: APT OK: 35 packages available for upgrade (0 critical updates). [13:19:38] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 2.23, 3.38, 2.36 [13:39:40] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.20, 7.10, 7.84 [13:43:51] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:45:39] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.30, 7.46, 7.69 [13:47:39] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.16, 7.22, 7.58 [13:50:11] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 7.776 second response time [13:51:40] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.05, 8.58, 8.06 [14:05:35] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.18, 6.85, 6.44 [14:12:50] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:13:31] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 9.41, 7.22, 6.72 [14:14:55] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 7.184 second response time [14:15:30] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.78, 7.22, 6.77 [14:17:29] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 6.29, 6.80, 6.67 [14:25:39] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 9.23, 5.52, 2.94 [14:26:22] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 8.64, 7.44, 6.86 [14:27:39] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 7.67, 6.14, 3.47 [14:27:45] PROBLEM - os141 Current Load on os141 is CRITICAL: CRITICAL - load average: 4.43, 3.45, 1.83 [14:28:21] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 6.60, 7.04, 6.78 [14:29:39] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 1.93, 4.51, 3.19 [14:29:44] RECOVERY - os141 Current Load on os141 is OK: OK - load average: 0.73, 2.38, 1.64 [14:30:20] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 5.37, 6.50, 6.62 [14:41:57] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:48:15] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 5.839 second response time [14:56:08] PROBLEM - swiftobject112 Disk Space on swiftobject112 is CRITICAL: DISK CRITICAL - free space: / 28691 MB (5% inode=83%); [14:58:21] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 7.11, 6.54, 5.80 [15:00:21] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 6.05, 6.17, 5.75 [15:01:40] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 5.17, 3.58, 2.34 [15:03:04] PROBLEM - cp23 APT on cp23 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:05:02] RECOVERY - cp23 APT on cp23 is OK: APT OK: 35 packages available for upgrade (0 critical updates). [15:05:40] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 2.18, 3.28, 2.54 [15:06:57] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:06:57] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.19, 7.00, 6.52 [15:11:07] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 9.186 second response time [15:12:38] PROBLEM - os141 Current Load on os141 is CRITICAL: CRITICAL - load average: 5.00, 3.30, 1.94 [15:12:41] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:12:54] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 5.79, 6.52, 6.49 [15:14:09] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 10.89, 7.46, 5.02 [15:14:41] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 2.971 second response time [15:18:26] PROBLEM - os141 Current Load on os141 is WARNING: WARNING - load average: 2.43, 3.44, 2.49 [15:20:01] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 7.90, 7.92, 6.11 [15:20:22] RECOVERY - os141 Current Load on os141 is OK: OK - load average: 1.78, 2.86, 2.39 [15:23:08] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:23:55] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 1.61, 5.27, 5.51 [15:34:01] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [15:37:59] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 9.468 second response time [15:44:29] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:48:42] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 7.844 second response time [16:03:04] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:03:30] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [16:08:30] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [16:11:39] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 10.92, 6.10, 3.45 [16:15:39] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 5.55, 6.87, 4.47 [16:17:39] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 8.06, 6.72, 4.67 [16:19:39] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 3.75, 5.58, 4.51 [16:23:33] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:23:49] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 1.393 second response time [16:31:39] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.86, 7.10, 7.99 [16:33:40] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.29, 7.95, 8.19 [16:34:02] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.974 second response time [16:36:08] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:42:22] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:42:23] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 8.385 second response time [16:45:45] PROBLEM - os141 Current Load on os141 is WARNING: WARNING - load average: 3.58, 3.14, 2.61 [16:48:35] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [16:48:48] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:49:44] RECOVERY - os141 Current Load on os141 is OK: OK - load average: 2.01, 2.92, 2.68 [16:50:45] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 3.416 second response time [16:51:05] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - uk.religiononfire.mar.in.ua All nameservers failed to answer the query. [16:51:39] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.55, 7.46, 7.96 [16:53:07] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 9.41, 7.57, 6.35 [16:53:49] PROBLEM - os141 Current Load on os141 is WARNING: WARNING - load average: 3.33, 3.45, 2.97 [16:55:04] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 7.46, 7.78, 6.59 [16:55:45] RECOVERY - os141 Current Load on os141 is OK: OK - load average: 2.69, 3.34, 3.00 [16:57:01] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 9.83, 7.68, 6.63 [16:57:39] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 11.88, 8.63, 8.18 [16:58:29] PROBLEM - matomo121 Current Load on matomo121 is CRITICAL: CRITICAL - load average: 4.45, 3.40, 2.75 [16:58:58] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 7.45, 7.54, 6.71 [16:59:14] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 8.406 second response time [16:59:49] PROBLEM - os141 Current Load on os141 is CRITICAL: CRITICAL - load average: 4.05, 3.69, 3.23 [17:00:29] PROBLEM - matomo121 Current Load on matomo121 is WARNING: WARNING - load average: 3.76, 3.41, 2.83 [17:00:56] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 10.50, 8.51, 7.16 [17:01:45] RECOVERY - os141 Current Load on os141 is OK: OK - load average: 2.29, 3.33, 3.16 [17:02:49] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.63, 6.67, 6.12 [17:02:53] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 7.50, 7.49, 6.92 [17:03:26] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 5903 bytes in 0.019 second response time [17:04:29] RECOVERY - matomo121 Current Load on matomo121 is OK: OK - load average: 1.96, 2.78, 2.72 [17:04:48] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 5.96, 6.51, 6.13 [17:06:00] PROBLEM - os141 Current Load on os141 is WARNING: WARNING - load average: 3.59, 3.62, 3.33 [17:06:47] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 8.46, 7.81, 7.18 [17:07:59] PROBLEM - os141 Current Load on os141 is CRITICAL: CRITICAL - load average: 4.10, 3.94, 3.49 [17:08:35] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [17:12:39] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 5.16, 7.47, 7.34 [17:13:48] PROBLEM - os141 Current Load on os141 is WARNING: WARNING - load average: 2.10, 3.60, 3.58 [17:13:51] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 7.934 second response time [17:15:26] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:17:32] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 9.132 second response time [17:18:14] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:18:31] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 4.58, 6.20, 6.79 [17:21:03] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [17:22:23] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 12.92, 8.97, 7.71 [17:24:28] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 7.594 second response time [17:26:17] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 6.77, 7.84, 7.53 [17:27:53] PROBLEM - os141 Current Load on os141 is CRITICAL: CRITICAL - load average: 4.85, 3.81, 3.59 [17:28:14] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 8.45, 8.25, 7.73 [17:30:11] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:30:11] PROBLEM - os141 PowerDNS Recursor on os141 is CRITICAL: CRITICAL - Plugin timed out while executing system call [17:32:10] RECOVERY - os141 PowerDNS Recursor on os141 is OK: DNS OK: 0.880 seconds response time. miraheze.org returns 109.228.51.216,217.174.247.33,2a00:da00:1800:326::1,2a00:da00:1800:328::1 [17:32:52] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:34:23] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 7.644 second response time [17:35:30] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 8.26, 7.00, 6.58 [17:36:03] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 7.06, 7.91, 7.90 [17:37:29] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 5.66, 6.57, 6.48 [17:37:46] PROBLEM - os141 Current Load on os141 is WARNING: WARNING - load average: 3.68, 3.97, 3.84 [17:38:01] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 8.48, 7.65, 7.78 [17:39:35] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [17:41:11] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 7.342 second response time [17:41:55] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 4.91, 6.74, 7.48 [17:43:49] i have a question will we still have our wiki projects in october or no? [17:46:49] PROBLEM - cp22 APT on cp22 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:47:38] PROBLEM - cp22 Current Load on cp22 is CRITICAL: CRITICAL - load average: 4.60, 3.86, 2.33 [17:47:44] RECOVERY - os141 Current Load on os141 is OK: OK - load average: 2.46, 2.91, 3.38 [17:47:47] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 3.55, 5.18, 6.62 [17:48:03] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:48:50] RECOVERY - cp22 APT on cp22 is OK: APT OK: 35 packages available for upgrade (0 critical updates). [17:49:38] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 1.63, 2.92, 2.17 [17:52:59] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mar.in.ua. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [17:56:14] Cocopuff2018: you are on thin ice. You know the answer. Behave or your sanctions will escalate. [17:56:37] what did i do this time @RhinosF1 [17:59:00] [02dns] 07paladox closed pull request 03#396: Add opensearch to dns - 13https://github.com/miraheze/dns/pull/396 [18:04:54] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 1.184 second response time [18:05:15] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:07:27] PROBLEM - spcodex.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for spcodex.wiki could not be found [18:07:39] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 8.66, 6.10, 5.45 [18:07:46] PROBLEM - os141 Current Load on os141 is CRITICAL: CRITICAL - load average: 6.49, 4.39, 3.47 [18:11:06] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 5903 bytes in 0.009 second response time [18:12:07] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.70, 2.78, 1.71 [18:14:12] PROBLEM - wiki.ravynos.com - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.ravynos.com could not be found [18:14:50] PROBLEM - mw142 Current Load on mw142 is CRITICAL: CRITICAL - load average: 12.65, 10.67, 8.17 [18:15:00] PROBLEM - mw142 MediaWiki Rendering on mw142 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:15:06] PROBLEM - mw141 MediaWiki Rendering on mw141 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:16:07] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.71, 3.01, 2.09 [18:17:02] RECOVERY - mw141 MediaWiki Rendering on mw141 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.376 second response time [18:18:49] PROBLEM - mw142 Current Load on mw142 is WARNING: WARNING - load average: 11.06, 10.99, 8.90 [18:19:09] RECOVERY - mw142 MediaWiki Rendering on mw142 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.260 second response time [18:19:35] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [18:20:49] RECOVERY - mw142 Current Load on mw142 is OK: OK - load average: 8.09, 9.81, 8.71 [18:21:39] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 7.44, 7.83, 7.37 [18:22:57] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [18:23:39] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 9.75, 8.98, 7.88 [18:25:52] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 3.626 second response time [18:28:02] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 6.697 second response time [18:31:26] PROBLEM - os141 PowerDNS Recursor on os141 is CRITICAL: CRITICAL - Plugin timed out while executing system call [18:32:20] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:33:28] RECOVERY - os141 PowerDNS Recursor on os141 is OK: DNS OK: 3.970 seconds response time. miraheze.org returns 109.228.51.216,217.174.247.33,2a00:da00:1800:326::1,2a00:da00:1800:328::1 [18:33:40] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 3.98, 7.11, 7.75 [18:34:25] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:35:39] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 8.60, 7.82, 7.93 [18:36:35] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 9.609 second response time [18:37:39] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 6.74, 7.80, 7.94 [18:38:34] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 8.254 second response time [18:40:29] PROBLEM - matomo121 Current Load on matomo121 is WARNING: WARNING - load average: 3.91, 3.01, 2.47 [18:40:57] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:41:39] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 2.09, 4.96, 6.74 [18:41:44] PROBLEM - os141 Current Load on os141 is WARNING: WARNING - load average: 1.31, 2.51, 3.78 [18:42:29] RECOVERY - matomo121 Current Load on matomo121 is OK: OK - load average: 3.39, 3.09, 2.57 [18:45:44] RECOVERY - os141 Current Load on os141 is OK: OK - load average: 1.69, 2.20, 3.35 [18:49:39] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 8.53, 6.99, 6.93 [18:51:39] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 7.06, 6.93, 6.92 [18:53:07] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [18:55:39] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 8.62, 7.86, 7.30 [18:56:03] [02miraheze/puppet] 07The-Voidwalker pushed 031 commit to 03The-Voidwalker-patch-1 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/8497f01fb477...cd9b28ec5a0f [18:56:05] [02miraheze/puppet] 07The-Voidwalker 03cd9b28e - block some additional bots [18:56:07] [02puppet] 07The-Voidwalker synchronize pull request 03#3226: Update robots.php - 13https://github.com/miraheze/puppet/pull/3226 [18:57:39] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 5.62, 7.40, 7.23 [18:59:39] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 14.24, 9.52, 7.99 [18:59:40] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 6.95, 3.86, 2.18 [18:59:53] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.69, 6.72, 6.13 [19:00:21] PROBLEM - os141 PowerDNS Recursor on os141 is CRITICAL: CRITICAL - Plugin timed out while executing system call [19:01:44] PROBLEM - os141 Current Load on os141 is CRITICAL: CRITICAL - load average: 5.20, 4.18, 3.51 [19:01:53] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 5.79, 6.37, 6.07 [19:02:20] RECOVERY - os141 PowerDNS Recursor on os141 is OK: DNS OK: 0.248 seconds response time. miraheze.org returns 109.228.51.216,217.174.247.33,2a00:da00:1800:326::1,2a00:da00:1800:328::1 [19:03:44] PROBLEM - os141 Current Load on os141 is WARNING: WARNING - load average: 3.12, 3.80, 3.46 [19:04:07] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 6.285 second response time [19:07:40] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 0.95, 3.12, 2.93 [19:07:56] [02puppet] 07paladox opened pull request 03#3227: Revert "remove paladox shell and icinga access" - 13https://github.com/miraheze/puppet/pull/3227 [19:09:16] [02miraheze/puppet] 07The-Voidwalker pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/puppet/compare/ec999b36bed3...1989cb0f7ce0 [19:09:18] [02miraheze/puppet] 07paladox 031989cb0 - Revert "remove paladox shell and icinga access" (#3227) [19:09:21] [02puppet] 07The-Voidwalker closed pull request 03#3227: Revert "remove paladox shell and icinga access" - 13https://github.com/miraheze/puppet/pull/3227 [19:09:39] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 3.84, 7.28, 7.81 [19:09:44] RECOVERY - os141 Current Load on os141 is OK: OK - load average: 1.49, 2.88, 3.19 [19:10:33] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:11:39] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.00, 7.00, 7.92 [19:11:53] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.62, 7.44, 6.72 [19:13:39] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 4.03, 5.16, 6.77 [19:15:40] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.22, 8.07, 8.13 [19:16:55] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 9.424 second response time [19:17:53] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 4.92, 6.34, 6.49 [19:19:17] [02miraheze/dns] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://github.com/miraheze/dns/commit/4e2eec81d45d [19:19:20] [02miraheze/dns] 07paladox 034e2eec8 - Add mw133/34 and mw143 to dns [19:19:21] [02dns] 07paladox created branch 03paladox-patch-2 - 13https://github.com/miraheze/dns [19:19:23] [02dns] 07paladox opened pull request 03#418: Add mw133/34 and mw143 to dns - 13https://github.com/miraheze/dns/pull/418 [19:21:17] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:23:22] PROBLEM - puppet141 Current Load on puppet141 is WARNING: WARNING - load average: 3.59, 3.18, 2.32 [19:23:38] [02puppet] 07The-Voidwalker closed pull request 03#3226: Update robots.php - 13https://github.com/miraheze/puppet/pull/3226 [19:23:39] [02miraheze/puppet] 07The-Voidwalker pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/1989cb0f7ce0...8091aaed8104 [19:23:41] [02miraheze/puppet] 07The-Voidwalker 038091aae - Update robots.php (#3226) [19:23:43] [02miraheze/puppet] 07The-Voidwalker deleted branch 03The-Voidwalker-patch-1 [19:23:46] [02puppet] 07The-Voidwalker deleted branch 03The-Voidwalker-patch-1 - 13https://github.com/miraheze/puppet [19:25:22] PROBLEM - puppet141 Current Load on puppet141 is CRITICAL: CRITICAL - load average: 4.51, 3.78, 2.66 [19:25:30] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 8.323 second response time [19:26:25] [02dns] 07paladox closed pull request 03#418: Add mw133/34 and mw143 to dns - 13https://github.com/miraheze/dns/pull/418 [19:26:28] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/01e254b45427...8dc62737e801 [19:26:30] [02miraheze/dns] 07paladox 038dc6273 - Add mw133/34 and mw143 to dns (#418) [19:26:31] [02miraheze/dns] 07paladox deleted branch 03paladox-patch-2 [19:26:34] [02dns] 07paladox deleted branch 03paladox-patch-2 - 13https://github.com/miraheze/dns [19:27:21] PROBLEM - puppet141 Current Load on puppet141 is WARNING: WARNING - load average: 3.12, 3.59, 2.73 [19:28:19] [02miraheze/dns] 07The-Voidwalker pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/8dc62737e801...b9499977a3f8 [19:28:20] [02miraheze/dns] 07jchristin 03b949997 - Update MX fields for arru.xyz (#417) [19:28:21] [02dns] 07The-Voidwalker closed pull request 03#417: Update MX fields for arru.xyz - 13https://github.com/miraheze/dns/pull/417 [19:28:50] [02miraheze/dns] 07The-Voidwalker pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/b9499977a3f8...f98e52241db1 [19:28:52] [02miraheze/dns] 07MacFan4000 03f98e522 - add www.pastport.org (#416) [19:28:54] [02dns] 07The-Voidwalker closed pull request 03#416: add www.pastport.org - 13https://github.com/miraheze/dns/pull/416 [19:29:19] RECOVERY - puppet141 Current Load on puppet141 is OK: OK - load average: 2.48, 3.06, 2.63 [19:32:41] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:33:38] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 8.12, 6.84, 6.61 [19:35:35] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 6.66, 6.66, 6.57 [19:36:21] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:38:59] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 8.278 second response time [19:39:28] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 7.18, 5.30, 4.62 [19:39:43] PROBLEM - os141 Current Load on os141 is WARNING: WARNING - load average: 3.75, 2.80, 2.11 [19:41:27] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 3.89, 4.72, 4.48 [19:41:43] RECOVERY - os141 Current Load on os141 is OK: OK - load average: 1.37, 2.29, 2.01 [19:45:37] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/b9aa77715bcb [19:45:39] [02miraheze/puppet] 07paladox 03b9aa777 - deploy-mediawiki: add mw133/34 [19:45:42] [02puppet] 07paladox created branch 03paladox-patch-4 - 13https://github.com/miraheze/puppet [19:45:43] [02puppet] 07paladox opened pull request 03#3228: deploy-mediawiki: add mw133/34 - 13https://github.com/miraheze/puppet/pull/3228 [19:46:01] PROBLEM - mail121 Current Load on mail121 is CRITICAL: CRITICAL - load average: 2.10, 1.59, 1.08 [19:46:30] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:46:53] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/8091aaed8104...effc4e1f3c1a [19:46:54] [02miraheze/puppet] 07paladox 03effc4e1 - deploy-mediawiki: add mw133/34 (#3228) [19:46:56] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-4 [19:46:57] [02puppet] 07paladox closed pull request 03#3228: deploy-mediawiki: add mw133/34 - 13https://github.com/miraheze/puppet/pull/3228 [19:46:58] [02puppet] 07paladox deleted branch 03paladox-patch-4 - 13https://github.com/miraheze/puppet [19:47:05] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 6.134 second response time [19:48:01] RECOVERY - mail121 Current Load on mail121 is OK: OK - load average: 0.95, 1.28, 1.02 [19:48:34] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 6.985 second response time [19:51:18] PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[gdnsd-syntax] [19:54:40] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/f98e52241db1...72677a69d1a4 [19:54:41] [02miraheze/dns] 07paladox 0372677a6 - Fix arru.xyz [19:55:13] PROBLEM - mw133 ferm_active on mw133 is UNKNOWN: NRPE: Unable to read output [19:55:17] PROBLEM - mw134 Puppet on mw134 is UNKNOWN: NRPE: Unable to read output [19:55:20] PROBLEM - mw133 conntrack_table_size on mw133 is UNKNOWN: NRPE: Unable to read output [19:55:22] PROBLEM - mw134 conntrack_table_size on mw134 is UNKNOWN: NRPE: Unable to read output [19:55:27] PROBLEM - mw134 ferm_active on mw134 is UNKNOWN: NRPE: Unable to read output [19:55:42] PROBLEM - mw133 Puppet on mw133 is UNKNOWN: NRPE: Unable to read output [19:55:55] RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [19:56:27] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:57:14] RECOVERY - mw134 Puppet on mw134 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [19:57:15] RECOVERY - mw133 conntrack_table_size on mw133 is OK: OK: nf_conntrack is 0 % full [19:57:17] RECOVERY - mw134 conntrack_table_size on mw134 is OK: OK: nf_conntrack is 0 % full [19:57:26] RECOVERY - mw134 ferm_active on mw134 is OK: OK ferm input default policy is set [19:57:36] RECOVERY - mw133 Puppet on mw133 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [19:59:01] RECOVERY - mw133 ferm_active on mw133 is OK: OK ferm input default policy is set [19:59:45] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/d8c669549638 [19:59:46] [02miraheze/puppet] 07paladox 03d8c6695 - Add mw133/34/43 [19:59:48] [02puppet] 07paladox created branch 03paladox-patch-4 - 13https://github.com/miraheze/puppet [19:59:49] [02puppet] 07paladox opened pull request 03#3229: Add mw133/34/43 - 13https://github.com/miraheze/puppet/pull/3229 [20:00:20] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+1/-0/±0] 13https://github.com/miraheze/puppet/compare/d8c669549638...6cd9b7199e77 [20:00:21] [02miraheze/puppet] 07paladox 036cd9b71 - Create mw143.yaml [20:00:24] [02puppet] 07paladox synchronize pull request 03#3229: Add mw133/34/43 - 13https://github.com/miraheze/puppet/pull/3229 [20:00:29] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+1/-0/±0] 13https://github.com/miraheze/puppet/compare/6cd9b7199e77...7d1fa4fa83ec [20:00:31] [02miraheze/puppet] 07paladox 037d1fa4f - Create mw133.yaml [20:00:32] [02puppet] 07paladox synchronize pull request 03#3229: Add mw133/34/43 - 13https://github.com/miraheze/puppet/pull/3229 [20:00:40] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+1/-0/±0] 13https://github.com/miraheze/puppet/compare/7d1fa4fa83ec...664c799ac41c [20:00:42] [02miraheze/puppet] 07paladox 03664c799 - Create mw134.yaml [20:00:44] [02puppet] 07paladox synchronize pull request 03#3229: Add mw133/34/43 - 13https://github.com/miraheze/puppet/pull/3229 [20:01:01] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+3/-0/±1] 13https://github.com/miraheze/puppet/compare/effc4e1f3c1a...1d7e1e5ce147 [20:01:02] [02miraheze/puppet] 07paladox 031d7e1e5 - Add mw133/34/43 (#3229) [20:01:05] [02puppet] 07paladox closed pull request 03#3229: Add mw133/34/43 - 13https://github.com/miraheze/puppet/pull/3229 [20:01:06] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-4 [20:01:07] [02puppet] 07paladox deleted branch 03paladox-patch-4 - 13https://github.com/miraheze/puppet [20:01:47] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 7.15, 6.38, 4.80 [20:03:43] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 11.68, 8.51, 5.78 [20:05:55] PROBLEM - os141 Current Load on os141 is CRITICAL: CRITICAL - load average: 4.07, 3.36, 2.42 [20:06:29] PROBLEM - puppet141 Current Load on puppet141 is CRITICAL: CRITICAL - load average: 6.30, 3.94, 2.52 [20:07:12] PROBLEM - ldap141 Puppet on ldap141 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:13:09] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [20:14:32] PROBLEM - mem141 Current Load on mem141 is WARNING: WARNING - load average: 1.91, 1.43, 0.85 [20:15:06] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:16:31] RECOVERY - mem141 Current Load on mem141 is OK: OK - load average: 1.64, 1.61, 0.99 [20:16:35] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 4.89, 6.40, 7.74 [20:19:22] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 8.970 second response time [20:19:50] PROBLEM - puppet141 Current Load on puppet141 is WARNING: WARNING - load average: 2.27, 3.77, 3.70 [20:20:27] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 10.09, 7.79, 7.92 [20:21:16] PROBLEM - mw133 Puppet on mw133 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): File[/srv/mediawiki/w/skins/Femiwiki/node_modules],Service[cgroup] [20:21:24] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 5.809 second response time [20:23:01] PROBLEM - os141 Current Load on os141 is WARNING: WARNING - load average: 1.25, 3.35, 3.57 [20:23:05] RECOVERY - puppet141 Current Load on puppet141 is OK: OK - load average: 1.31, 2.63, 3.26 [20:23:22] PROBLEM - mw134 Puppet on mw134 is CRITICAL: CRITICAL: Puppet has 5 failures. Last run 3 minutes ago with 5 failures. Failed resources (up to 3 shown): Package[firejail],File[/etc/firejail/mediawiki.local],File[/etc/firejail/mediawiki-converters.profile],File[/srv/mediawiki/w/skins/Femiwiki/node_modules] [20:23:47] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 1.59, 6.21, 7.76 [20:23:53] PROBLEM - mw134 MediaWiki Rendering on mw134 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 251 bytes in 0.017 second response time [20:24:04] PROBLEM - mw133 MediaWiki Rendering on mw133 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 251 bytes in 0.014 second response time [20:24:53] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query uk.religiononfire.mar.in.ua. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [20:24:59] RECOVERY - os141 Current Load on os141 is OK: OK - load average: 0.41, 2.33, 3.17 [20:26:15] PROBLEM - matomo121 Current Load on matomo121 is WARNING: WARNING - load average: 3.84, 3.08, 2.53 [20:27:38] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 1.33, 3.69, 6.37 [20:31:57] RECOVERY - matomo121 Current Load on matomo121 is OK: OK - load average: 3.24, 3.37, 2.85 [20:33:35] PROBLEM - mw133 MediaWiki Rendering on mw133 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 241 bytes in 0.015 second response time [20:34:26] RECOVERY - ldap141 Puppet on ldap141 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:35:30] RECOVERY - mw133 MediaWiki Rendering on mw133 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 1.211 second response time [20:35:33] RECOVERY - mw134 MediaWiki Rendering on mw134 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 1.244 second response time [20:35:36] RECOVERY - mw133 Puppet on mw133 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [20:35:39] !log [paladox@mwtask141] finished deploy of {'config': True, 'world': True, 'landing': True, 'errorpages': True, 'l10n': True, 'extensionlist': True, 'force': True, 'servers': 'mw133,mw134', 'ignoretime': True} to mw133,mw134 - FAIL: [5888, 0, 0, 0, 0, 0] in 182s [20:36:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:36:48] !log [paladox@mwtask141] finished deploy of {'config': True, 'world': True, 'landing': True, 'errorpages': True, 'l10n': True, 'extensionlist': True, 'force': True, 'servers': 'mw133,mw134', 'ignoretime': True} to mw133,mw134 - FAIL: [5888, 0, 0, 0, 0, 0] in 64s [20:37:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:37:33] RECOVERY - mw134 Puppet on mw134 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [20:40:05] !log [paladox@mwtask141] finished deploy of {'config': True, 'world': True, 'landing': True, 'errorpages': True, 'l10n': True, 'extensionlist': True, 'force': True, 'servers': 'mw133,mw134', 'ignoretime': True} to mw133,mw134 - FAIL: [5888, 0, 0, 0, 0, 0] in 97s [20:40:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:42:24] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:42:30] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [20:42:40] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/e121034b3b42 [20:42:42] [02miraheze/puppet] 07paladox 03e121034 - Add mw133/34 to varnish [20:42:45] [02puppet] 07paladox created branch 03paladox-patch-4 - 13https://github.com/miraheze/puppet [20:42:47] [02puppet] 07paladox opened pull request 03#3230: Add mw133/34 to varnish - 13https://github.com/miraheze/puppet/pull/3230 [20:43:39] !log [paladox@mwtask141] DEPLOY ABORTED: Canary check failed for publictestwiki.com@mw133.miraheze.org [20:44:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:47:09] !log [paladox@mwtask141] DEPLOY ABORTED: Canary check failed for publictestwiki.com@mw133.miraheze.org [20:47:30] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [20:47:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:48:35] !log [paladox@mwtask141] finished deploy of {'config': True, 'world': True, 'landing': True, 'errorpages': True, 'l10n': True, 'extensionlist': True, 'force': True, 'servers': 'mw133,mw134', 'ignoretime': True} to mw133,mw134 - SUCCESS in 32s [20:48:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:48:56] [02puppet] 07paladox closed pull request 03#3230: Add mw133/34 to varnish - 13https://github.com/miraheze/puppet/pull/3230 [20:48:59] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/1d7e1e5ce147...4d30a40f414d [20:49:02] [02miraheze/puppet] 07paladox 034d30a40 - Add mw133/34 to varnish (#3230) [20:49:04] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-4 [20:49:05] [02puppet] 07paladox deleted branch 03paladox-patch-4 - 13https://github.com/miraheze/puppet [20:53:07] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 7.33, 6.59, 6.03 [20:53:56] PROBLEM - cp23 Stunnel for mw133 on cp23 is UNKNOWN: NRPE: Command 'check_Stunnel for mw133' not defined [20:53:58] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - uk.religiononfire.mar.in.ua All nameservers failed to answer the query. [20:54:05] PROBLEM - cp23 APT on cp23 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [20:54:12] PROBLEM - cp32 Stunnel for mw133 on cp32 is UNKNOWN: NRPE: Command 'check_Stunnel for mw133' not defined [20:54:32] PROBLEM - cp32 Stunnel for mw134 on cp32 is UNKNOWN: NRPE: Command 'check_Stunnel for mw134' not defined [20:54:44] PROBLEM - cp23 Stunnel for mw134 on cp23 is UNKNOWN: NRPE: Command 'check_Stunnel for mw134' not defined [20:54:49] !log ran changePassword.php for user Nemo_bis [20:55:04] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 3.85, 5.49, 5.69 [20:55:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:55:40] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 3.76, 4.30, 2.65 [20:55:47] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 8.034 second response time [20:56:08] RECOVERY - cp32 Stunnel for mw133 on cp32 is OK: TCP OK - 0.000 second response time on localhost port 8110 [20:56:11] RECOVERY - cp23 APT on cp23 is OK: APT OK: 35 packages available for upgrade (0 critical updates). [20:56:28] RECOVERY - cp32 Stunnel for mw134 on cp32 is OK: TCP OK - 0.000 second response time on localhost port 8111 [20:56:42] RECOVERY - cp23 Stunnel for mw134 on cp23 is OK: TCP OK - 0.000 second response time on localhost port 8111 [20:57:39] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 1.92, 3.62, 2.60 [20:57:54] RECOVERY - cp23 Stunnel for mw133 on cp23 is OK: TCP OK - 0.000 second response time on localhost port 8110 [20:59:39] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 2.40, 3.16, 2.55 [20:59:54] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/4d30a40f414d...b1a17bf7ad11 [20:59:55] [02miraheze/puppet] 07paladox 03b1a17bf - mwupgradetool: add mw133/34 [20:59:58] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.39, 7.14, 6.55 [21:00:24] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/b1a17bf7ad11...a87f0eb383dd [21:00:26] [02miraheze/puppet] 07paladox 03a87f0eb - depool mw121/122 [21:01:56] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 5.06, 6.51, 6.40 [21:05:52] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.34, 7.03, 6.63 [21:07:03] PROBLEM - mw132 Current Load on mw132 is CRITICAL: CRITICAL - load average: 12.15, 9.98, 8.20 [21:07:50] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 5.28, 6.47, 6.48 [21:09:11] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 0.99, 5.66, 11.23 [21:09:29] PROBLEM - mw133 Current Load on mw133 is WARNING: WARNING - load average: 10.95, 9.57, 5.92 [21:09:58] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 9.55, 6.14, 3.95 [21:10:35] PROBLEM - os141 Current Load on os141 is CRITICAL: CRITICAL - load average: 6.61, 4.27, 2.29 [21:10:57] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 1.41, 4.42, 11.09 [21:11:01] PROBLEM - mw132 Current Load on mw132 is WARNING: WARNING - load average: 11.02, 11.10, 9.12 [21:11:07] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 0.95, 4.13, 10.02 [21:11:23] RECOVERY - mw133 Current Load on mw133 is OK: OK - load average: 9.23, 9.12, 6.15 [21:12:34] PROBLEM - mw131 Current Load on mw131 is CRITICAL: CRITICAL - load average: 12.14, 10.66, 8.55 [21:12:53] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 1.13, 3.36, 9.93 [21:13:43] [02miraheze/MirahezeDebug] 07MacFan4000 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/MirahezeDebug/compare/c93012eb5af8...710a89f58767 [21:13:44] [02miraheze/MirahezeDebug] 07MacFan4000 03710a89f - Add mw133/mw134/mw143 [21:13:47] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 4.27, 5.98, 4.46 [21:14:34] RECOVERY - os141 Current Load on os141 is OK: OK - load average: 1.90, 3.25, 2.37 [21:16:32] PROBLEM - mw131 Current Load on mw131 is WARNING: WARNING - load average: 10.99, 11.14, 9.26 [21:18:04] PROBLEM - mw134 Current Load on mw134 is WARNING: WARNING - load average: 10.73, 10.14, 7.31 [21:18:31] PROBLEM - mw131 Current Load on mw131 is CRITICAL: CRITICAL - load average: 12.08, 11.41, 9.59 [21:18:57] RECOVERY - mw132 Current Load on mw132 is OK: OK - load average: 9.19, 10.19, 9.50 [21:19:06] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/a87f0eb383dd...ba48f6657bb6 [21:19:09] [02miraheze/puppet] 07paladox 03ba48f66 - deploy-mediawiki: add mw143 [21:19:27] PROBLEM - swiftobject121 APT on swiftobject121 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [21:19:59] PROBLEM - mw133 Current Load on mw133 is WARNING: WARNING - load average: 10.74, 10.68, 8.17 [21:21:26] PROBLEM - swiftobject121 APT on swiftobject121 is CRITICAL: APT CRITICAL: 52 packages available for upgrade (7 critical updates). [21:22:29] PROBLEM - mw131 Current Load on mw131 is WARNING: WARNING - load average: 11.12, 11.63, 10.12 [21:22:55] PROBLEM - mw132 Current Load on mw132 is WARNING: WARNING - load average: 9.53, 10.94, 10.01 [21:23:48] RECOVERY - mw133 Current Load on mw133 is OK: OK - load average: 7.61, 9.35, 8.21 [21:23:55] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [21:24:04] RECOVERY - mw134 Current Load on mw134 is OK: OK - load average: 8.43, 9.29, 7.96 [21:25:01] PROBLEM - mw131 Current Load on mw131 is CRITICAL: CRITICAL - load average: 12.03, 11.05, 10.06 [21:25:28] PROBLEM - mw143 JobRunner Service on mw143 is CRITICAL: PROCS CRITICAL: 0 processes with args 'redisJobRunnerService' [21:25:48] PROBLEM - mw143 ferm_active on mw143 is UNKNOWN: NRPE: Unable to read output [21:26:07] PROBLEM - mw143 Puppet on mw143 is UNKNOWN: NRPE: Unable to read output [21:26:17] PROBLEM - mw143 conntrack_table_size on mw143 is UNKNOWN: NRPE: Unable to read output [21:26:22] PROBLEM - mw143 MediaWiki Rendering on mw143 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 322 bytes in 0.008 second response time [21:27:09] RECOVERY - mw132 Current Load on mw132 is OK: OK - load average: 8.34, 9.92, 9.81 [21:27:46] RECOVERY - mw143 ferm_active on mw143 is OK: OK ferm input default policy is set [21:28:10] PROBLEM - mw143 Puppet on mw143 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 38 seconds ago with 2 failures. Failed resources (up to 3 shown): File[/srv/mediawiki/w/skins/Femiwiki/node_modules],Service[cgroup] [21:28:13] RECOVERY - mw143 conntrack_table_size on mw143 is OK: OK: nf_conntrack is 0 % full [21:28:18] PROBLEM - mw143 MediaWiki Rendering on mw143 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 241 bytes in 0.016 second response time [21:29:00] PROBLEM - mw131 Current Load on mw131 is WARNING: WARNING - load average: 11.85, 11.59, 10.50 [21:29:18] RECOVERY - mw143 JobRunner Service on mw143 is OK: PROCS OK: 1 process with args 'redisJobRunnerService' [21:31:03] PROBLEM - mw132 Current Load on mw132 is WARNING: WARNING - load average: 11.22, 10.75, 10.18 [21:32:56] !log [paladox@mwtask141] finished deploy of {'config': True, 'world': True, 'landing': True, 'errorpages': True, 'l10n': True, 'extensionlist': True, 'force': True, 'servers': 'mw143', 'ignoretime': True} to mw143 - FAIL: [5888, 0, 0, 0, 0, 0] in 283s [21:33:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:34:01] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/30322192f0c8 [21:34:04] [02miraheze/puppet] 07paladox 033032219 - varnish: add mw143 [21:34:07] [02puppet] 07paladox created branch 03paladox-patch-4 - 13https://github.com/miraheze/puppet [21:34:09] [02puppet] 07paladox opened pull request 03#3231: varnish: add mw143 - 13https://github.com/miraheze/puppet/pull/3231 [21:34:15] RECOVERY - mw143 MediaWiki Rendering on mw143 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 1.440 second response time [21:34:58] PROBLEM - mw131 Current Load on mw131 is CRITICAL: CRITICAL - load average: 12.36, 11.69, 10.88 [21:36:10] RECOVERY - mw143 Puppet on mw143 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [21:36:55] PROBLEM - os141 Current Load on os141 is CRITICAL: CRITICAL - load average: 4.39, 3.38, 2.55 [21:36:57] PROBLEM - mw132 Current Load on mw132 is CRITICAL: CRITICAL - load average: 13.36, 11.63, 10.72 [21:36:58] PROBLEM - mw131 Current Load on mw131 is WARNING: WARNING - load average: 11.20, 11.29, 10.82 [21:37:23] !log [paladox@mwtask141] finished deploy of {'config': True, 'world': True, 'landing': True, 'errorpages': True, 'l10n': True, 'extensionlist': True, 'force': True, 'servers': 'mw143', 'ignoretime': True} to mw143 - SUCCESS in 255s [21:37:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:37:37] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-4 [21:37:39] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/ba48f6657bb6...3756bf36e48e [21:37:41] [02miraheze/puppet] 07paladox 033756bf3 - varnish: add mw143 (#3231) [21:37:42] [02puppet] 07paladox closed pull request 03#3231: varnish: add mw143 - 13https://github.com/miraheze/puppet/pull/3231 [21:37:45] [02puppet] 07paladox deleted branch 03paladox-patch-4 - 13https://github.com/miraheze/puppet [21:37:50] PROBLEM - mw134 Current Load on mw134 is CRITICAL: CRITICAL - load average: 13.08, 10.65, 9.21 [21:38:12] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 8.51, 7.71, 5.98 [21:38:55] PROBLEM - mw132 Current Load on mw132 is WARNING: WARNING - load average: 11.89, 11.89, 10.95 [21:38:55] RECOVERY - os141 Current Load on os141 is OK: OK - load average: 3.14, 3.33, 2.64 [21:39:48] PROBLEM - mw134 Current Load on mw134 is WARNING: WARNING - load average: 10.76, 10.41, 9.28 [21:40:09] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 7.38, 7.20, 5.98 [21:40:43] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 6.96, 6.66, 6.47 [21:42:56] PROBLEM - mw131 Current Load on mw131 is CRITICAL: CRITICAL - load average: 12.55, 11.43, 10.98 [21:44:01] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 4.52, 5.87, 5.76 [21:44:40] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 6.37, 6.58, 6.49 [21:44:56] PROBLEM - mw131 Current Load on mw131 is WARNING: WARNING - load average: 8.40, 10.50, 10.71 [21:45:42] RECOVERY - mw134 Current Load on mw134 is OK: OK - load average: 8.77, 9.67, 9.35 [21:46:01] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/3756bf36e48e...b3104d3baa0e [21:46:03] [02miraheze/puppet] 07paladox 03b3104d3 - mwupgradetool: add mw143 [21:47:56] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 8.59, 6.73, 6.09 [21:48:54] RECOVERY - mw131 Current Load on mw131 is OK: OK - load average: 6.95, 8.70, 9.92 [21:49:52] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 3.70, 5.81, 5.85 [21:55:52] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - uk.religiononfire.mar.in.ua All nameservers failed to answer the query. [21:59:02] RECOVERY - mw132 Current Load on mw132 is OK: OK - load average: 8.54, 9.32, 9.97 [22:07:39] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.47, 6.98, 6.61 [22:08:31] PROBLEM - mw132 Current Load on mw132 is WARNING: WARNING - load average: 9.09, 10.24, 10.07 [22:09:36] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 5.68, 6.46, 6.46 [22:10:24] RECOVERY - mw132 Current Load on mw132 is OK: OK - load average: 8.36, 9.48, 9.80 [22:29:05] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/17631a04c52e [22:29:08] [02miraheze/puppet] 07paladox 0317631a0 - Fixup robots.php [22:29:09] [02puppet] 07paladox created branch 03paladox-patch-4 - 13https://github.com/miraheze/puppet [22:29:10] [02puppet] 07paladox opened pull request 03#3232: Fixup robots.php - 13https://github.com/miraheze/puppet/pull/3232 [22:31:09] [02puppet] 07The-Voidwalker reviewed pull request 03#3232 commit - 13https://github.com/miraheze/puppet/pull/3232#discussion_r1234529210 [22:31:37] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/17631a04c52e...47bb8e8a85db [22:31:40] [02miraheze/puppet] 07paladox 0347bb8e8 - Update modules/mediawiki/files/robots.php [22:31:42] [02puppet] 07paladox synchronize pull request 03#3232: Fixup robots.php - 13https://github.com/miraheze/puppet/pull/3232 [22:32:02] [02puppet] 07paladox closed pull request 03#3232: Fixup robots.php - 13https://github.com/miraheze/puppet/pull/3232 [22:32:03] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/b3104d3baa0e...e90a5c8da33a [22:32:05] [02miraheze/puppet] 07paladox 03e90a5c8 - Fixup robots.php (#3232) [22:32:08] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-4 [22:32:10] [02puppet] 07paladox deleted branch 03paladox-patch-4 - 13https://github.com/miraheze/puppet [22:32:11] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.49, 7.69, 7.98 [22:37:53] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.41, 7.59, 7.76 [22:41:18] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 26.92, 12.19, 5.70 [22:44:04] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.02, 6.59, 6.23 [22:45:01] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/767e2501cefc [22:45:04] [02miraheze/puppet] 07paladox 03767e250 - mw121/mw122: disable job runner [22:45:07] [02puppet] 07paladox created branch 03paladox-patch-4 - 13https://github.com/miraheze/puppet [22:45:10] [02puppet] 07paladox opened pull request 03#3233: mw121/mw122: disable job runner - 13https://github.com/miraheze/puppet/pull/3233 [22:45:21] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/767e2501cefc...57fd9dbd67db [22:45:22] [02miraheze/puppet] 07paladox 0357fd9db - Update mw122.yaml [22:45:23] [02puppet] 07paladox synchronize pull request 03#3233: mw121/mw122: disable job runner - 13https://github.com/miraheze/puppet/pull/3233 [22:45:32] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/puppet/compare/e90a5c8da33a...e4b3af73f90d [22:45:35] [02miraheze/puppet] 07paladox 03e4b3af7 - mw121/mw122: disable job runner (#3233) [22:45:36] [02puppet] 07paladox closed pull request 03#3233: mw121/mw122: disable job runner - 13https://github.com/miraheze/puppet/pull/3233 [22:45:37] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-4 [22:45:39] [02puppet] 07paladox deleted branch 03paladox-patch-4 - 13https://github.com/miraheze/puppet [22:45:59] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 6.56, 6.59, 6.27 [22:47:21] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/453708327e89 [22:47:23] [02miraheze/puppet] 07paladox 034537083 - Remove mw121/122 from varnish [22:47:26] [02puppet] 07paladox created branch 03paladox-patch-4 - 13https://github.com/miraheze/puppet [22:47:29] [02puppet] 07paladox opened pull request 03#3234: Remove mw121/122 from varnish - 13https://github.com/miraheze/puppet/pull/3234 [22:47:31] [02puppet] 07paladox closed pull request 03#3234: Remove mw121/122 from varnish - 13https://github.com/miraheze/puppet/pull/3234 [22:47:34] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/e4b3af73f90d...820ff68367c5 [22:47:37] [02miraheze/puppet] 07paladox 03820ff68 - Remove mw121/122 from varnish (#3234) [22:47:38] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-4 [22:47:41] [02puppet] 07paladox deleted branch 03paladox-patch-4 - 13https://github.com/miraheze/puppet [22:48:09] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/820ff68367c5...e03369bb9ab5 [22:48:11] [02miraheze/puppet] 07paladox 03e03369b - Remove mw121/22 from deploy-mediawiki [22:48:12] PROBLEM - mw121 JobRunner Service on mw121 is CRITICAL: PROCS CRITICAL: 0 processes with args 'redisJobRunnerService' [22:48:13] PROBLEM - mw122 JobRunner Service on mw122 is CRITICAL: PROCS CRITICAL: 0 processes with args 'redisJobRunnerService' [22:49:10] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 0.69, 3.30, 3.85 [22:53:15] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 0.90, 2.10, 3.22 [22:54:32] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query uk.religiononfire.mar.in.ua. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [22:54:39] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/df2e4f196fcb [22:54:41] [02miraheze/puppet] 07paladox 03df2e4f1 - jobrunner: run all jobs on mwtask141 [22:54:42] [02puppet] 07paladox created branch 03paladox-patch-4 - 13https://github.com/miraheze/puppet [22:54:45] [02puppet] 07paladox opened pull request 03#3235: jobrunner: run all jobs on mwtask141 - 13https://github.com/miraheze/puppet/pull/3235 [22:57:49] PROBLEM - cp33 Stunnel for mw121 on cp33 is CRITICAL: connect to address localhost and port 8104: Connection refused [22:59:25] PROBLEM - cp33 Stunnel for mw122 on cp33 is CRITICAL: connect to address localhost and port 8105: Connection refused [23:00:42] PROBLEM - cp23 Stunnel for mw121 on cp23 is CRITICAL: connect to address localhost and port 8104: Connection refused [23:00:58] PROBLEM - os141 Current Load on os141 is WARNING: WARNING - load average: 3.91, 3.36, 2.57 [23:01:42] PROBLEM - cp23 Stunnel for mw122 on cp23 is CRITICAL: connect to address localhost and port 8105: Connection refused [23:02:13] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.28, 6.83, 6.47 [23:04:13] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 8.09, 7.16, 6.63 [23:04:53] PROBLEM - os141 Current Load on os141 is CRITICAL: CRITICAL - load average: 6.00, 4.19, 3.05 [23:05:26] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 10.23, 8.15, 6.50 [23:06:13] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.49, 7.22, 6.71 [23:07:25] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 6.62, 7.22, 6.34 [23:08:43] PROBLEM - os141 Current Load on os141 is WARNING: WARNING - load average: 3.04, 3.74, 3.13 [23:09:23] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 10.28, 8.13, 6.76 [23:10:52] PROBLEM - cp32 Stunnel for mw121 on cp32 is CRITICAL: connect to address localhost and port 8104: Connection refused [23:11:22] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 5.03, 6.97, 6.51 [23:11:25] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-4 [23:11:28] [02puppet] 07paladox closed pull request 03#3235: jobrunner: run all jobs on mwtask141 - 13https://github.com/miraheze/puppet/pull/3235 [23:11:30] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/e03369bb9ab5...464c3b689f7a [23:11:33] [02miraheze/puppet] 07paladox 03464c3b6 - jobrunner: run all jobs on mwtask141 (#3235) [23:11:34] [02puppet] 07paladox deleted branch 03paladox-patch-4 - 13https://github.com/miraheze/puppet [23:11:56] PROBLEM - cp32 Stunnel for mw122 on cp32 is CRITICAL: connect to address localhost and port 8105: Connection refused [23:12:02] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/f5fa75afe482 [23:12:03] [02miraheze/puppet] 07paladox 03f5fa75a - mw133/134/143: disable jobrunner [23:12:04] [02puppet] 07paladox created branch 03paladox-patch-4 - 13https://github.com/miraheze/puppet [23:12:07] [02puppet] 07paladox opened pull request 03#3236: mw133/134/143: disable jobrunner - 13https://github.com/miraheze/puppet/pull/3236 [23:12:22] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/f5fa75afe482...77eecf489ec3 [23:12:25] [02miraheze/puppet] 07paladox 0377eecf4 - Update mw134.yaml [23:12:29] [02puppet] 07paladox synchronize pull request 03#3236: mw133/134/143: disable jobrunner - 13https://github.com/miraheze/puppet/pull/3236 [23:12:32] RECOVERY - os141 Current Load on os141 is OK: OK - load average: 1.50, 2.88, 2.94 [23:12:32] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/77eecf489ec3...1927fdf38a81 [23:12:33] [02miraheze/puppet] 07paladox 031927fdf - Update mw143.yaml [23:12:34] [02puppet] 07paladox synchronize pull request 03#3236: mw133/134/143: disable jobrunner - 13https://github.com/miraheze/puppet/pull/3236 [23:12:39] [02puppet] 07paladox closed pull request 03#3236: mw133/134/143: disable jobrunner - 13https://github.com/miraheze/puppet/pull/3236 [23:12:42] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-4 [23:12:45] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±3] 13https://github.com/miraheze/puppet/compare/464c3b689f7a...ed401faa74d7 [23:12:48] [02miraheze/puppet] 07paladox 03ed401fa - mw133/134/143: disable jobrunner (#3236) [23:12:49] [02puppet] 07paladox deleted branch 03paladox-patch-4 - 13https://github.com/miraheze/puppet [23:13:21] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 10.08, 7.55, 6.73 [23:14:14] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 5.80, 6.57, 6.65 [23:15:40] PROBLEM - mw133 JobRunner Service on mw133 is CRITICAL: PROCS CRITICAL: 0 processes with args 'redisJobRunnerService' [23:16:30] PROBLEM - mw134 JobRunner Service on mw134 is CRITICAL: PROCS CRITICAL: 0 processes with args 'redisJobRunnerService' [23:17:19] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 6.66, 7.72, 7.02 [23:18:13] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.19, 7.30, 6.96 [23:18:18] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [23:18:19] PROBLEM - mw143 JobRunner Service on mw143 is CRITICAL: PROCS CRITICAL: 0 processes with args 'redisJobRunnerService' [23:18:38] PROBLEM - cp22 Stunnel for mw121 on cp22 is CRITICAL: connect to address localhost and port 8104: Connection refused [23:19:18] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 10.42, 9.10, 7.63 [23:19:25] PROBLEM - cp22 Stunnel for mw122 on cp22 is CRITICAL: connect to address localhost and port 8105: Connection refused [23:23:20] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - uk.religiononfire.mar.in.ua All nameservers failed to answer the query. [23:24:16] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/cebd2c7b5b0d [23:24:17] [02miraheze/puppet] 07paladox 03cebd2c7 - jobrunner-hi: remove use of --maxjobs arg [23:24:19] [02puppet] 07paladox created branch 03paladox-patch-4 - 13https://github.com/miraheze/puppet [23:24:20] [02puppet] 07paladox opened pull request 03#3237: jobrunner-hi: remove use of --maxjobs arg - 13https://github.com/miraheze/puppet/pull/3237 [23:24:23] [02puppet] 07paladox closed pull request 03#3237: jobrunner-hi: remove use of --maxjobs arg - 13https://github.com/miraheze/puppet/pull/3237 [23:24:26] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/ed401faa74d7...e059480ee8f6 [23:24:29] [02miraheze/puppet] 07paladox 03e059480 - jobrunner-hi: remove use of --maxjobs arg (#3237) [23:24:31] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-4 [23:24:33] [02puppet] 07paladox deleted branch 03paladox-patch-4 - 13https://github.com/miraheze/puppet [23:26:12] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 9.10, 7.80, 7.24 [23:27:06] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/e059480ee8f6...da238344ce8a [23:27:09] [02miraheze/puppet] 07paladox 03da23834 - Update jobrunner-hi.json.erb [23:27:13] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 4.98, 7.94, 7.91 [23:27:20] PROBLEM - os141 Current Load on os141 is WARNING: WARNING - load average: 3.54, 3.66, 3.30 [23:29:08] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 9.27, 7.56, 7.72 [23:30:10] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.18, 7.47, 7.23 [23:30:58] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.77, 3.37, 2.77 [23:31:17] PROBLEM - os141 Current Load on os141 is CRITICAL: CRITICAL - load average: 4.23, 3.78, 3.42 [23:32:57] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.89, 3.18, 2.78 [23:32:58] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 5.92, 8.00, 7.94 [23:33:15] RECOVERY - os141 Current Load on os141 is OK: OK - load average: 1.85, 3.13, 3.23 [23:34:55] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 9.20, 8.34, 8.06 [23:37:28] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 7.10, 6.82, 6.29 [23:38:06] PROBLEM - swiftobject111 Current Load on swiftobject111 is CRITICAL: CRITICAL - load average: 9.18, 8.16, 7.58 [23:39:55] PROBLEM - cp32 APT on cp32 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [23:40:05] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 7.66, 7.95, 7.58 [23:41:20] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 6.79, 6.75, 6.39 [23:41:52] RECOVERY - cp32 APT on cp32 is OK: APT OK: 35 packages available for upgrade (0 critical updates). [23:42:16] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 8.33, 9.30, 5.04 [23:42:55] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 3.71, 6.70, 7.63 [23:46:30] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 7.92, 6.98, 6.25 [23:46:55] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 0.77, 3.65, 6.18 [23:48:02] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns.ankh.fr.eu.org.', 'ns1.eu.org.', 'ns1.eriomem.net.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [23:48:26] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 5.83, 6.57, 6.19 [23:50:15] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.57, 3.27, 3.73 [23:52:02] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 6.92, 7.00, 6.67 [23:53:07] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [23:53:58] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 5.75, 6.53, 6.54 [23:54:15] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.45, 2.36, 3.25