[00:01:46] RECOVERY - db182 Puppet on db182 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:03:02] PROBLEM - swiftproxy131 Current Load on swiftproxy131 is CRITICAL: CRITICAL - load average: 10.38, 7.22, 6.00 [00:03:05] RECOVERY - mon181 Puppet on mon181 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:03:17] PROBLEM - db112 Current Load on db112 is CRITICAL: CRITICAL - load average: 16.20, 7.21, 3.24 [00:04:48] PROBLEM - mw133 HTTPS on mw133 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [00:05:00] PROBLEM - swiftproxy131 Current Load on swiftproxy131 is WARNING: WARNING - load average: 7.26, 7.27, 6.18 [00:05:56] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 4.76, 4.04, 3.57 [00:06:45] PROBLEM - mw133 HTTPS on mw133 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 1.048 second response time [00:06:59] RECOVERY - swiftproxy131 Current Load on swiftproxy131 is OK: OK - load average: 5.49, 6.78, 6.14 [00:07:42] RECOVERY - Host cloud11 is UP: PING OK - Packet loss = 0%, RTA = 128.89 ms [00:07:43] PROBLEM - os131 Current Load on os131 is CRITICAL: LOAD CRITICAL - total load average: 7.15, 4.40, 2.46 [00:08:22] PROBLEM - cp35 HTTPS on cp35 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [00:08:33] PROBLEM - cp34 HTTPS on cp34 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [00:08:50] PROBLEM - mw131 HTTPS on mw131 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10000 milliseconds with 0 bytes received [00:08:50] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 11.75, 9.55, 8.11 [00:09:06] RECOVERY - ping6 on cloud11 is OK: PING OK - Packet loss = 0%, RTA = 129.03 ms [00:09:36] RECOVERY - cloud11 SSH on cloud11 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u3 (protocol 2.0) [00:10:17] PROBLEM - cp35 HTTPS on cp35 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.761 second response time [00:10:33] PROBLEM - cp34 HTTPS on cp34 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.590 second response time [00:10:48] PROBLEM - mw131 HTTPS on mw131 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.909 second response time [00:10:50] RECOVERY - mw133 Current Load on mw133 is OK: LOAD OK - total load average: 6.94, 8.51, 7.91 [00:10:56] PROBLEM - swiftproxy131 Current Load on swiftproxy131 is WARNING: WARNING - load average: 6.34, 6.93, 6.39 [00:14:54] PROBLEM - swiftproxy131 Current Load on swiftproxy131 is CRITICAL: CRITICAL - load average: 8.18, 7.38, 6.66 [00:15:16] RECOVERY - db112 Current Load on db112 is OK: OK - load average: 2.13, 6.75, 6.59 [00:16:53] RECOVERY - swiftproxy131 Current Load on swiftproxy131 is OK: OK - load average: 3.92, 6.22, 6.33 [00:17:36] PROBLEM - os131 Current Load on os131 is WARNING: LOAD WARNING - total load average: 3.40, 3.86, 3.26 [00:18:25] RECOVERY - cloud11 Current Load on cloud11 is OK: OK - load average: 0.85, 0.49, 0.22 [00:18:53] PROBLEM - cloud11 SMART on cloud11 is WARNING: WARNING: [cciss,2] - Reallocated_Sector_Ct is non-zero (32) --- [cciss,3] - Reallocated_Sector_Ct is non-zero (42) --- [cciss,0] - Device is clean --- [cciss,1] - Device is clean --- [cciss,4] - Device is clean --- [cciss,5] - Device is clean --- [cciss,6] - Device is clean| [00:18:53] PROBLEM - cloud11 conntrack_table_size on cloud11 is UNKNOWN: NRPE: Unable to read output [00:18:56] RECOVERY - cloud11 IPMI Sensors on cloud11 is OK: IPMI Status: OK [00:18:57] RECOVERY - cloud11 Disk Space on cloud11 is OK: DISK OK - free space: / 23984 MB (84% inode=95%); [00:19:06] RECOVERY - cloud11 NTP time on cloud11 is OK: NTP OK: Offset 0.0003690719604 secs [00:19:15] RECOVERY - cloud11 PowerDNS Recursor on cloud11 is OK: DNS OK: 0.236 seconds response time. miraheze.org returns 2001:41d0:801:2000::3a18,2001:41d0:801:2000::5d68,51.195.201.140,51.89.139.24 [00:19:16] RECOVERY - cloud11 APT on cloud11 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [00:19:32] RECOVERY - os131 Current Load on os131 is OK: LOAD OK - total load average: 2.57, 3.37, 3.15 [00:20:48] RECOVERY - cloud11 conntrack_table_size on cloud11 is OK: OK: nf_conntrack is 0 % full [00:20:48] RECOVERY - cloud11 ferm_active on cloud11 is OK: OK ferm input default policy is set [00:20:49] RECOVERY - cloud11 Puppet on cloud11 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [00:21:55] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 2.59, 3.57, 3.86 [00:23:36] PROBLEM - os131 Current Load on os131 is WARNING: LOAD WARNING - total load average: 3.37, 3.44, 3.23 [00:25:32] RECOVERY - os131 Current Load on os131 is OK: LOAD OK - total load average: 3.29, 3.36, 3.22 [00:29:49] PROBLEM - cloud11 SMART on cloud11 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [00:31:28] PROBLEM - ping6 on cloud11 is CRITICAL: PING CRITICAL - Packet loss = 100% [00:31:32] PROBLEM - os131 Current Load on os131 is WARNING: LOAD WARNING - total load average: 3.22, 3.44, 3.30 [00:31:55] PROBLEM - cloud11 SSH on cloud11 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:32:21] PROBLEM - cloud11 SMART on cloud11 is WARNING: WARNING: [cciss,2] - Reallocated_Sector_Ct is non-zero (32) --- [cciss,3] - Reallocated_Sector_Ct is non-zero (42) --- [cciss,0] - Device is clean --- [cciss,1] - Device is clean --- [cciss,4] - Device is clean --- [cciss,5] - Device is clean --- [cciss,6] - Device is clean| [00:33:27] RECOVERY - ping6 on cloud11 is OK: PING OK - Packet loss = 0%, RTA = 128.87 ms [00:33:32] RECOVERY - os131 Current Load on os131 is OK: LOAD OK - total load average: 3.20, 3.30, 3.26 [00:33:48] PROBLEM - db182 Puppet on db182 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/apt/keyrings/mariadb_release_signing_key.pgp] [00:33:55] RECOVERY - cloud11 SSH on cloud11 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u3 (protocol 2.0) [00:33:56] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 4.02, 3.52, 3.61 [00:34:19] RECOVERY - Host swiftobject113 is UP: PING OK - Packet loss = 0%, RTA = 128.93 ms [00:34:38] RECOVERY - Host swiftobject112 is UP: PING OK - Packet loss = 0%, RTA = 129.04 ms [00:34:39] RECOVERY - swiftobject113 Swift Object Service on swiftobject113 is OK: TCP OK - 0.129 second response time on 2a10:6740::6:205 port 6000 [00:34:41] PROBLEM - swiftobject113 Puppet on swiftobject113 is WARNING: WARNING: Puppet last ran 4 hours ago [00:34:48] RECOVERY - Host swiftobject111 is UP: PING OK - Packet loss = 0%, RTA = 129.12 ms [00:34:48] RECOVERY - swiftobject112 Swift Object Service on swiftobject112 is OK: TCP OK - 0.129 second response time on 2a10:6740::6:204 port 6000 [00:35:25] RECOVERY - swiftobject112 SSH on swiftobject112 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u3 (protocol 2.0) [00:35:51] RECOVERY - swiftobject111 SSH on swiftobject111 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u3 (protocol 2.0) [00:35:56] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 2.68, 3.24, 3.51 [00:36:01] RECOVERY - ping6 on swiftobject112 is OK: PING OK - Packet loss = 0%, RTA = 129.01 ms [00:36:16] RECOVERY - ping6 on swiftobject111 is OK: PING OK - Packet loss = 0%, RTA = 129.11 ms [00:36:17] RECOVERY - ping6 on swiftobject113 is OK: PING OK - Packet loss = 0%, RTA = 128.95 ms [00:36:39] RECOVERY - swiftobject113 Puppet on swiftobject113 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:36:41] RECOVERY - swiftobject111 Swift Object Service on swiftobject111 is OK: TCP OK - 0.129 second response time on 2a10:6740::6:203 port 6000 [00:41:34] RECOVERY - Host swiftac111 is UP: PING OK - Packet loss = 0%, RTA = 128.87 ms [00:41:46] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.75, 3.93, 3.24 [00:41:46] PROBLEM - swiftac111 conntrack_table_size on swiftac111 is CRITICAL: connect to address 2a10:6740::6:202 port 5666: Connection refusedconnect to host 2a10:6740::6:202 port 5666: Connection refused [00:41:57] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 4.65, 4.01, 3.74 [00:42:01] PROBLEM - swiftac111 Disk Space on swiftac111 is CRITICAL: connect to address 2a10:6740::6:202 port 5666: Connection refusedconnect to host 2a10:6740::6:202 port 5666: Connection refused [00:42:16] PROBLEM - swiftac111 ferm_active on swiftac111 is CRITICAL: connect to address 2a10:6740::6:202 port 5666: Connection refusedconnect to host 2a10:6740::6:202 port 5666: Connection refused [00:42:26] PROBLEM - swiftac111 Backups Swift Account Container on swiftac111 is CRITICAL: connect to address 2a10:6740::6:202 port 5666: Connection refusedconnect to host 2a10:6740::6:202 port 5666: Connection refused [00:42:31] PROBLEM - swiftac111 PowerDNS Recursor on swiftac111 is CRITICAL: connect to address 2a10:6740::6:202 port 5666: Connection refusedconnect to host 2a10:6740::6:202 port 5666: Connection refused [00:42:51] PROBLEM - swiftac111 NTP time on swiftac111 is CRITICAL: connect to address 2a10:6740::6:202 port 5666: Connection refusedconnect to host 2a10:6740::6:202 port 5666: Connection refused [00:43:01] PROBLEM - swiftac111 Current Load on swiftac111 is CRITICAL: connect to address 2a10:6740::6:202 port 5666: Connection refusedconnect to host 2a10:6740::6:202 port 5666: Connection refused [00:43:31] RECOVERY - ping6 on swiftac111 is OK: PING OK - Packet loss = 0%, RTA = 128.98 ms [00:43:36] PROBLEM - os131 Current Load on os131 is CRITICAL: LOAD CRITICAL - total load average: 6.56, 4.50, 3.67 [00:44:15] RECOVERY - Host swiftproxy111 is UP: PING OK - Packet loss = 0%, RTA = 128.93 ms [00:44:16] RECOVERY - ping6 on swiftproxy111 is OK: PING OK - Packet loss = 0%, RTA = 128.80 ms [00:44:46] PROBLEM - swiftproxy111 Disk Space on swiftproxy111 is CRITICAL: connect to address 2a10:6740::6:201 port 5666: Connection refusedconnect to host 2a10:6740::6:201 port 5666: Connection refused [00:44:47] PROBLEM - swiftproxy131 Current Load on swiftproxy131 is WARNING: WARNING - load average: 6.73, 6.84, 6.39 [00:44:51] PROBLEM - swiftproxy111 PowerDNS Recursor on swiftproxy111 is CRITICAL: connect to address 2a10:6740::6:201 port 5666: Connection refusedconnect to host 2a10:6740::6:201 port 5666: Connection refused [00:45:06] PROBLEM - swiftproxy111 Current Load on swiftproxy111 is CRITICAL: connect to address 2a10:6740::6:201 port 5666: Connection refusedconnect to host 2a10:6740::6:201 port 5666: Connection refused [00:45:46] PROBLEM - swiftproxy111 NTP time on swiftproxy111 is CRITICAL: connect to address 2a10:6740::6:201 port 5666: Connection refusedconnect to host 2a10:6740::6:201 port 5666: Connection refused [00:45:47] PROBLEM - swiftproxy111 conntrack_table_size on swiftproxy111 is CRITICAL: connect to address 2a10:6740::6:201 port 5666: Connection refusedconnect to host 2a10:6740::6:201 port 5666: Connection refused [00:45:51] PROBLEM - swiftproxy111 ferm_active on swiftproxy111 is CRITICAL: connect to address 2a10:6740::6:201 port 5666: Connection refusedconnect to host 2a10:6740::6:201 port 5666: Connection refused [00:46:47] RECOVERY - swiftproxy131 Current Load on swiftproxy131 is OK: OK - load average: 5.83, 6.70, 6.41 [00:47:32] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 5.53, 4.42, 3.61 [00:49:27] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 2.57, 3.68, 3.43 [00:53:20] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.54, 3.94, 3.57 [00:54:10] RECOVERY - swiftac111 SSH on swiftac111 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u3 (protocol 2.0) [00:56:47] PROBLEM - swiftproxy131 Current Load on swiftproxy131 is WARNING: WARNING - load average: 7.02, 7.24, 6.75 [00:57:11] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.37, 3.74, 3.59 [00:59:07] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 6.02, 4.52, 3.89 [01:01:14] RECOVERY - swiftproxy111 SSH on swiftproxy111 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u3 (protocol 2.0) [01:02:47] RECOVERY - swiftproxy131 Current Load on swiftproxy131 is OK: OK - load average: 6.60, 6.62, 6.63 [01:03:32] PROBLEM - os131 Current Load on os131 is WARNING: LOAD WARNING - total load average: 2.27, 3.18, 3.90 [01:06:12] RECOVERY - swiftac111 Swift Container Service on swiftac111 is OK: TCP OK - 0.129 second response time on 2a10:6740::6:202 port 6001 [01:06:27] RECOVERY - swiftac111 Disk Space on swiftac111 is OK: DISK OK - free space: / 120638 MB (97% inode=99%); [01:06:33] PROBLEM - swiftac111 Puppet on swiftac111 is UNKNOWN: UNKNOWN: Failed to check. Reason is: no_summary_file [01:06:38] PROBLEM - swiftac111 conntrack_table_size on swiftac111 is UNKNOWN: NRPE: Unable to read output [01:06:44] RECOVERY - swiftac111 Swift Account Service on swiftac111 is OK: TCP OK - 0.129 second response time on 2a10:6740::6:202 port 6002 [01:07:18] RECOVERY - swiftac111 APT on swiftac111 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [01:07:52] PROBLEM - swiftac111 ferm_active on swiftac111 is UNKNOWN: NRPE: Unable to read output [01:08:12] RECOVERY - swiftac111 PowerDNS Recursor on swiftac111 is OK: DNS OK: 0.079 seconds response time. miraheze.org returns 2001:41d0:801:2000::3a18,2001:41d0:801:2000::5d68,51.195.201.140,51.89.139.24 [01:08:20] RECOVERY - swiftac111 NTP time on swiftac111 is OK: NTP OK: Offset -0.003402531147 secs [01:08:24] RECOVERY - swiftac111 Current Load on swiftac111 is OK: OK - load average: 0.46, 0.89, 0.53 [01:09:51] RECOVERY - swiftac111 ferm_active on swiftac111 is OK: OK ferm input default policy is set [01:10:00] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 2.89, 3.43, 4.00 [01:10:35] PROBLEM - swiftac111 Puppet on swiftac111 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 1 minute ago with 0 failures [01:10:38] PROBLEM - swiftac111 Swift Account Service on swiftac111 is CRITICAL: connect to address 2a10:6740::6:202 and port 6002: Connection refused [01:10:38] RECOVERY - swiftac111 conntrack_table_size on swiftac111 is OK: OK: nf_conntrack is 0 % full [01:11:42] RECOVERY - swiftproxy111 APT on swiftproxy111 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [01:11:47] PROBLEM - swiftproxy111 Puppet on swiftproxy111 is UNKNOWN: NRPE: Unable to read output [01:12:00] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 4.38, 3.58, 3.97 [01:12:01] RECOVERY - swiftproxy111 Swift Proxy on swiftproxy111 is OK: TCP OK - 0.129 second response time on 2a10:6740::6:201 port 80 [01:12:05] PROBLEM - swiftac111 Swift Container Service on swiftac111 is CRITICAL: connect to address 2a10:6740::6:202 and port 6001: Connection refused [01:12:20] RECOVERY - swiftproxy111 Disk Space on swiftproxy111 is OK: DISK OK - free space: / 25814 MB (90% inode=95%); [01:12:29] RECOVERY - swiftproxy111 NTP time on swiftproxy111 is OK: NTP OK: Offset 0.001952558756 secs [01:12:38] PROBLEM - swiftproxy111 conntrack_table_size on swiftproxy111 is UNKNOWN: NRPE: Unable to read output [01:12:42] PROBLEM - swiftproxy111 ferm_active on swiftproxy111 is UNKNOWN: NRPE: Unable to read output [01:12:59] RECOVERY - swiftproxy111 Current Load on swiftproxy111 is OK: OK - load average: 1.99, 1.52, 0.73 [01:13:08] RECOVERY - swiftproxy111 PowerDNS Recursor on swiftproxy111 is OK: DNS OK: 0.087 seconds response time. miraheze.org returns 2001:41d0:801:2000::3a18,2001:41d0:801:2000::5d68,51.195.201.140,51.89.139.24 [01:13:33] PROBLEM - os131 Current Load on os131 is CRITICAL: LOAD CRITICAL - total load average: 5.03, 3.59, 3.71 [01:13:46] RECOVERY - swiftproxy111 Puppet on swiftproxy111 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [01:14:33] RECOVERY - swiftproxy111 conntrack_table_size on swiftproxy111 is OK: OK: nf_conntrack is 8 % full [01:14:39] RECOVERY - swiftproxy111 ferm_active on swiftproxy111 is OK: OK ferm input default policy is set [01:15:33] PROBLEM - os131 Current Load on os131 is WARNING: LOAD WARNING - total load average: 3.82, 3.64, 3.71 [01:16:55] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.30, 3.68, 3.98 [01:19:32] RECOVERY - os131 Current Load on os131 is OK: LOAD OK - total load average: 0.93, 2.42, 3.23 [01:23:36] PROBLEM - swiftproxy131 Current Load on swiftproxy131 is WARNING: WARNING - load average: 7.17, 7.23, 6.86 [01:28:57] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 3.03, 2.83, 3.34 [01:31:50] RECOVERY - db182 Puppet on db182 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [01:31:53] PROBLEM - db181 Puppet on db181 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/apt/keyrings/mariadb_release_signing_key.pgp] [01:32:56] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 5.60, 3.93, 3.64 [01:33:30] PROBLEM - swiftproxy131 Current Load on swiftproxy131 is CRITICAL: CRITICAL - load average: 8.25, 7.37, 7.12 [01:34:02] PROBLEM - mw131 Current Load on mw131 is CRITICAL: LOAD CRITICAL - total load average: 12.06, 9.37, 7.24 [01:34:19] PROBLEM - mw133 Current Load on mw133 is CRITICAL: LOAD CRITICAL - total load average: 15.87, 13.26, 9.42 [01:34:22] PROBLEM - mw134 Current Load on mw134 is CRITICAL: LOAD CRITICAL - total load average: 13.58, 11.33, 8.35 [01:34:56] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.88, 3.96, 3.70 [01:35:10] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.andreijiroh.uk.eu.org All nameservers failed to answer the query. [01:35:29] PROBLEM - swiftproxy131 Current Load on swiftproxy131 is WARNING: WARNING - load average: 5.04, 6.55, 6.86 [01:35:44] RECOVERY - swiftproxy181 HTTP on swiftproxy181 is OK: HTTP OK: Status line output matched "HTTP/1.1 404" - 352 bytes in 2.128 second response time [01:35:50] RECOVERY - swiftproxy131 HTTP on swiftproxy131 is OK: HTTP OK: Status line output matched "HTTP/1.1 404" - 352 bytes in 0.315 second response time [01:36:02] RECOVERY - mw131 Current Load on mw131 is OK: LOAD OK - total load average: 5.80, 7.90, 6.97 [01:36:04] RECOVERY - swiftac111 Swift Container Service on swiftac111 is OK: TCP OK - 0.129 second response time on 2a10:6740::6:202 port 6001 [01:36:06] RECOVERY - swiftproxy111 HTTPS on swiftproxy111 is OK: HTTP OK: Status line output matched "HTTP/1.1 404" - 352 bytes in 0.593 second response time [01:36:15] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 7.05, 10.70, 8.95 [01:36:21] RECOVERY - mw134 Current Load on mw134 is OK: LOAD OK - total load average: 6.90, 9.72, 8.14 [01:36:33] RECOVERY - swiftproxy111 HTTP on swiftproxy111 is OK: HTTP OK: Status line output matched "HTTP/1.1 404" - 352 bytes in 0.312 second response time [01:36:38] RECOVERY - swiftac111 Swift Account Service on swiftac111 is OK: TCP OK - 0.129 second response time on 2a10:6740::6:202 port 6002 [01:36:55] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 3.98, 4.18, 3.83 [01:37:15] RECOVERY - swiftproxy181 HTTPS on swiftproxy181 is OK: HTTP OK: Status line output matched "HTTP/1.1 404" - 352 bytes in 0.925 second response time [01:37:34] RECOVERY - swiftac111 Puppet on swiftac111 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:37:59] RECOVERY - swiftproxy131 Current Load on swiftproxy131 is OK: OK - load average: 1.38, 0.31, 0.10 [01:38:11] RECOVERY - mw133 Current Load on mw133 is OK: LOAD OK - total load average: 4.85, 8.72, 8.43 [01:38:26] RECOVERY - swiftproxy111 memcached on swiftproxy111 is OK: TCP OK - 0.129 second response time on 2a10:6740::6:201 port 11211 [01:39:17] RECOVERY - swiftproxy131 HTTPS on swiftproxy131 is OK: HTTP OK: Status line output matched "HTTP/1.1 404" - 352 bytes in 0.560 second response time [01:40:56] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 2.20, 3.48, 3.66 [01:41:22] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 12.29, 8.44, 5.29 [01:41:56] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 2.59, 3.27, 3.97 [01:41:59] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 11.42, 7.79, 5.18 [01:43:12] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/a8f2889eaf3b...a308984425bc [01:43:15] [02miraheze/dns] 07paladox 03a308984 - Match changes upstream [01:44:55] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 0.76, 2.25, 3.14 [01:47:22] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.62, 7.89, 6.11 [01:49:05] PROBLEM - mon181 Puppet on mon181 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/apt/keyrings/mariadb_release_signing_key.pgp] [01:49:22] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.39, 8.68, 6.66 [01:49:57] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 1.91, 2.44, 3.30 [01:50:21] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.93, 6.80, 5.23 [01:52:21] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 5.56, 6.41, 5.29 [01:53:22] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 5.46, 7.55, 6.73 [01:57:22] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 10.32, 8.71, 7.37 [01:57:59] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.66, 7.63, 7.19 [01:59:53] RECOVERY - db181 Puppet on db181 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [02:01:59] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 7.61, 8.04, 7.50 [02:03:50] RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp34.miraheze.org - CNAME OK [02:03:59] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.98, 7.32, 7.30 [02:07:59] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.51, 7.62, 7.39 [02:10:02] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.98, 7.20, 7.25 [02:11:18] PROBLEM - ping6 on mw134 is CRITICAL: PING CRITICAL - Packet loss = 0%, RTA = 372.99 ms [02:11:22] PROBLEM - ping6 on bast121 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 303.20 ms [02:11:25] PROBLEM - ping6 on ns2 is CRITICAL: PING CRITICAL - Packet loss = 0%, RTA = 341.72 ms [02:11:27] PROBLEM - ping6 on swiftac111 is CRITICAL: PING CRITICAL - Packet loss = 0%, RTA = 300.21 ms [02:11:35] PROBLEM - ping6 on matomo121 is CRITICAL: PING CRITICAL - Packet loss = 37%, RTA = 338.24 ms [02:11:51] PROBLEM - ping6 on mw141 is CRITICAL: PING CRITICAL - Packet loss = 0%, RTA = 327.70 ms [02:11:53] PROBLEM - ping6 on swiftobject113 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 329.86 ms [02:11:54] PROBLEM - ping6 on db112 is CRITICAL: PING CRITICAL - Packet loss = 0%, RTA = 309.64 ms [02:12:00] PROBLEM - ping6 on mem131 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 189.81 ms [02:13:19] RECOVERY - ping6 on mw134 is OK: PING OK - Packet loss = 0%, RTA = 129.26 ms [02:13:22] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.25, 7.10, 7.53 [02:13:23] [02miraheze/mw-config] 07MacFan4000 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/b06c00fdbe70...c60228b39036 [02:13:23] RECOVERY - ping6 on bast121 is OK: PING OK - Packet loss = 0%, RTA = 129.07 ms [02:13:26] [02miraheze/mw-config] 07MacFan4000 03c60228b - rm notice [02:13:28] RECOVERY - ping6 on ns2 is OK: PING OK - Packet loss = 0%, RTA = 128.98 ms [02:13:30] RECOVERY - ping6 on swiftac111 is OK: PING OK - Packet loss = 0%, RTA = 128.93 ms [02:13:37] RECOVERY - ping6 on matomo121 is OK: PING OK - Packet loss = 0%, RTA = 128.77 ms [02:13:50] RECOVERY - ping6 on mw141 is OK: PING OK - Packet loss = 0%, RTA = 128.84 ms [02:13:52] RECOVERY - ping6 on db112 is OK: PING OK - Packet loss = 0%, RTA = 129.01 ms [02:13:52] RECOVERY - ping6 on swiftobject113 is OK: PING OK - Packet loss = 0%, RTA = 128.86 ms [02:14:00] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.09, 7.57, 7.40 [02:14:00] RECOVERY - ping6 on mem131 is OK: PING OK - Packet loss = 0%, RTA = 128.97 ms [02:14:22] miraheze/mw-config - MacFan4000 the build passed. [02:14:27] [02miraheze/mw-config] 07MacFan4000 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/c60228b39036...b163000e98fd [02:14:28] [02miraheze/mw-config] 07MacFan4000 03b163000 - re-enable uploads [02:15:03] !log [macfan@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [02:15:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:15:23] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 10.71, 8.06, 7.79 [02:15:24] miraheze/mw-config - MacFan4000 the build passed. [02:15:45] !log [macfan@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 42s [02:15:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:15:59] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.89, 7.29, 7.32 [02:17:05] RECOVERY - mon181 Puppet on mon181 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:17:51] [02miraheze/dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/a308984425bc...bd1614ff8549 [02:17:54] [02miraheze/dns] 07Universal-Omega 03bd1614f - Use ns1 again [02:17:59] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 7.45, 8.02, 7.63 [02:20:28] [02miraheze/dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/bd1614ff8549...ac5bcb2f6572 [02:20:29] [02miraheze/dns] 07Universal-Omega 03ac5bcb2 - Remove cloud2 and use miraheze ns [02:21:39] [02miraheze/dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/ac5bcb2f6572...65083673234d [02:21:42] [02miraheze/dns] 07Universal-Omega 036508367 - Update [02:23:22] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.13, 7.74, 7.71 [02:26:01] [02miraheze/dns] 07Universal-Omega pushed 031 commit to 03WikiTide-cp [+0/-0/±1] 13https://github.com/miraheze/dns/commit/ed2dc21168ed [02:26:02] [02miraheze/dns] 07Universal-Omega 03ed2dc21 - Add WikiTide cp map [02:26:04] [02dns] 07Universal-Omega created branch 03WikiTide-cp - 13https://github.com/miraheze/dns [02:26:06] [02dns] 07Universal-Omega opened pull request 03#458: Add WikiTide cp map - 13https://github.com/miraheze/dns/pull/458 [02:26:22] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 11.08, 8.99, 7.26 [02:26:54] [02miraheze/dns] 07Universal-Omega pushed 031 commit to 03WikiTide-cp [+0/-0/±1] 13https://github.com/miraheze/dns/compare/ed2dc21168ed...141701c1110b [02:26:57] [02miraheze/dns] 07Universal-Omega 03141701c - Use cp2 [02:26:58] [02dns] 07Universal-Omega synchronize pull request 03#458: Add WikiTide cp map - 13https://github.com/miraheze/dns/pull/458 [02:27:22] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.94, 8.39, 7.98 [02:28:21] RECOVERY - mw134 Current Load on mw134 is OK: LOAD OK - total load average: 8.41, 8.68, 7.36 [02:31:53] PROBLEM - db181 Puppet on db181 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/apt/keyrings/mariadb_release_signing_key.pgp] [02:31:59] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.75, 7.51, 7.88 [02:32:38] [02miraheze/dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/65083673234d...b1732b4e86c2 [02:32:41] [02miraheze/dns] 07Universal-Omega 03b1732b4 - Remove cp map [02:34:30] RECOVERY - wikitide.net - reverse DNS on sslhost is OK: SSL OK - wikitide.net reverse DNS resolves to cp35.miraheze.org - NS RECORDS OK [02:35:22] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 5.88, 7.53, 7.86 [02:37:22] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 9.40, 8.16, 8.04 [02:37:59] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.50, 7.96, 7.90 [02:39:59] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 8.00, 7.88, 7.87 [02:43:59] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 7.96, 8.55, 8.18 [02:48:52] [02dns] 07Universal-Omega edited pull request 03#458: Add WikiTide cp map for static - 13https://github.com/miraheze/dns/pull/458 [02:49:56] [02miraheze/dns] 07Universal-Omega pushed 032 commits to 03WikiTide-cp [+0/-0/±2] 13https://github.com/miraheze/dns/compare/141701c1110b...bf126c97a062 [02:49:58] [02miraheze/dns] 07Universal-Omega 03bf126c9 - Merge branch 'master' into WikiTide-cp [02:49:59] [02dns] 07Universal-Omega synchronize pull request 03#458: Add WikiTide cp map for static - 13https://github.com/miraheze/dns/pull/458 [02:50:57] [02miraheze/dns] 07Universal-Omega pushed 031 commit to 03WikiTide-cp [+0/-0/±1] 13https://github.com/miraheze/dns/compare/bf126c97a062...bfbde76f14ef [02:50:59] [02miraheze/dns] 07Universal-Omega 03bfbde76 - Add [02:51:01] [02dns] 07Universal-Omega synchronize pull request 03#458: Add WikiTide cp map for static - 13https://github.com/miraheze/dns/pull/458 [02:51:19] [02dns] 07Universal-Omega closed pull request 03#458: Add WikiTide cp map for static - 13https://github.com/miraheze/dns/pull/458 [02:51:22] [02miraheze/dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/dns/compare/b1732b4e86c2...8ee7dcf8cf73 [02:51:24] [02miraheze/dns] 07Universal-Omega 038ee7dcf - Add WikiTide cp map for static (#458) [02:51:25] [02miraheze/dns] 07Universal-Omega deleted branch 03WikiTide-cp [02:51:26] [02dns] 07Universal-Omega deleted branch 03WikiTide-cp - 13https://github.com/miraheze/dns [02:56:01] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 11 datacenters are down: 139.99.236.151/cpweb, 2402:1f00:8100:400::1d23/cpweb, 51.79.55.151/cpweb, 2607:5300:205:200::2aa8/cpweb, 51.75.170.66/cpweb, 2001:41d0:801:2000::4089/cpweb, 146.59.44.171/cpweb, 2001:41d0:601:1100::4062/cpweb, 15.235.167.159/cpweb, 2402:1f00:8000:800::1b3d/cpweb, 63.141.240.6/cpweb [02:56:27] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 11 datacenters are down: 139.99.236.151/cpweb, 2402:1f00:8100:400::1d23/cpweb, 51.79.55.151/cpweb, 2607:5300:205:200::2aa8/cpweb, 51.75.170.66/cpweb, 2001:41d0:801:2000::4089/cpweb, 146.59.44.171/cpweb, 2001:41d0:601:1100::4062/cpweb, 15.235.167.159/cpweb, 2402:1f00:8000:800::1b3d/cpweb, 63.141.240.6/cpweb [02:59:59] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.84, 7.33, 7.90 [03:01:27] [02miraheze/dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/8ee7dcf8cf73...0e213e52997a [03:01:29] [02miraheze/dns] 07Universal-Omega 030e213e5 - Add cpweb2 [03:03:53] PROBLEM - db182 Puppet on db182 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/apt/keyrings/mariadb_release_signing_key.pgp] [03:05:22] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 5.52, 7.39, 7.98 [03:05:26] PROBLEM - wikitide.net - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wikitide.net could not be found [03:07:03] [02miraheze/dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/0e213e52997a...bb05eb99f597 [03:07:05] [02miraheze/dns] 07Universal-Omega 03bb05eb9 - Use cp2 for other services as well [03:10:37] [02miraheze/dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/bb05eb99f597...cdcb85035786 [03:10:38] [02miraheze/dns] 07Universal-Omega 03cdcb850 - Remove unnecessary txt records [03:11:22] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.64, 7.33, 7.69 [03:12:15] [02miraheze/dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/cdcb85035786...451dae08eeb8 [03:12:16] [02miraheze/dns] 07Universal-Omega 03451dae0 - Remove mail1 [03:13:23] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.39, 6.92, 7.49 [03:13:52] PROBLEM - mw143 Current Load on mw143 is CRITICAL: LOAD CRITICAL - total load average: 13.30, 8.58, 5.92 [03:15:52] RECOVERY - mw143 Current Load on mw143 is OK: LOAD OK - total load average: 9.94, 8.78, 6.30 [03:16:50] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 11.40, 9.40, 7.47 [03:17:23] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 11.00, 9.23, 6.92 [03:17:59] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.04, 6.09, 6.70 [03:18:50] RECOVERY - mw133 Current Load on mw133 is OK: LOAD OK - total load average: 6.99, 8.43, 7.35 [03:19:05] PROBLEM - mon181 Puppet on mon181 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/apt/keyrings/mariadb_release_signing_key.pgp] [03:19:24] RECOVERY - mw131 Current Load on mw131 is OK: LOAD OK - total load average: 6.81, 8.38, 6.89 [03:23:22] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.48, 7.01, 7.13 [03:23:59] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.80, 6.99, 6.87 [03:25:22] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.39, 7.12, 7.16 [03:27:22] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 10.55, 8.32, 7.59 [03:27:59] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.07, 7.59, 7.09 [03:29:22] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.91, 7.80, 7.47 [03:29:53] RECOVERY - db181 Puppet on db181 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [03:29:59] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.21, 7.08, 6.97 [03:37:22] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 6.46, 6.31, 6.77 [03:37:59] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 10.50, 8.15, 7.37 [03:39:59] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.17, 7.31, 7.16 [03:41:05] [02miraheze/dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/451dae08eeb8...94684ca4ea23 [03:41:06] [02miraheze/dns] 07Universal-Omega 0394684ca - Update CAA record for wikitide.net [03:41:22] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.06, 6.98, 6.95 [03:45:22] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 9.71, 7.75, 7.23 [03:45:59] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 10.15, 8.25, 7.50 [03:47:05] RECOVERY - mon181 Puppet on mon181 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [03:47:59] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.87, 7.52, 7.32 [03:49:22] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.92, 7.53, 7.23 [03:49:59] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.65, 8.23, 7.63 [03:51:59] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.77, 7.81, 7.55 [04:01:22] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 6.55, 6.36, 6.73 [04:01:53] PROBLEM - db181 Puppet on db181 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/apt/keyrings/mariadb_release_signing_key.pgp] [04:01:55] RECOVERY - db182 Puppet on db182 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:05:59] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.79, 6.20, 6.80 [04:08:18] PROBLEM - mw132 Puppet on mw132 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [04:23:04] PROBLEM - cp24 HTTPS on cp24 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [04:23:07] PROBLEM - mw141 HTTPS on mw141 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [04:24:58] PROBLEM - cp24 HTTPS on cp24 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.602 second response time [04:25:04] PROBLEM - mw141 HTTPS on mw141 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.563 second response time [04:25:05] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 6.39, 3.55, 1.66 [04:27:00] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 1.36, 2.55, 1.51 [04:29:53] RECOVERY - db181 Puppet on db181 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [04:36:18] RECOVERY - mw132 Puppet on mw132 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:38:38] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.79, 7.28, 6.89 [04:38:49] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.14, 7.00, 6.61 [04:40:38] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 14.99, 9.31, 7.62 [04:42:46] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 9.58, 7.51, 6.85 [04:44:35] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.74, 7.64, 7.33 [04:44:45] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.25, 7.39, 6.89 [04:46:43] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 4.82, 6.51, 6.64 [04:52:30] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 4.58, 6.21, 6.80 [04:58:30] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.22, 6.87, 6.55 [05:04:24] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 5.42, 6.30, 6.45 [05:16:12] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.66, 7.46, 6.82 [05:16:12] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.41, 7.85, 6.96 [05:18:10] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.44, 7.10, 6.77 [05:18:11] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.05, 7.45, 6.92 [05:20:08] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 5.27, 6.65, 6.66 [05:22:08] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.73, 6.43, 6.62 [05:23:54] PROBLEM - cloud11 Puppet on cloud11 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [05:34:37] PROBLEM - cp34 HTTPS on cp34 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [05:36:36] PROBLEM - cp34 HTTPS on cp34 is WARNING: HTTP WARNING: HTTP/2 404 - 3480 bytes in 0.614 second response time [05:49:05] PROBLEM - mon181 Puppet on mon181 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/apt/keyrings/mariadb_release_signing_key.pgp] [05:51:52] RECOVERY - cloud11 Puppet on cloud11 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [05:58:27] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.32, 6.79, 6.37 [05:58:52] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.54, 7.68, 6.68 [06:00:26] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 5.20, 6.08, 6.16 [06:04:48] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 4.69, 6.46, 6.52 [06:14:11] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.29, 6.50, 6.21 [06:16:09] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 5.44, 6.17, 6.14 [06:17:05] RECOVERY - mon181 Puppet on mon181 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [06:27:30] PROBLEM - mw134 HTTPS on mw134 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [06:27:53] PROBLEM - cp25 HTTPS on cp25 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [06:28:22] PROBLEM - cp35 HTTPS on cp35 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [06:28:33] PROBLEM - cp34 HTTPS on cp34 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [06:29:07] PROBLEM - mw141 HTTPS on mw141 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [06:29:48] PROBLEM - cp25 HTTPS on cp25 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.612 second response time [06:29:56] PROBLEM - cp24 Varnish Backends on cp24 is CRITICAL: 2 backends are down. mw131 mw134 [06:30:00] PROBLEM - cp34 Varnish Backends on cp34 is CRITICAL: 2 backends are down. mw131 mw134 [06:30:17] PROBLEM - cp35 HTTPS on cp35 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.570 second response time [06:30:33] PROBLEM - cp34 HTTPS on cp34 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.747 second response time [06:31:04] PROBLEM - mw141 HTTPS on mw141 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.680 second response time [06:31:36] PROBLEM - mw134 HTTPS on mw134 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 3.066 second response time [06:31:53] PROBLEM - db181 Puppet on db181 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/apt/keyrings/mariadb_release_signing_key.pgp] [06:31:56] RECOVERY - cp24 Varnish Backends on cp24 is OK: All 17 backends are healthy [06:32:00] RECOVERY - cp34 Varnish Backends on cp34 is OK: All 17 backends are healthy [06:34:47] !log DELETED AND DROPPED 321nailswiki (reset) [06:34:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:37:04] PROBLEM - cp24 HTTPS on cp24 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [06:38:00] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/importDump.php --wiki=321nailswiki --no-updates /home/reception/ImportDump/321nailswiki-20231230022150.xml (START) [06:38:01] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/importDump.php --wiki=321nailswiki --no-updates /home/reception/ImportDump/321nailswiki-20231230022150.xml (END - exit=256) [06:38:02] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/rebuildall.php --wiki=321nailswiki (START) [06:38:03] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/rebuildall.php --wiki=321nailswiki (END - exit=0) [06:38:04] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/initSiteStats.php --wiki=321nailswiki --active --update (END - exit=0) [06:38:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:38:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:38:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:38:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:38:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:38:59] PROBLEM - cp24 HTTPS on cp24 is WARNING: HTTP WARNING: HTTP/2 404 - 3457 bytes in 0.962 second response time [06:49:05] PROBLEM - mon181 Puppet on mon181 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/apt/keyrings/mariadb_release_signing_key.pgp] [07:01:53] RECOVERY - db181 Puppet on db181 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:04:04] PROBLEM - db182 Puppet on db182 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/apt/keyrings/mariadb_release_signing_key.pgp] [07:12:46] !log DELETED AND DROPPED https://phabricator.miraheze.org/P497 [07:12:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:17:05] RECOVERY - mon181 Puppet on mon181 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:32:05] RECOVERY - db182 Puppet on db182 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:33:55] [02mw-config] 07Reception123 closed pull request 03#5436: add SaneCase extension T11276 - 13https://github.com/miraheze/mw-config/pull/5436 [07:33:56] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±3] 13https://github.com/miraheze/mw-config/compare/b163000e98fd...1d20f3ccf52a [07:33:58] [02miraheze/mw-config] 07Reception123 031d20f3c - add SaneCase extension T11276 (#5436) [07:33:59] [02miraheze/mw-config] 07Reception123 deleted branch 03Reception123-patch-1 [07:34:01] [02mw-config] 07Reception123 deleted branch 03Reception123-patch-1 - 13https://github.com/miraheze/mw-config [07:34:02] !log [reception@mwtask141] starting deploy of {'pull': 'config', 'config': True, 'versions': ['1.40']} to [mw131, mw132, mw133, mw134, mw141, mw142, mw143, mw181, mw182, mwtask141, mwtask181] [07:34:03] !log [reception@mwtask141] DEPLOY ABORTED: Non-Zero Exit Code in prep, see output. [07:34:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:34:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:34:45] !log [reception@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [07:34:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:34:56] miraheze/mw-config - Reception123 the build passed. [07:35:31] !log [reception@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 46s [07:35:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:40:21] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 10.86, 9.40, 7.87 [07:44:21] RECOVERY - mw134 Current Load on mw134 is OK: LOAD OK - total load average: 5.86, 8.10, 7.73 [07:49:05] PROBLEM - mon181 Puppet on mon181 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/apt/keyrings/mariadb_release_signing_key.pgp] [07:58:09] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/importDump.php --wiki=321nailswiki --no-updates --username-prefix=millennium /home/reception/ImportDump/321nailswiki-20240116074520.xml (START) [07:58:10] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/importDump.php --wiki=321nailswiki --no-updates --username-prefix=millennium /home/reception/ImportDump/321nailswiki-20240116074520.xml (END - exit=256) [07:58:11] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/rebuildall.php --wiki=321nailswiki (START) [07:58:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:58:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:58:25] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/rebuildall.php --wiki=321nailswiki (END - exit=0) [07:58:27]