[00:04:22] PROBLEM - es141 Current Load on es141 is WARNING: WARNING - load average: 3.53, 2.73, 1.44 [00:06:18] RECOVERY - es141 Current Load on es141 is OK: OK - load average: 2.47, 2.61, 1.55 [00:12:55] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [01:00:10] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 7.91, 6.51, 4.90 [01:02:00] PROBLEM - cloud11 SMART on cloud11 is CRITICAL: connect to address 2a10:6740::6:200 port 5666: No route to hostconnect to host 2a10:6740::6:200 port 5666: No route to host [01:02:09] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 2.79, 5.11, 4.59 [01:03:05] PROBLEM - cloud11 SSH on cloud11 is CRITICAL: connect to address 2a10:6740::6:200 and port 22: No route to host [01:03:16] PROBLEM - Host cloud11 is DOWN: CRITICAL - Destination Unreachable (2a10:6740::6:200) [01:11:31] RECOVERY - Host cloud11 is UP: PING OK - Packet loss = 0%, RTA = 0.47 ms [01:12:23] PROBLEM - cloud11 SMART on cloud11 is WARNING: WARNING: [cciss,2] - Reallocated_Sector_Ct is non-zero (21) --- [cciss,3] - Reallocated_Sector_Ct is non-zero (7) --- [cciss,0] - Device is clean --- [cciss,1] - Device is clean --- [cciss,4] - Device is clean --- [cciss,5] - Device is clean --- [cciss,6] - Device is clean| [01:12:58] RECOVERY - cloud11 SSH on cloud11 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [01:28:15] PROBLEM - cp32 NTP time on cp32 is WARNING: NTP WARNING: Offset 0.1068920493 secs [01:46:15] RECOVERY - cp32 NTP time on cp32 is OK: NTP OK: Offset 0.09552758932 secs [01:53:17] RECOVERY - Host swiftobject112 is UP: PING OK - Packet loss = 0%, RTA = 1.38 ms [01:53:28] RECOVERY - ping6 on swiftobject112 is OK: PING OK - Packet loss = 0%, RTA = 0.75 ms [01:53:48] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 2.86, 0.92, 0.32 [01:53:49] RECOVERY - swiftobject112 ferm_active on swiftobject112 is OK: OK ferm input default policy is set [01:53:50] RECOVERY - swiftobject112 NTP time on swiftobject112 is OK: NTP OK: Offset 0.03003382683 secs [01:54:15] PROBLEM - cp32 NTP time on cp32 is WARNING: NTP WARNING: Offset 0.1313976049 secs [01:54:29] RECOVERY - Host swiftobject113 is UP: PING OK - Packet loss = 0%, RTA = 0.66 ms [01:54:49] RECOVERY - Host swiftobject111 is UP: PING OK - Packet loss = 0%, RTA = 0.74 ms [01:54:53] RECOVERY - swiftobject113 NTP time on swiftobject113 is OK: NTP OK: Offset 0.008232086897 secs [01:55:00] RECOVERY - swiftobject112 APT on swiftobject112 is OK: APT OK: 22 packages available for upgrade (0 critical updates). [01:55:00] RECOVERY - swiftobject112 conntrack_table_size on swiftobject112 is OK: OK: nf_conntrack is 0 % full [01:55:01] RECOVERY - swiftobject112 SSH on swiftobject112 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [01:55:08] RECOVERY - swiftobject112 Disk Space on swiftobject112 is OK: DISK OK - free space: / 59110 MB (11% inode=84%); [01:55:09] RECOVERY - swiftobject111 PowerDNS Recursor on swiftobject111 is OK: DNS OK: 0.020 seconds response time. miraheze.org returns 109.228.51.216,217.174.247.33,2a00:da00:1800:326::1,2a00:da00:1800:328::1 [01:55:09] RECOVERY - swiftobject112 Puppet on swiftobject112 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [01:55:12] RECOVERY - swiftobject112 Swift Object Service on swiftobject112 is OK: TCP OK - 0.001 second response time on 2a10:6740::6:204 port 6000 [01:55:13] RECOVERY - swiftobject112 PowerDNS Recursor on swiftobject112 is OK: DNS OK: 0.023 seconds response time. miraheze.org returns 109.228.51.216,217.174.247.33,2a00:da00:1800:326::1,2a00:da00:1800:328::1 [01:55:23] RECOVERY - swiftobject111 ferm_active on swiftobject111 is OK: OK ferm input default policy is set [01:55:24] RECOVERY - swiftobject111 SSH on swiftobject111 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [01:55:28] RECOVERY - swiftobject111 NTP time on swiftobject111 is OK: NTP OK: Offset 0.001561433077 secs [01:55:38] RECOVERY - swiftobject111 conntrack_table_size on swiftobject111 is OK: OK: nf_conntrack is 2 % full [01:55:43] RECOVERY - ping6 on swiftobject111 is OK: PING OK - Packet loss = 0%, RTA = 0.73 ms [01:55:48] RECOVERY - swiftobject113 ferm_active on swiftobject113 is OK: OK ferm input default policy is set [01:55:58] RECOVERY - swiftobject113 SSH on swiftobject113 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [01:56:03] RECOVERY - swiftobject111 Swift Object Service on swiftobject111 is OK: TCP OK - 0.001 second response time on 2a10:6740::6:203 port 6000 [01:56:08] RECOVERY - swiftobject113 PowerDNS Recursor on swiftobject113 is OK: DNS OK: 0.018 seconds response time. miraheze.org returns 109.228.51.216,217.174.247.33,2a00:da00:1800:326::1,2a00:da00:1800:328::1 [01:56:28] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 2.16, 1.45, 0.63 [01:56:43] PROBLEM - swiftobject111 Disk Space on swiftobject111 is WARNING: DISK WARNING - free space: / 58066 MB (10% inode=84%); [01:56:48] RECOVERY - swiftobject111 Puppet on swiftobject111 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [02:02:15] RECOVERY - cp32 NTP time on cp32 is OK: NTP OK: Offset 0.09619277716 secs [02:04:18] RECOVERY - Host swiftac111 is UP: PING OK - Packet loss = 0%, RTA = 0.91 ms [02:04:25] RECOVERY - ping6 on swiftac111 is OK: PING OK - Packet loss = 0%, RTA = 1.35 ms [02:05:08] PROBLEM - swiftac111 NTP time on swiftac111 is CRITICAL: connect to address 2a10:6740::6:202 port 5666: Connection refusedconnect to host 2a10:6740::6:202 port 5666: Connection refused [02:05:09] PROBLEM - swiftac111 Disk Space on swiftac111 is CRITICAL: connect to address 2a10:6740::6:202 port 5666: Connection refusedconnect to host 2a10:6740::6:202 port 5666: Connection refused [02:06:18] PROBLEM - swiftac111 PowerDNS Recursor on swiftac111 is CRITICAL: connect to address 2a10:6740::6:202 port 5666: Connection refusedconnect to host 2a10:6740::6:202 port 5666: Connection refused [02:07:00] RECOVERY - Host swiftproxy111 is UP: PING OK - Packet loss = 0%, RTA = 0.56 ms [02:07:53] PROBLEM - swiftproxy111 conntrack_table_size on swiftproxy111 is CRITICAL: connect to address 2a10:6740::6:201 port 5666: Connection refusedconnect to host 2a10:6740::6:201 port 5666: Connection refused [02:07:58] PROBLEM - swiftproxy111 Swift Proxy on swiftproxy111 is CRITICAL: connect to address 2a10:6740::6:201 and port 80: Connection refused [02:08:53] RECOVERY - ping6 on swiftproxy111 is OK: PING OK - Packet loss = 0%, RTA = 0.83 ms [02:14:04] RECOVERY - swiftac111 SSH on swiftac111 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [02:14:50] RECOVERY - swiftproxy111 SSH on swiftproxy111 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [02:20:12] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/7cd41d855ab5 [02:20:15] [02miraheze/puppet] 07paladox 037cd41d8 - swift: make sure swift is installed before creating /var/log/swift [02:20:16] [02puppet] 07paladox created branch 03paladox-patch-4 - 13https://github.com/miraheze/puppet [02:20:18] [02puppet] 07paladox opened pull request 03#3188: swift: make sure swift is installed before creating /var/log/swift - 13https://github.com/miraheze/puppet/pull/3188 [02:20:28] [02puppet] 07paladox closed pull request 03#3188: swift: make sure swift is installed before creating /var/log/swift - 13https://github.com/miraheze/puppet/pull/3188 [02:20:30] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/fecdaf3c39c8...1bcc1564e928 [02:20:33] [02miraheze/puppet] 07paladox 031bcc156 - swift: make sure swift is installed before creating /var/log/swift (#3188) [02:20:35] [02puppet] 07paladox deleted branch 03paladox-patch-4 - 13https://github.com/miraheze/puppet [02:20:38] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-4 [02:25:29] PROBLEM - swiftac111 conntrack_table_size on swiftac111 is UNKNOWN: NRPE: Unable to read output [02:25:38] RECOVERY - swiftac111 Disk Space on swiftac111 is OK: DISK OK - free space: / 120641 MB (97% inode=99%); [02:26:15] RECOVERY - swiftac111 Swift Container Service on swiftac111 is OK: TCP OK - 0.001 second response time on 2a10:6740::6:202 port 6001 [02:26:53] RECOVERY - swiftac111 Swift Account Service on swiftac111 is OK: TCP OK - 0.001 second response time on 2a10:6740::6:202 port 6002 [02:27:07] RECOVERY - swiftac111 PowerDNS Recursor on swiftac111 is OK: DNS OK: 0.191 seconds response time. miraheze.org returns 109.228.51.216,217.174.247.33,2a00:da00:1800:326::1,2a00:da00:1800:328::1 [02:27:17] RECOVERY - swiftac111 Puppet on swiftac111 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:27:28] RECOVERY - swiftac111 conntrack_table_size on swiftac111 is OK: OK: nf_conntrack is 0 % full [02:27:36] RECOVERY - swiftac111 NTP time on swiftac111 is OK: NTP OK: Offset -0.004924893379 secs [02:27:39] RECOVERY - swiftac111 APT on swiftac111 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [02:27:47] RECOVERY - swiftac111 Current Load on swiftac111 is OK: OK - load average: 0.29, 0.63, 0.39 [02:27:48] RECOVERY - swiftac111 ferm_active on swiftac111 is OK: OK ferm input default policy is set [02:29:58] PROBLEM - cp23 NTP time on cp23 is WARNING: NTP WARNING: Offset 0.1052594781 secs [02:31:43] [02miraheze/puppet] 07The-Voidwalker pushed 031 commit to 03revert-3186-static-return-empty-response [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/4de189f3ddc6 [02:31:46] [02miraheze/puppet] 07The-Voidwalker 034de189f - Revert "Return Varnish's default 200 response on all requests to static (#3186)" [02:31:47] [02puppet] 07The-Voidwalker created branch 03revert-3186-static-return-empty-response - 13https://github.com/miraheze/puppet [02:31:51] [02puppet] 07The-Voidwalker opened pull request 03#3189: Revert "Return Varnish's default 200 response on all requests to static" - 13https://github.com/miraheze/puppet/pull/3189 [02:32:01] RECOVERY - swiftproxy111 Swift Proxy on swiftproxy111 is OK: TCP OK - 0.001 second response time on 2a10:6740::6:201 port 80 [02:32:42] PROBLEM - swiftproxy111 conntrack_table_size on swiftproxy111 is UNKNOWN: NRPE: Unable to read output [02:32:54] PROBLEM - swiftproxy111 Puppet on swiftproxy111 is UNKNOWN: NRPE: Unable to read output [02:33:00] RECOVERY - swiftproxy111 NTP time on swiftproxy111 is OK: NTP OK: Offset -6.219744682e-05 secs [02:33:19] RECOVERY - swiftproxy111 APT on swiftproxy111 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [02:34:43] PROBLEM - swiftproxy111 conntrack_table_size on swiftproxy111 is CRITICAL: connect to address 2a10:6740::6:201 port 5666: No route to hostconnect to host 2a10:6740::6:201 port 5666: No route to host [02:34:51] PROBLEM - swiftproxy111 Puppet on swiftproxy111 is CRITICAL: connect to address 2a10:6740::6:201 port 5666: No route to hostconnect to host 2a10:6740::6:201 port 5666: No route to host [02:35:54] RECOVERY - swiftproxy111 Current Load on swiftproxy111 is OK: OK - load average: 0.60, 0.13, 0.04 [02:35:54] RECOVERY - swiftproxy111 Disk Space on swiftproxy111 is OK: DISK OK - free space: / 25830 MB (90% inode=95%); [02:36:00] RECOVERY - swiftproxy111 PowerDNS Recursor on swiftproxy111 is OK: DNS OK: 0.076 seconds response time. miraheze.org returns 109.228.51.216,217.174.247.33,2a00:da00:1800:326::1,2a00:da00:1800:328::1 [02:36:05] RECOVERY - swiftproxy111 ferm_active on swiftproxy111 is OK: OK ferm input default policy is set [02:36:40] RECOVERY - swiftproxy111 conntrack_table_size on swiftproxy111 is OK: OK: nf_conntrack is 0 % full [02:36:45] RECOVERY - swiftproxy111 Puppet on swiftproxy111 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [02:36:49] RECOVERY - swiftproxy111 memcached on swiftproxy111 is OK: TCP OK - 0.002 second response time on 2a10:6740::6:201 port 11211 [02:47:57] RECOVERY - cp23 NTP time on cp23 is OK: NTP OK: Offset 0.07812747359 secs [02:53:57] PROBLEM - cp23 NTP time on cp23 is WARNING: NTP WARNING: Offset 0.1242698133 secs [02:55:56] PROBLEM - swiftac111 SSH on swiftac111 is CRITICAL: connect to address 2a10:6740::6:202 and port 22: No route to host [02:56:10] PROBLEM - Host swiftac111 is DOWN: CRITICAL - Destination Unreachable (2a10:6740::6:202) [02:56:10] PROBLEM - swiftac111 NTP time on swiftac111 is CRITICAL: connect to address 2a10:6740::6:202 port 5666: No route to hostconnect to host 2a10:6740::6:202 port 5666: No route to host [02:59:58] RECOVERY - cp23 NTP time on cp23 is OK: NTP OK: Offset 0.09422200918 secs [03:00:13] RECOVERY - Host swiftac111 is UP: PING OK - Packet loss = 0%, RTA = 1.49 ms [03:01:59] RECOVERY - swiftac111 SSH on swiftac111 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [03:02:19] RECOVERY - swiftac111 NTP time on swiftac111 is OK: NTP OK: Offset 0.002157866955 secs [03:08:45] RECOVERY - swiftproxy111 HTTPS on swiftproxy111 is OK: HTTP OK: Status line output matched "HTTP/1.1 404" - 352 bytes in 0.062 second response time [03:09:21] RECOVERY - swiftproxy111 HTTP on swiftproxy111 is OK: HTTP OK: Status line output matched "HTTP/1.1 404" - 352 bytes in 0.021 second response time [03:16:17] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mar.in.ua. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [03:17:27] PROBLEM - swiftproxy111 HTTP on swiftproxy111 is CRITICAL: connect to address 2a10:6740::6:201 and port 80: No route to hostHTTP CRITICAL - Unable to open TCP socket [03:17:36] PROBLEM - ping6 on swiftproxy111 is CRITICAL: CRITICAL - Destination Unreachable (2a10:6740::6:201) [03:17:49] PROBLEM - swiftproxy111 conntrack_table_size on swiftproxy111 is CRITICAL: connect to address 2a10:6740::6:201 port 5666: No route to hostconnect to host 2a10:6740::6:201 port 5666: No route to host [03:17:57] PROBLEM - swiftproxy111 Disk Space on swiftproxy111 is CRITICAL: connect to address 2a10:6740::6:201 port 5666: No route to hostconnect to host 2a10:6740::6:201 port 5666: No route to host [03:17:57] PROBLEM - swiftproxy111 NTP time on swiftproxy111 is CRITICAL: connect to address 2a10:6740::6:201 port 5666: No route to hostconnect to host 2a10:6740::6:201 port 5666: No route to host [03:17:57] PROBLEM - swiftproxy111 Swift Proxy on swiftproxy111 is CRITICAL: connect to address 2a10:6740::6:201 and port 80: No route to host [03:17:57] PROBLEM - swiftproxy111 PowerDNS Recursor on swiftproxy111 is CRITICAL: connect to address 2a10:6740::6:201 port 5666: No route to hostconnect to host 2a10:6740::6:201 port 5666: No route to host [03:18:03] PROBLEM - swiftproxy111 Puppet on swiftproxy111 is CRITICAL: connect to address 2a10:6740::6:201 port 5666: No route to hostconnect to host 2a10:6740::6:201 port 5666: No route to host [03:18:03] PROBLEM - Host swiftproxy111 is DOWN: CRITICAL - Destination Unreachable (2a10:6740::6:201) [03:20:03] RECOVERY - Host swiftproxy111 is UP: PING OK - Packet loss = 0%, RTA = 0.68 ms [03:21:19] RECOVERY - swiftproxy111 HTTP on swiftproxy111 is OK: HTTP OK: Status line output matched "HTTP/1.1 404" - 352 bytes in 0.028 second response time [03:21:34] RECOVERY - ping6 on swiftproxy111 is OK: PING OK - Packet loss = 0%, RTA = 4.13 ms [03:21:49] RECOVERY - swiftproxy111 Disk Space on swiftproxy111 is OK: DISK OK - free space: / 25807 MB (90% inode=95%); RECOVERY - swiftproxy111 conntrack_table_size on swiftproxy111 is OK: OK: nf_conntrack is 0 % full [03:21:54] RECOVERY - swiftproxy111 PowerDNS Recursor on swiftproxy111 is OK: DNS OK: 0.023 seconds response time. miraheze.org returns 109.228.51.216,217.174.247.33,2a00:da00:1800:326::1,2a00:da00:1800:328::1 [03:21:59] RECOVERY - swiftproxy111 Swift Proxy on swiftproxy111 is OK: TCP OK - 0.001 second response time on 2a10:6740::6:201 port 80 [03:21:59] RECOVERY - swiftproxy111 NTP time on swiftproxy111 is OK: NTP OK: Offset 0.001459091902 secs [03:22:04] RECOVERY - swiftproxy111 Puppet on swiftproxy111 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [03:24:40] RECOVERY - Host swiftproxy131 is UP: PING OK - Packet loss = 0%, RTA = 0.67 ms [03:26:09] RECOVERY - swiftproxy131 HTTPS on swiftproxy131 is OK: HTTP OK: Status line output matched "HTTP/1.1 404" - 352 bytes in 0.052 second response time [03:26:34] RECOVERY - swiftproxy131 HTTP on swiftproxy131 is OK: HTTP OK: Status line output matched "HTTP/1.1 404" - 352 bytes in 0.030 second response time [03:26:34] RECOVERY - swiftproxy131 Current Load on swiftproxy131 is OK: OK - load average: 1.06, 0.54, 0.22 [03:27:00] RECOVERY - Host swiftobject122 is UP: PING OK - Packet loss = 0%, RTA = 2.13 ms [03:27:22] RECOVERY - swiftobject122 SSH on swiftobject122 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [03:27:29] RECOVERY - swiftobject122 NTP time on swiftobject122 is OK: NTP OK: Offset 0.000243216753 secs [03:27:51] RECOVERY - Host swiftobject121 is UP: PING OK - Packet loss = 0%, RTA = 0.80 ms [03:28:34] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 2.86, 1.68, 0.68 [03:28:44] RECOVERY - swiftobject121 ferm_active on swiftobject121 is OK: OK ferm input default policy is set [03:29:14] RECOVERY - swiftobject121 conntrack_table_size on swiftobject121 is OK: OK: nf_conntrack is 1 % full [03:29:29] RECOVERY - swiftobject121 Swift Object Service on swiftobject121 is OK: TCP OK - 0.001 second response time on 2a10:6740::6:316 port 6000 [03:29:34] RECOVERY - swiftobject121 SSH on swiftobject121 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [03:29:34] RECOVERY - swiftobject121 APT on swiftobject121 is OK: APT OK: 15 packages available for upgrade (0 critical updates). [03:29:49] RECOVERY - swiftobject121 PowerDNS Recursor on swiftobject121 is OK: DNS OK: 0.039 seconds response time. miraheze.org returns 109.228.51.216,217.174.247.33,2a00:da00:1800:326::1,2a00:da00:1800:328::1 [03:29:49] RECOVERY - swiftobject121 Puppet on swiftobject121 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [03:31:57] [02puppet] 07The-Voidwalker closed pull request 03#3189: Revert "Return Varnish's default 200 response on all requests to static" - 13https://github.com/miraheze/puppet/pull/3189 [03:31:59] [02miraheze/puppet] 07The-Voidwalker deleted branch 03revert-3186-static-return-empty-response [03:32:00] [02miraheze/puppet] 07The-Voidwalker pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/1bcc1564e928...13b672e07fc7 [03:32:02] [02miraheze/puppet] 07The-Voidwalker 0313b672e - Revert "Return Varnish's default 200 response on all requests to static" (#3189) [03:32:05] [02puppet] 07The-Voidwalker deleted branch 03revert-3186-static-return-empty-response - 13https://github.com/miraheze/puppet [03:45:39] PROBLEM - cp22 HTTP 4xx/5xx ERROR Rate on cp22 is WARNING: WARNING - NGINX Error Rate is 48% [03:45:52] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [03:47:39] PROBLEM - cp22 HTTP 4xx/5xx ERROR Rate on cp22 is CRITICAL: CRITICAL - NGINX Error Rate is 84% [03:55:39] RECOVERY - cp22 HTTP 4xx/5xx ERROR Rate on cp22 is OK: OK - NGINX Error Rate is 37% [03:57:07] PROBLEM - cp33 HTTP 4xx/5xx ERROR Rate on cp33 is CRITICAL: CRITICAL - NGINX Error Rate is 65% [03:58:16] PROBLEM - cp33 NTP time on cp33 is WARNING: NTP WARNING: Offset 0.1097498238 secs [03:59:07] PROBLEM - cp33 HTTP 4xx/5xx ERROR Rate on cp33 is WARNING: WARNING - NGINX Error Rate is 55% [04:01:06] PROBLEM - cp23 HTTP 4xx/5xx ERROR Rate on cp23 is CRITICAL: CRITICAL - NGINX Error Rate is 72% [04:02:31] PROBLEM - cp22 HTTP 4xx/5xx ERROR Rate on cp22 is WARNING: WARNING - NGINX Error Rate is 56% [04:03:06] PROBLEM - cp23 HTTP 4xx/5xx ERROR Rate on cp23 is WARNING: WARNING - NGINX Error Rate is 47% [04:03:07] PROBLEM - cp33 HTTP 4xx/5xx ERROR Rate on cp33 is CRITICAL: CRITICAL - NGINX Error Rate is 73% [04:05:06] PROBLEM - cp23 HTTP 4xx/5xx ERROR Rate on cp23 is CRITICAL: CRITICAL - NGINX Error Rate is 63% [04:07:06] PROBLEM - cp23 HTTP 4xx/5xx ERROR Rate on cp23 is WARNING: WARNING - NGINX Error Rate is 47% [04:07:07] PROBLEM - cp33 HTTP 4xx/5xx ERROR Rate on cp33 is WARNING: WARNING - NGINX Error Rate is 47% [04:08:15] RECOVERY - cp33 NTP time on cp33 is OK: NTP OK: Offset 0.08148241043 secs [04:08:23] PROBLEM - cp22 HTTP 4xx/5xx ERROR Rate on cp22 is CRITICAL: CRITICAL - NGINX Error Rate is 74% [04:09:06] PROBLEM - cp23 HTTP 4xx/5xx ERROR Rate on cp23 is CRITICAL: CRITICAL - NGINX Error Rate is 68% [04:11:06] PROBLEM - cp23 HTTP 4xx/5xx ERROR Rate on cp23 is WARNING: WARNING - NGINX Error Rate is 50% [04:11:07] PROBLEM - cp33 HTTP 4xx/5xx ERROR Rate on cp33 is CRITICAL: CRITICAL - NGINX Error Rate is 86% [04:11:31] PROBLEM - cp32 HTTP 4xx/5xx ERROR Rate on cp32 is CRITICAL: CRITICAL - NGINX Error Rate is 78% [04:12:18] PROBLEM - graylog121 Current Load on graylog121 is CRITICAL: CRITICAL - load average: 4.36, 3.77, 2.74 [04:14:14] PROBLEM - cp22 HTTP 4xx/5xx ERROR Rate on cp22 is WARNING: WARNING - NGINX Error Rate is 56% [04:14:18] PROBLEM - graylog121 Current Load on graylog121 is WARNING: WARNING - load average: 3.49, 3.68, 2.84 [04:15:07] PROBLEM - cp33 HTTP 4xx/5xx ERROR Rate on cp33 is WARNING: WARNING - NGINX Error Rate is 55% [04:15:31] PROBLEM - cp32 HTTP 4xx/5xx ERROR Rate on cp32 is WARNING: WARNING - NGINX Error Rate is 58% [04:17:06] PROBLEM - cp23 HTTP 4xx/5xx ERROR Rate on cp23 is CRITICAL: CRITICAL - NGINX Error Rate is 60% [04:17:07] PROBLEM - cp33 HTTP 4xx/5xx ERROR Rate on cp33 is CRITICAL: CRITICAL - NGINX Error Rate is 73% [04:18:09] PROBLEM - cp22 HTTP 4xx/5xx ERROR Rate on cp22 is CRITICAL: CRITICAL - NGINX Error Rate is 72% [04:19:06] PROBLEM - cp23 HTTP 4xx/5xx ERROR Rate on cp23 is WARNING: WARNING - NGINX Error Rate is 52% [04:19:31] PROBLEM - cp32 HTTP 4xx/5xx ERROR Rate on cp32 is CRITICAL: CRITICAL - NGINX Error Rate is 84% [04:20:18] PROBLEM - graylog121 Current Load on graylog121 is CRITICAL: CRITICAL - load average: 4.57, 3.80, 3.14 [04:21:07] PROBLEM - cp33 HTTP 4xx/5xx ERROR Rate on cp33 is WARNING: WARNING - NGINX Error Rate is 56% [04:22:34] PROBLEM - swiftproxy131 Current Load on swiftproxy131 is CRITICAL: CRITICAL - load average: 9.18, 7.12, 4.73 [04:23:06] RECOVERY - cp23 HTTP 4xx/5xx ERROR Rate on cp23 is OK: OK - NGINX Error Rate is 39% [04:23:07] PROBLEM - cp33 HTTP 4xx/5xx ERROR Rate on cp33 is CRITICAL: CRITICAL - NGINX Error Rate is 63% [04:24:18] PROBLEM - graylog121 Current Load on graylog121 is WARNING: WARNING - load average: 3.84, 3.76, 3.27 [04:26:33] PROBLEM - swiftproxy131 Current Load on swiftproxy131 is WARNING: WARNING - load average: 7.90, 6.78, 5.10 [04:27:06] PROBLEM - cp23 HTTP 4xx/5xx ERROR Rate on cp23 is WARNING: WARNING - NGINX Error Rate is 56% [04:28:33] RECOVERY - swiftproxy131 Current Load on swiftproxy131 is OK: OK - load average: 4.84, 6.06, 5.04 [04:29:07] PROBLEM - cp23 HTTP 4xx/5xx ERROR Rate on cp23 is CRITICAL: CRITICAL - NGINX Error Rate is 65% [04:29:52] RECOVERY - cp22 HTTP 4xx/5xx ERROR Rate on cp22 is OK: OK - NGINX Error Rate is 39% [04:30:18] PROBLEM - graylog121 Current Load on graylog121 is CRITICAL: CRITICAL - load average: 3.84, 4.03, 3.57 [04:31:07] PROBLEM - cp33 HTTP 4xx/5xx ERROR Rate on cp33 is WARNING: WARNING - NGINX Error Rate is 52% [04:32:18] PROBLEM - graylog121 Current Load on graylog121 is WARNING: WARNING - load average: 3.71, 3.83, 3.55 [04:33:07] PROBLEM - cp33 HTTP 4xx/5xx ERROR Rate on cp33 is CRITICAL: CRITICAL - NGINX Error Rate is 69% [04:33:44] PROBLEM - cp22 HTTP 4xx/5xx ERROR Rate on cp22 is WARNING: WARNING - NGINX Error Rate is 59% [04:34:18] PROBLEM - graylog121 Current Load on graylog121 is CRITICAL: CRITICAL - load average: 4.19, 4.05, 3.67 [04:35:07] PROBLEM - cp23 HTTP 4xx/5xx ERROR Rate on cp23 is WARNING: WARNING - NGINX Error Rate is 40% [04:35:31] PROBLEM - cp32 HTTP 4xx/5xx ERROR Rate on cp32 is WARNING: WARNING - NGINX Error Rate is 56% [04:36:08] !log [void@mwtask141] sudo -u www-data php /srv/mediawiki/w/extensions/CreateWiki/maintenance/setContainersAccess.php --wiki=tincwiki (END - exit=0) [04:36:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:36:18] PROBLEM - graylog121 Current Load on graylog121 is WARNING: WARNING - load average: 2.70, 3.58, 3.55 [04:36:38] !log [void@mwtask141] sudo -u www-data php /srv/mediawiki/w/extensions/CreateWiki/maintenance/setContainersAccess.php --wiki=cpwptwiki (END - exit=0) [04:36:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:37:15] !log [void@mwtask141] sudo -u www-data php /srv/mediawiki/w/extensions/CreateWiki/maintenance/setContainersAccess.php --wiki=metawiki (END - exit=0) [04:37:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:37:31] PROBLEM - cp32 HTTP 4xx/5xx ERROR Rate on cp32 is CRITICAL: CRITICAL - NGINX Error Rate is 63% [04:37:40] PROBLEM - cp22 HTTP 4xx/5xx ERROR Rate on cp22 is CRITICAL: CRITICAL - NGINX Error Rate is 81% [04:37:51] !log [void@mwtask141] sudo -u www-data php /srv/mediawiki/w/extensions/CreateWiki/maintenance/setContainersAccess.php --wiki=commonswiki (END - exit=0) [04:37:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:38:18] PROBLEM - graylog121 Current Load on graylog121 is CRITICAL: CRITICAL - load average: 4.30, 4.11, 3.76 [04:39:31] PROBLEM - cp32 HTTP 4xx/5xx ERROR Rate on cp32 is WARNING: WARNING - NGINX Error Rate is 52% [04:40:18] PROBLEM - graylog121 Current Load on graylog121 is WARNING: WARNING - load average: 3.83, 3.84, 3.70 [04:41:07] PROBLEM - cp23 HTTP 4xx/5xx ERROR Rate on cp23 is CRITICAL: CRITICAL - NGINX Error Rate is 68% [04:41:07] PROBLEM - cp33 HTTP 4xx/5xx ERROR Rate on cp33 is WARNING: WARNING - NGINX Error Rate is 48% [04:43:06] PROBLEM - cp23 HTTP 4xx/5xx ERROR Rate on cp23 is WARNING: WARNING - NGINX Error Rate is 50% [04:43:07] PROBLEM - cp33 HTTP 4xx/5xx ERROR Rate on cp33 is CRITICAL: CRITICAL - NGINX Error Rate is 67% [04:43:31] PROBLEM - cp32 HTTP 4xx/5xx ERROR Rate on cp32 is CRITICAL: CRITICAL - NGINX Error Rate is 66% [04:43:39] PROBLEM - cp22 HTTP 4xx/5xx ERROR Rate on cp22 is WARNING: WARNING - NGINX Error Rate is 57% [04:45:06] RECOVERY - cp23 HTTP 4xx/5xx ERROR Rate on cp23 is OK: OK - NGINX Error Rate is 32% [04:45:39] PROBLEM - cp22 HTTP 4xx/5xx ERROR Rate on cp22 is CRITICAL: CRITICAL - NGINX Error Rate is 63% [04:47:31] RECOVERY - cp32 HTTP 4xx/5xx ERROR Rate on cp32 is OK: OK - NGINX Error Rate is 31% [04:47:32] !log [void@mwtask141] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/w/extensions/CreateWiki/maintenance/setContainersAccess.php (START) [04:47:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:47:40] PROBLEM - cp22 HTTP 4xx/5xx ERROR Rate on cp22 is WARNING: WARNING - NGINX Error Rate is 41% [04:49:06] PROBLEM - cp23 HTTP 4xx/5xx ERROR Rate on cp23 is WARNING: WARNING - NGINX Error Rate is 51% [04:49:39] PROBLEM - cp22 HTTP 4xx/5xx ERROR Rate on cp22 is CRITICAL: CRITICAL - NGINX Error Rate is 65% [04:50:18] PROBLEM - graylog121 Current Load on graylog121 is CRITICAL: CRITICAL - load average: 4.15, 3.67, 3.62 [04:51:07] PROBLEM - cp33 HTTP 4xx/5xx ERROR Rate on cp33 is WARNING: WARNING - NGINX Error Rate is 54% [04:51:31] PROBLEM - cp32 HTTP 4xx/5xx ERROR Rate on cp32 is CRITICAL: CRITICAL - NGINX Error Rate is 82% [04:51:39] PROBLEM - cp22 HTTP 4xx/5xx ERROR Rate on cp22 is WARNING: WARNING - NGINX Error Rate is 50% [04:54:18] PROBLEM - graylog121 Current Load on graylog121 is WARNING: WARNING - load average: 3.41, 3.73, 3.67 [04:55:07] RECOVERY - cp23 HTTP 4xx/5xx ERROR Rate on cp23 is OK: OK - NGINX Error Rate is 36% [04:55:31] PROBLEM - cp32 HTTP 4xx/5xx ERROR Rate on cp32 is WARNING: WARNING - NGINX Error Rate is 54% [04:55:39] PROBLEM - cp22 HTTP 4xx/5xx ERROR Rate on cp22 is CRITICAL: CRITICAL - NGINX Error Rate is 74% [04:57:07] PROBLEM - cp33 HTTP 4xx/5xx ERROR Rate on cp33 is CRITICAL: CRITICAL - NGINX Error Rate is 60% [04:57:39] PROBLEM - cp22 HTTP 4xx/5xx ERROR Rate on cp22 is WARNING: WARNING - NGINX Error Rate is 44% [04:59:31] PROBLEM - cp32 HTTP 4xx/5xx ERROR Rate on cp32 is CRITICAL: CRITICAL - NGINX Error Rate is 66% [05:00:18] PROBLEM - graylog121 Current Load on graylog121 is CRITICAL: CRITICAL - load average: 4.69, 3.79, 3.64 [05:01:07] RECOVERY - cp33 HTTP 4xx/5xx ERROR Rate on cp33 is OK: OK - NGINX Error Rate is 35% [05:01:31] PROBLEM - cp32 HTTP 4xx/5xx ERROR Rate on cp32 is WARNING: WARNING - NGINX Error Rate is 43% [05:01:39] RECOVERY - cp22 HTTP 4xx/5xx ERROR Rate on cp22 is OK: OK - NGINX Error Rate is 17% [05:02:18] PROBLEM - graylog121 Current Load on graylog121 is WARNING: WARNING - load average: 2.79, 3.37, 3.51 [05:05:31] PROBLEM - cp32 HTTP 4xx/5xx ERROR Rate on cp32 is CRITICAL: CRITICAL - NGINX Error Rate is 63% [05:06:18] PROBLEM - graylog121 Current Load on graylog121 is CRITICAL: CRITICAL - load average: 4.09, 3.46, 3.49 [05:07:06] PROBLEM - cp23 HTTP 4xx/5xx ERROR Rate on cp23 is WARNING: WARNING - NGINX Error Rate is 52% [05:07:31] PROBLEM - cp32 HTTP 4xx/5xx ERROR Rate on cp32 is WARNING: WARNING - NGINX Error Rate is 48% [05:09:06] PROBLEM - cp23 HTTP 4xx/5xx ERROR Rate on cp23 is CRITICAL: CRITICAL - NGINX Error Rate is 90% [05:09:07] PROBLEM - cp33 HTTP 4xx/5xx ERROR Rate on cp33 is WARNING: WARNING - NGINX Error Rate is 48% [05:10:18] PROBLEM - graylog121 Current Load on graylog121 is WARNING: WARNING - load average: 3.96, 3.96, 3.70 [05:11:06] RECOVERY - cp23 HTTP 4xx/5xx ERROR Rate on cp23 is OK: OK - NGINX Error Rate is 28% [05:12:31] PROBLEM - cp22 HTTP 4xx/5xx ERROR Rate on cp22 is CRITICAL: CRITICAL - NGINX Error Rate is 91% [05:13:07] PROBLEM - cp33 HTTP 4xx/5xx ERROR Rate on cp33 is CRITICAL: CRITICAL - NGINX Error Rate is 62% [05:13:31] PROBLEM - cp32 HTTP 4xx/5xx ERROR Rate on cp32 is CRITICAL: CRITICAL - NGINX Error Rate is 63% [05:14:18] PROBLEM - graylog121 Current Load on graylog121 is CRITICAL: CRITICAL - load average: 4.52, 4.12, 3.81 [05:14:28] PROBLEM - cp22 HTTP 4xx/5xx ERROR Rate on cp22 is WARNING: WARNING - NGINX Error Rate is 46% [05:16:26] PROBLEM - cp22 HTTP 4xx/5xx ERROR Rate on cp22 is CRITICAL: CRITICAL - NGINX Error Rate is 79% [05:17:31] PROBLEM - cp32 HTTP 4xx/5xx ERROR Rate on cp32 is WARNING: WARNING - NGINX Error Rate is 52% [05:18:23] RECOVERY - cp22 HTTP 4xx/5xx ERROR Rate on cp22 is OK: OK - NGINX Error Rate is 30% [05:19:07] PROBLEM - cp33 HTTP 4xx/5xx ERROR Rate on cp33 is WARNING: WARNING - NGINX Error Rate is 50% [05:21:07] RECOVERY - cp33 HTTP 4xx/5xx ERROR Rate on cp33 is OK: OK - NGINX Error Rate is 38% [05:22:18] PROBLEM - graylog121 Current Load on graylog121 is WARNING: WARNING - load average: 3.46, 3.71, 3.77 [05:25:07] PROBLEM - cp33 HTTP 4xx/5xx ERROR Rate on cp33 is CRITICAL: CRITICAL - NGINX Error Rate is 71% [05:25:31] PROBLEM - cp32 HTTP 4xx/5xx ERROR Rate on cp32 is CRITICAL: CRITICAL - NGINX Error Rate is 71% [05:27:07] PROBLEM - cp33 HTTP 4xx/5xx ERROR Rate on cp33 is WARNING: WARNING - NGINX Error Rate is 52% [05:27:31] PROBLEM - cp32 HTTP 4xx/5xx ERROR Rate on cp32 is WARNING: WARNING - NGINX Error Rate is 53% [05:28:18] PROBLEM - graylog121 Current Load on graylog121 is CRITICAL: CRITICAL - load average: 4.48, 3.72, 3.72 [05:29:31] PROBLEM - cp32 HTTP 4xx/5xx ERROR Rate on cp32 is CRITICAL: CRITICAL - NGINX Error Rate is 88% [05:31:07] PROBLEM - cp33 HTTP 4xx/5xx ERROR Rate on cp33 is CRITICAL: CRITICAL - NGINX Error Rate is 78% [05:31:31] RECOVERY - cp32 HTTP 4xx/5xx ERROR Rate on cp32 is OK: OK - NGINX Error Rate is 39% [05:33:07] PROBLEM - cp33 HTTP 4xx/5xx ERROR Rate on cp33 is WARNING: WARNING - NGINX Error Rate is 46% [05:34:18] PROBLEM - graylog121 Current Load on graylog121 is WARNING: WARNING - load average: 2.71, 3.84, 3.87 [05:35:31] PROBLEM - cp32 HTTP 4xx/5xx ERROR Rate on cp32 is WARNING: WARNING - NGINX Error Rate is 45% [05:36:20] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/d48392eefa44...1e84d427ffd5 [05:36:23] [02miraheze/mw-config] 07Reception123 031e84d42 - rm file issues sitenotice [05:36:59] [02miraheze/MirahezeMagic] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/MirahezeMagic/compare/31bd6b671e93...2f088220e448 [05:37:00] [02miraheze/MirahezeMagic] 07Reception123 032f08822 - rm upload issue notice [05:37:07] RECOVERY - cp33 HTTP 4xx/5xx ERROR Rate on cp33 is OK: OK - NGINX Error Rate is 29% [05:37:19] miraheze/mw-config - Reception123 the build passed. [05:38:19] !log [reception@mwtask141] starting deploy of {'pull': 'config', 'config': True} to all [05:38:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:38:27] !log [reception@mwtask141] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 8s [05:38:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:39:31] PROBLEM - cp32 HTTP 4xx/5xx ERROR Rate on cp32 is CRITICAL: CRITICAL - NGINX Error Rate is 75% [05:40:18] PROBLEM - graylog121 Current Load on graylog121 is CRITICAL: CRITICAL - load average: 4.15, 3.93, 3.89 [05:41:31] RECOVERY - cp32 HTTP 4xx/5xx ERROR Rate on cp32 is OK: OK - NGINX Error Rate is 37% [05:42:18] PROBLEM - graylog121 Current Load on graylog121 is WARNING: WARNING - load average: 3.75, 3.69, 3.79 [05:45:31] PROBLEM - cp32 HTTP 4xx/5xx ERROR Rate on cp32 is WARNING: WARNING - NGINX Error Rate is 56% [05:47:00] miraheze/MirahezeMagic - Reception123 the build passed. [05:47:07] PROBLEM - cp33 HTTP 4xx/5xx ERROR Rate on cp33 is WARNING: WARNING - NGINX Error Rate is 51% [05:49:07] PROBLEM - cp33 HTTP 4xx/5xx ERROR Rate on cp33 is CRITICAL: CRITICAL - NGINX Error Rate is 68% [05:50:19] PROBLEM - cp23 HTTP 4xx/5xx ERROR Rate on cp23 is CRITICAL: CRITICAL - NGINX Error Rate is 62% [05:50:19] PROBLEM - cp22 HTTP 4xx/5xx ERROR Rate on cp22 is CRITICAL: CRITICAL - NGINX Error Rate is 91% [05:51:07] !log [salt-user@mw121] sudo -u www-data php /srv/mediawiki/w/extensions/SemanticMediaWiki/maintenance/setupStore.php --wiki=twistedwonderlandwiki (END - exit=0) [05:51:08] !log [salt-user@mw131] sudo -u www-data php /srv/mediawiki/w/extensions/SemanticMediaWiki/maintenance/setupStore.php --wiki=twistedwonderlandwiki (END - exit=0) [05:51:09] !log [salt-user@mw132] sudo -u www-data php /srv/mediawiki/w/extensions/SemanticMediaWiki/maintenance/setupStore.php --wiki=twistedwonderlandwiki (END - exit=0) [05:51:10] !log [salt-user@mw142] sudo -u www-data php /srv/mediawiki/w/extensions/SemanticMediaWiki/maintenance/setupStore.php --wiki=twistedwonderlandwiki (END - exit=0) [05:51:11] !log [salt-user@mwtask141] sudo -u www-data php /srv/mediawiki/w/extensions/SemanticMediaWiki/maintenance/setupStore.php --wiki=twistedwonderlandwiki (END - exit=0) [05:51:12] !log [salt-user@mw122] sudo -u www-data php /srv/mediawiki/w/extensions/SemanticMediaWiki/maintenance/setupStore.php --wiki=twistedwonderlandwiki (END - exit=0) [05:51:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:51:13] !log [salt-user@mw141] sudo -u www-data php /srv/mediawiki/w/extensions/SemanticMediaWiki/maintenance/setupStore.php --wiki=twistedwonderlandwiki (END - exit=0) [05:51:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:51:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:51:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:51:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:51:31] RECOVERY - cp32 HTTP 4xx/5xx ERROR Rate on cp32 is OK: OK - NGINX Error Rate is 38% [05:51:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:51:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:52:13] PROBLEM - cp23 HTTP 4xx/5xx ERROR Rate on cp23 is WARNING: WARNING - NGINX Error Rate is 43% [05:53:07] PROBLEM - cp33 HTTP 4xx/5xx ERROR Rate on cp33 is WARNING: WARNING - NGINX Error Rate is 52% [05:54:07] RECOVERY - cp23 HTTP 4xx/5xx ERROR Rate on cp23 is OK: OK - NGINX Error Rate is 24% [05:55:07] RECOVERY - cp33 HTTP 4xx/5xx ERROR Rate on cp33 is OK: OK - NGINX Error Rate is 18% [05:56:11] PROBLEM - cp22 HTTP 4xx/5xx ERROR Rate on cp22 is WARNING: WARNING - NGINX Error Rate is 40% [05:57:21] !log [@test131] starting deploy of {'config': True} to all [05:57:22] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [05:57:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:57:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:58:18] PROBLEM - graylog121 Current Load on graylog121 is CRITICAL: CRITICAL - load average: 4.80, 3.98, 3.67 [06:00:05] PROBLEM - cp22 HTTP 4xx/5xx ERROR Rate on cp22 is CRITICAL: CRITICAL - NGINX Error Rate is 76% [06:00:14] PROBLEM - cp32 HTTP 4xx/5xx ERROR Rate on cp32 is CRITICAL: CRITICAL - NGINX Error Rate is 79% [06:00:18] PROBLEM - graylog121 Current Load on graylog121 is WARNING: WARNING - load average: 3.61, 3.73, 3.61 [06:02:02] PROBLEM - cp22 HTTP 4xx/5xx ERROR Rate on cp22 is WARNING: WARNING - NGINX Error Rate is 49% [06:02:08] RECOVERY - cp32 HTTP 4xx/5xx ERROR Rate on cp32 is OK: OK - NGINX Error Rate is 22% [06:04:00] PROBLEM - cp22 HTTP 4xx/5xx ERROR Rate on cp22 is CRITICAL: CRITICAL - NGINX Error Rate is 61% [06:05:34] PROBLEM - cp23 HTTP 4xx/5xx ERROR Rate on cp23 is WARNING: WARNING - NGINX Error Rate is 43% [06:07:28] RECOVERY - cp23 HTTP 4xx/5xx ERROR Rate on cp23 is OK: OK - NGINX Error Rate is 22% [06:07:54] PROBLEM - cp22 HTTP 4xx/5xx ERROR Rate on cp22 is WARNING: WARNING - NGINX Error Rate is 49% [06:09:51] PROBLEM - cp22 HTTP 4xx/5xx ERROR Rate on cp22 is CRITICAL: CRITICAL - NGINX Error Rate is 67% [06:11:49] PROBLEM - cp22 HTTP 4xx/5xx ERROR Rate on cp22 is WARNING: WARNING - NGINX Error Rate is 47% [06:13:35] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.05, 10.11, 7.92 [06:13:46] RECOVERY - cp22 HTTP 4xx/5xx ERROR Rate on cp22 is OK: OK - NGINX Error Rate is 26% [06:14:18] PROBLEM - graylog121 Current Load on graylog121 is CRITICAL: CRITICAL - load average: 5.05, 3.96, 3.64 [06:15:35] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 9.48, 9.62, 8.00 [06:16:16] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.60, 5.75, 4.52 [06:17:48] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 7.00, 6.52, 4.79 [06:18:31] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.65, 7.11, 5.16 [06:19:49] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 8.54, 7.02, 5.18 [06:20:16] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.73, 6.44, 5.09 [06:20:17] PROBLEM - cp22 NTP time on cp22 is WARNING: NTP WARNING: Offset 0.2314616144 secs [06:20:31] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.76, 7.14, 5.39 [06:21:49] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 4.97, 6.07, 5.05 [06:22:18] PROBLEM - graylog121 Current Load on graylog121 is WARNING: WARNING - load average: 2.43, 3.44, 3.58 [06:24:31] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.39, 7.30, 5.82 [06:26:31] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.36, 7.36, 6.02 [06:28:18] RECOVERY - graylog121 Current Load on graylog121 is OK: OK - load average: 2.52, 2.96, 3.31 [06:28:31] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 4.68, 6.36, 5.82 [06:32:17] RECOVERY - cp22 NTP time on cp22 is OK: NTP OK: Offset -0.01894167066 secs [06:36:17] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 6.83, 4.83, 2.84 [06:38:17] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 5.79, 5.55, 3.38 [06:40:18] PROBLEM - graylog121 Current Load on graylog121 is CRITICAL: CRITICAL - load average: 4.22, 3.40, 3.25 [06:41:13] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.50, 7.16, 6.38 [06:42:18] RECOVERY - graylog121 Current Load on graylog121 is OK: OK - load average: 2.94, 3.25, 3.22 [06:43:08] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 6.31, 6.75, 6.31 [06:49:02] PROBLEM - wiki.nj.cn.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.nj.cn.eu.org All nameservers failed to answer the query. [06:58:03] !log [void@mwtask141] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/w/extensions/CreateWiki/maintenance/setContainersAccess.php (END - exit=0) [06:58:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:07:54] [02mw-config] 07AgentIsai opened pull request 03#5198: Re-enable file uploads - 13https://github.com/miraheze/mw-config/pull/5198 [07:08:54] miraheze/mw-config - AgentIsai the build passed. [07:13:30] [02mw-config] 07Reception123 closed pull request 03#5198: Re-enable file uploads - 13https://github.com/miraheze/mw-config/pull/5198 [07:13:32] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/1e84d427ffd5...ca1b85482240 [07:13:33] [02miraheze/mw-config] 07AgentIsai 03ca1b854 - Re-enable file uploads (#5198) [07:14:27] miraheze/mw-config - Reception123 the build passed. [07:14:55] [02miraheze/mediawiki] 07Reception123 pushed 031 commit to 03REL1_39 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/37e4cff5d3ff...a0effa59102a [07:14:56] [02miraheze/mediawiki] 07Reception123 03a0effa5 - Update MirahezeMagic [07:15:29] !log [reception@mwtask141] starting deploy of {'pull': 'config', 'config': True} to all [07:15:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:15:39] !log [reception@mwtask141] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 9s [07:15:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:16:01] !log [reception@mwtask141] starting deploy of {'world': True, 'l10n': True} to all [07:16:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:18:05] RECOVERY - wiki.nj.cn.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.nj.cn.eu.org reverse DNS resolves to cp22.miraheze.org - CNAME OK [07:27:23] !log [@test131] starting deploy of {'config': True} to all [07:27:24] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [07:27:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:27:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:30:18] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 8.65, 6.49, 3.77 [08:32:17] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 2.91, 5.24, 3.66 [09:15:57] PROBLEM - cp23 NTP time on cp23 is WARNING: NTP WARNING: Offset 0.2765727341 secs [09:35:57] RECOVERY - cp23 NTP time on cp23 is OK: NTP OK: Offset -0.01267427206 secs [10:24:17] PROBLEM - cp22 NTP time on cp22 is WARNING: NTP WARNING: Offset 0.1132329404 secs [10:27:37] !log [reception@mwtask141] starting deploy of {'world': True, 'l10n': True} to all [10:27:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:32:17] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 8.71, 6.38, 3.57 [10:34:17] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 3.92, 5.33, 3.53 [10:39:54] PROBLEM - wiki.andreijiroh.uk.eu.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.andreijiroh.uk.eu.org' expires in 15 day(s) (Tue 02 May 2023 10:34:09 GMT +0000). [10:42:31] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/70b841a3fb3b...da11aa0a8a34 [10:42:32] [02miraheze/ssl] 07MirahezeSSLBot 03da11aa0 - Bot: Update SSL cert for wiki.andreijiroh.uk.eu.org [10:44:50] !log [reception@mwtask141] finished deploy of {'world': True, 'l10n': True} to all - SUCCESS in 1032s [10:44:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:52:17] RECOVERY - cp22 NTP time on cp22 is OK: NTP OK: Offset 0.09286889434 secs [11:01:03] PROBLEM - test131 Puppet on test131 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [11:10:38] Why has nginx failed [11:10:43] !sre [11:10:54] Orange_Star, Reception123: please check that ^ [11:11:17] Someone needs to run conftest [11:17:48] weird, will look [11:29:01] RECOVERY - test131 Puppet on test131 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [11:37:54] RECOVERY - wiki.andreijiroh.uk.eu.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.andreijiroh.uk.eu.org' will expire on Sat 15 Jul 2023 09:42:22 GMT +0000. [11:41:57] PROBLEM - cp23 NTP time on cp23 is WARNING: NTP WARNING: Offset 0.1213723421 secs [11:43:30] PROBLEM - pubwiki.lab.co.pl - reverse DNS on sslhost is WARNING: Timeout: The DNS operation timed out after 5.4065210819244385 seconds [11:49:57] RECOVERY - cp23 NTP time on cp23 is OK: NTP OK: Offset 0.0742841661 secs [11:53:34] PROBLEM - swiftobject112 Disk Space on swiftobject112 is WARNING: DISK WARNING - free space: / 58597 MB (10% inode=84%); [12:10:14] [02puppet] 07RhinosF1 opened pull request 03#3190: Update mwscript.py - 13https://github.com/miraheze/puppet/pull/3190 [12:16:31] [02puppet] 07RhinosF1 synchronize pull request 03#3190: Update mwscript.py - 13https://github.com/miraheze/puppet/pull/3190 [12:24:17] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 8.28, 5.76, 3.45 [12:26:17] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 6.41, 6.38, 4.00 [12:42:16]