[00:01:19] RECOVERY - mw132 Disk Space on mw132 is OK: DISK OK - free space: / 3149MiB (13% inode=69%); [00:01:33] PROBLEM - mw134 Current Load on mw134 is CRITICAL: LOAD CRITICAL - total load average: 14.05, 10.88, 9.33 [00:01:46] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 11.39, 11.15, 9.18 [00:02:52] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.83, 6.54, 6.46 [00:02:58] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.71, 6.94, 6.49 [00:03:13] PROBLEM - mw132 Current Load on mw132 is CRITICAL: LOAD CRITICAL - total load average: 13.98, 10.91, 8.87 [00:03:17] PROBLEM - cloud11 IPMI Sensors on cloud11 is CRITICAL: IPMI Status: Critical [Cntlr 2 Bay 8 = Critical] [00:03:43] PROBLEM - mw133 Current Load on mw133 is CRITICAL: LOAD CRITICAL - total load average: 12.57, 11.36, 9.47 [00:04:57] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.23, 7.24, 6.65 [00:06:14] PROBLEM - mw132 HTTPS on mw132 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10000 milliseconds with 0 bytes received [00:06:49] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.10, 6.45, 6.47 [00:06:55] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 6.05, 6.68, 6.51 [00:07:12] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 10.93, 11.82, 9.71 [00:08:12] PROBLEM - mw132 HTTPS on mw132 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.940 second response time [00:08:12] PROBLEM - os131 Current Load on os131 is CRITICAL: LOAD CRITICAL - total load average: 6.87, 4.34, 2.37 [00:08:42] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 4.63, 3.64, 2.71 [00:09:12] PROBLEM - mw132 Current Load on mw132 is CRITICAL: LOAD CRITICAL - total load average: 13.00, 12.15, 10.07 [00:09:24] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 10.34, 11.70, 10.47 [00:10:47] PROBLEM - mw131 HTTPS on mw131 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [00:11:11] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 11.94, 11.43, 10.02 [00:12:42] PROBLEM - mw131 HTTPS on mw131 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.566 second response time [00:12:52] PROBLEM - mw131 Current Load on mw131 is CRITICAL: LOAD CRITICAL - total load average: 12.37, 12.29, 10.30 [00:13:20] PROBLEM - mw134 Current Load on mw134 is CRITICAL: LOAD CRITICAL - total load average: 12.23, 10.85, 10.32 [00:14:35] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 1.88, 3.30, 2.93 [00:14:50] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 9.46, 11.45, 10.24 [00:15:18] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 9.99, 11.07, 10.51 [00:17:09] RECOVERY - mw132 Current Load on mw132 is OK: LOAD OK - total load average: 6.98, 10.00, 10.00 [00:17:19] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 7.36, 10.77, 10.78 [00:19:16] RECOVERY - mw134 Current Load on mw134 is OK: LOAD OK - total load average: 8.89, 9.48, 9.99 [00:20:10] PROBLEM - os131 Current Load on os131 is WARNING: LOAD WARNING - total load average: 2.77, 3.60, 3.30 [00:20:37] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.73, 7.07, 6.57 [00:20:44] RECOVERY - mw131 Current Load on mw131 is OK: LOAD OK - total load average: 7.61, 9.33, 9.66 [00:21:12] RECOVERY - mw133 Current Load on mw133 is OK: LOAD OK - total load average: 7.71, 9.08, 10.09 [00:22:07] RECOVERY - os131 Current Load on os131 is OK: LOAD OK - total load average: 3.09, 3.33, 3.23 [00:22:35] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.88, 6.50, 6.41 [00:28:30] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.23, 6.91, 6.62 [00:29:05] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 10.39, 9.81, 9.51 [00:29:37] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.37, 6.91, 6.42 [00:30:28] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 4.68, 6.11, 6.37 [00:31:05] RECOVERY - mw132 Current Load on mw132 is OK: LOAD OK - total load average: 9.72, 9.35, 9.35 [00:33:20] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.45, 2.98, 2.56 [00:35:20] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 2.25, 2.67, 2.50 [00:35:33] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 5.42, 6.16, 6.26 [00:39:19] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.10, 3.46, 2.85 [00:41:17] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 3.12, 3.28, 2.85 [00:42:15] PROBLEM - os131 Current Load on os131 is CRITICAL: LOAD CRITICAL - total load average: 8.61, 5.84, 4.14 [00:42:45] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 4.98, 3.67, 3.05 [00:43:38] [02miraheze/dns] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/dns/commit/b120853881fa [00:43:40] [02miraheze/dns] 07paladox 03b120853 - Add ipv6 for cloud16/17 [00:43:42] [02dns] 07paladox created branch 03paladox-patch-1 - 13https://github.com/miraheze/dns [00:43:44] [02dns] 07paladox opened pull request 03#463: Add ipv6 for cloud16/17 - 13https://github.com/miraheze/dns/pull/463 [00:43:48] [02dns] 07paladox closed pull request 03#463: Add ipv6 for cloud16/17 - 13https://github.com/miraheze/dns/pull/463 [00:43:49] [02miraheze/dns] 07paladox deleted branch 03paladox-patch-1 [00:43:50] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/ea6efbb7f72e...a4c74e99232d [00:43:51] [02miraheze/dns] 07paladox 03a4c74e9 - Add ipv6 for cloud16/17 (#463) [00:43:53] [02dns] 07paladox deleted branch 03paladox-patch-1 - 13https://github.com/miraheze/dns [00:45:19] PROBLEM - mw131 Current Load on mw131 is CRITICAL: LOAD CRITICAL - total load average: 12.58, 10.67, 9.66 [00:47:17] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 11.48, 10.76, 9.80 [00:47:19] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 8.04, 10.25, 9.80 [00:47:36] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 11.87, 11.41, 10.39 [00:49:00] PROBLEM - mw133 HTTPS on mw133 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [00:49:33] PROBLEM - mw133 Current Load on mw133 is CRITICAL: LOAD CRITICAL - total load average: 16.44, 12.80, 10.99 [00:50:47] PROBLEM - mw131 HTTPS on mw131 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [00:51:00] PROBLEM - mw133 HTTPS on mw133 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 4.684 second response time [00:51:09] PROBLEM - mw131 Current Load on mw131 is CRITICAL: LOAD CRITICAL - total load average: 16.61, 13.31, 11.04 [00:51:49] PROBLEM - mw134 Current Load on mw134 is CRITICAL: LOAD CRITICAL - total load average: 14.36, 12.78, 11.01 [00:52:37] PROBLEM - cloud16 Puppet on cloud16 is UNKNOWN: NRPE: Unable to read output [00:52:39] PROBLEM - mw132 Current Load on mw132 is CRITICAL: LOAD CRITICAL - total load average: 14.23, 12.37, 10.56 [00:52:41] PROBLEM - cloud16 IPMI Sensors on cloud16 is CRITICAL: IPMI Status: Critical [4 system event log (SEL) entries present] [00:52:42] PROBLEM - mw131 HTTPS on mw131 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.594 second response time [00:53:05] PROBLEM - cloud17 ferm_active on cloud17 is UNKNOWN: NRPE: Unable to read output [00:53:08] PROBLEM - cloud17 Puppet on cloud17 is UNKNOWN: NRPE: Unable to read output [00:53:12] PROBLEM - cloud17 IPMI Sensors on cloud17 is CRITICAL: IPMI Status: Critical [12 system event log (SEL) entries present] [00:53:19] PROBLEM - cloud17 conntrack_table_size on cloud17 is UNKNOWN: NRPE: Unable to read output [00:53:47] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 10.37, 11.78, 10.85 [00:53:53] PROBLEM - mw141 HTTPS on mw141 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [00:54:36] RECOVERY - cloud16 Puppet on cloud16 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [00:54:39] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 10.73, 11.48, 10.44 [00:55:01] RECOVERY - cloud17 ferm_active on cloud17 is OK: OK ferm input default policy is set [00:55:07] RECOVERY - cloud17 Puppet on cloud17 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [00:55:17] RECOVERY - cloud17 conntrack_table_size on cloud17 is OK: OK: nf_conntrack is 0 % full [00:55:23] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.40, 3.61, 3.24 [00:55:53] PROBLEM - mw141 HTTPS on mw141 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.522 second response time [00:56:19] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 1.93, 3.36, 3.49 [00:57:08] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 10.27, 11.39, 10.96 [00:57:22] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.15, 3.78, 3.35 [00:58:18] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 2.82, 3.12, 3.38 [00:58:39] RECOVERY - mw132 Current Load on mw132 is OK: LOAD OK - total load average: 7.82, 9.59, 9.91 [01:02:12] PROBLEM - ping6 on cloud16 is CRITICAL: PING CRITICAL - Packet loss = 90%, RTA = 0.13 ms [01:02:21] PROBLEM - ping6 on cloud17 is CRITICAL: PING CRITICAL - Packet loss = 80%, RTA = 278.52 ms [01:02:38] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 9.52, 11.51, 11.82 [01:03:35] PROBLEM - mw134 Current Load on mw134 is CRITICAL: LOAD CRITICAL - total load average: 12.01, 11.08, 10.77 [01:04:14] RECOVERY - ping6 on cloud16 is OK: PING OK - Packet loss = 0%, RTA = 0.17 ms [01:04:23] RECOVERY - ping6 on cloud17 is OK: PING OK - Packet loss = 0%, RTA = 0.16 ms [01:04:39] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 11.70, 10.45, 10.12 [01:05:33] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 11.54, 11.12, 10.81 [01:06:05] PROBLEM - os131 Current Load on os131 is WARNING: LOAD WARNING - total load average: 2.82, 2.99, 3.86 [01:06:39] RECOVERY - mw132 Current Load on mw132 is OK: LOAD OK - total load average: 8.48, 9.71, 9.90 [01:08:03] PROBLEM - mw141 HTTPS on mw141 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [01:09:50] PROBLEM - mw143 HTTPS on mw143 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [01:10:03] PROBLEM - mw141 HTTPS on mw141 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.523 second response time [01:10:19] RECOVERY - mw133 Current Load on mw133 is OK: LOAD OK - total load average: 6.07, 8.10, 10.09 [01:11:26] RECOVERY - mw134 Current Load on mw134 is OK: LOAD OK - total load average: 8.62, 9.36, 10.07 [01:11:46] PROBLEM - mw143 HTTPS on mw143 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.544 second response time [01:12:04] PROBLEM - mw143 Current Load on mw143 is WARNING: LOAD WARNING - total load average: 11.35, 9.48, 8.22 [01:12:05] RECOVERY - os131 Current Load on os131 is OK: LOAD OK - total load average: 3.11, 2.60, 3.36 [01:14:00] RECOVERY - mw143 Current Load on mw143 is OK: LOAD OK - total load average: 7.62, 8.72, 8.10 [01:16:16] [02miraheze/mw-config] 07OAuthority pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/f3391f1bb169...43618efd8238 [01:16:18] [02miraheze/mw-config] 07OAuthority 0343618ef - add drafts to managewiki [01:17:05] RECOVERY - mw131 Current Load on mw131 is OK: LOAD OK - total load average: 7.08, 8.66, 9.87 [01:17:13] miraheze/mw-config - OAuthority the build passed. [01:18:20] !log [paladox@mwtask181] starting deploy of {'files': '../mediawiki/1.40/extensions/SemanticMediaWiki/.smw.json'} to all [01:18:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:18:55] [02miraheze/mw-config] 07OAuthority pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/43618efd8238...94f055eb8614 [01:18:56] [02miraheze/mw-config] 07OAuthority 0394f055e - add default drafts configuration [01:19:39] PROBLEM - mw141 Current Load on mw141 is CRITICAL: LOAD CRITICAL - total load average: 15.07, 10.53, 9.02 [01:19:52] miraheze/mw-config - OAuthority the build passed. [01:20:55] PROBLEM - cp24 HTTPS on cp24 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [01:21:06] PROBLEM - cp35 HTTPS on cp35 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [01:21:46] PROBLEM - mw143 Current Load on mw143 is CRITICAL: LOAD CRITICAL - total load average: 16.41, 12.00, 9.50 [01:21:59] !log [paladox@mwtask181] finished deploy of {'files': '../mediawiki/1.40/extensions/SemanticMediaWiki/.smw.json'} to all - SUCCESS in 219s [01:22:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:22:52] PROBLEM - cp24 HTTPS on cp24 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.595 second response time [01:23:06] PROBLEM - cp35 HTTPS on cp35 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.632 second response time [01:23:33] PROBLEM - mw141 Current Load on mw141 is WARNING: LOAD WARNING - total load average: 10.55, 11.39, 9.78 [01:23:42] PROBLEM - mw143 Current Load on mw143 is WARNING: LOAD WARNING - total load average: 11.31, 11.77, 9.72 [01:25:30] RECOVERY - mw141 Current Load on mw141 is OK: LOAD OK - total load average: 5.88, 9.45, 9.26 [01:25:38] RECOVERY - mw143 Current Load on mw143 is OK: LOAD OK - total load average: 7.50, 10.06, 9.32 [01:29:23] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 2.81, 3.38, 3.97 [01:31:42] !log [@mwtask181] starting deploy of {'config': True} to all [01:31:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:32:55] !log [@mwtask181] finished deploy of {'config': True} to all - SUCCESS in 73s [01:33:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:35:23] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.13, 3.35, 3.71 [01:37:21] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.57, 3.34, 3.66 [01:39:22] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 6.59, 4.38, 3.99 [01:40:10] RECOVERY - cloud11 IPMI Sensors on cloud11 is OK: IPMI Status: OK [01:42:08] PROBLEM - cp25 HTTPS on cp25 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [01:44:05] PROBLEM - cloud11 IPMI Sensors on cloud11 is CRITICAL: IPMI Status: Critical [Cntlr 2 Bay 8 = Critical] [01:44:08] PROBLEM - cp25 HTTPS on cp25 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.593 second response time [01:48:39] PROBLEM - db151 ferm_active on db151 is CRITICAL: connect to address fe80::be24:11ff:feef:2e52 port 5666: Invalid argumentconnect to host fe80::be24:11ff:feef:2e52 port 5666: Invalid argument [01:48:44] PROBLEM - db151 NTP time on db151 is CRITICAL: connect to address fe80::be24:11ff:feef:2e52 port 5666: Invalid argumentconnect to host fe80::be24:11ff:feef:2e52 port 5666: Invalid argument [01:48:46] PROBLEM - db151 Disk Space on db151 is CRITICAL: connect to address fe80::be24:11ff:feef:2e52 port 5666: Invalid argumentconnect to host fe80::be24:11ff:feef:2e52 port 5666: Invalid argument [01:48:50] PROBLEM - db151 Backups SQL mhglobal on db151 is CRITICAL: connect to address fe80::be24:11ff:feef:2e52 port 5666: Invalid argumentconnect to host fe80::be24:11ff:feef:2e52 port 5666: Invalid argument [01:48:51] PROBLEM - Host db151 is DOWN: CRITICAL - Host Unreachable (fe80::be24:11ff:feef:2e52) [01:48:59] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.79, 3.91, 3.99 [01:51:00] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/90d5f57d4946 [01:51:02] [02miraheze/puppet] 07paladox 0390d5f57 - base: support private ip [01:51:03] [02puppet] 07paladox created branch 03paladox-patch-12 - 13https://github.com/miraheze/puppet [01:51:05] [02puppet] 07paladox opened pull request 03#3670: base: support private ip - 13https://github.com/miraheze/puppet/pull/3670 [01:54:52] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 6.13, 4.70, 4.24 [01:55:48] RECOVERY - cloud11 IPMI Sensors on cloud11 is OK: IPMI Status: OK [01:56:48] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 2.58, 3.88, 4.00 [01:56:51] [02puppet] 07paladox closed pull request 03#3670: base: support private ip - 13https://github.com/miraheze/puppet/pull/3670 [01:56:54] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/bf9d5240be87...26a3440c8e9f [01:56:57] [02miraheze/puppet] 07paladox 0326a3440 - base: support private ip (#3670) [01:57:00] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-12 [01:57:03] [02puppet] 07paladox deleted branch 03paladox-patch-12 - 13https://github.com/miraheze/puppet [01:57:49] PROBLEM - mw152 MediaWiki Rendering on mw152 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:58:21] PROBLEM - mw151 MediaWiki Rendering on mw151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:59:43] PROBLEM - cloud11 IPMI Sensors on cloud11 is CRITICAL: IPMI Status: Critical [Cntlr 2 Bay 8 = Critical] [02:01:42] PROBLEM - mw142 Puppet on mw142 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [02:02:36] PROBLEM - cloud16 Puppet on cloud16 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [02:03:05] PROBLEM - matomo151 HTTPS on matomo151 is CRITICAL: HTTP CRITICAL: HTTP/2 502 - 176 bytes in 0.006 second response time [02:03:39] PROBLEM - db131 Puppet on db131 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [02:06:35] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 1.50, 2.20, 3.15 [02:07:14] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/9ddec7028e9b [02:07:17] [02miraheze/puppet] 07paladox 039ddec70 - base: fix firewall [02:07:20] [02puppet] 07paladox created branch 03paladox-patch-12 - 13https://github.com/miraheze/puppet [02:07:23] [02puppet] 07paladox opened pull request 03#3671: base: fix firewall - 13https://github.com/miraheze/puppet/pull/3671 [02:07:58] [02puppet] 07paladox closed pull request 03#3671: base: fix firewall - 13https://github.com/miraheze/puppet/pull/3671 [02:08:01] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-12 [02:08:03] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/26a3440c8e9f...37b44de95623 [02:08:04] [02miraheze/puppet] 07paladox 0337b44de - base: fix firewall (#3671) [02:08:05] [02puppet] 07paladox deleted branch 03paladox-patch-12 - 13https://github.com/miraheze/puppet [02:09:07] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/37b44de95623...fbfc00a7480f [02:09:10] [02miraheze/puppet] 07paladox 03fbfc00a - Fix typo [02:10:18] RECOVERY - cloud16 Puppet on cloud16 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [02:10:39] PROBLEM - cloud12 Puppet on cloud12 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [02:10:56] PROBLEM - phab121 Puppet on phab121 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [02:12:23] PROBLEM - cloud10 Puppet on cloud10 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [02:15:06] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/f5cc5f5e6aec [02:15:08] [02miraheze/puppet] 07paladox 03f5cc5f5 - prometheus: add support for private network in firewall [02:15:11] [02puppet] 07paladox created branch 03paladox-patch-12 - 13https://github.com/miraheze/puppet [02:15:14] [02puppet] 07paladox opened pull request 03#3672: prometheus: add support for private network in firewall - 13https://github.com/miraheze/puppet/pull/3672 [02:15:34] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/f5cc5f5e6aec...9593765105e3 [02:15:36] [02miraheze/puppet] 07paladox 039593765 - Update cadvisor.pp [02:15:38] [02puppet] 07paladox synchronize pull request 03#3672: prometheus: add support for private network in firewall - 13https://github.com/miraheze/puppet/pull/3672 [02:15:44] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/9593765105e3...bedcdcd53109 [02:15:45] RECOVERY - cloud12 Puppet on cloud12 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:15:47] [02miraheze/puppet] 07paladox 03bedcdcd - Update elasticsearch.pp [02:15:51] [02puppet] 07paladox synchronize pull request 03#3672: prometheus: add support for private network in firewall - 13https://github.com/miraheze/puppet/pull/3672 [02:15:57] RECOVERY - cloud10 Puppet on cloud10 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:16:02] PROBLEM - mw152 HTTPS on mw152 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 7 - Failed to connect to mw152.wikitide.net port 443 after 0 ms: Couldn't connect to server [02:16:06] PROBLEM - mw151 HTTPS on mw151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 7 - Failed to connect to mw151.wikitide.net port 443 after 0 ms: Couldn't connect to server [02:16:08] PROBLEM - test151 MediaWiki Rendering on test151 is CRITICAL: connect to address fe80::be24:11ff:fed9:92ca and port 443: Invalid argumentHTTP CRITICAL - Unable to open TCP socket [02:16:21] RECOVERY - db131 Puppet on db131 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [02:16:39] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 4.18, 3.16, 2.61 [02:16:40] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 10.89, 9.02, 8.02 [02:16:51] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/bedcdcd53109...e9b8405779b6 [02:16:54] [02miraheze/puppet] 07paladox 03e9b8405 - Update fpm.pp [02:16:57] [02puppet] 07paladox synchronize pull request 03#3672: prometheus: add support for private network in firewall - 13https://github.com/miraheze/puppet/pull/3672 [02:16:57] PROBLEM - mw142 Current Load on mw142 is WARNING: LOAD WARNING - total load average: 11.47, 9.62, 7.61 [02:16:58] RECOVERY - phab121 Puppet on phab121 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [02:17:13] RECOVERY - mw142 Puppet on mw142 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [02:17:14] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/e9b8405779b6...67b14a932bd6 [02:17:15] [02miraheze/puppet] 07paladox 0367b14a9 - Update jmx.pp [02:17:16] [02puppet] 07paladox synchronize pull request 03#3672: prometheus: add support for private network in firewall - 13https://github.com/miraheze/puppet/pull/3672 [02:17:24] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/67b14a932bd6...daaeb2f8a5af [02:17:27] [02miraheze/puppet] 07paladox 03daaeb2f - Update mariadb.pp [02:17:29] [02puppet] 07paladox synchronize pull request 03#3672: prometheus: add support for private network in firewall - 13https://github.com/miraheze/puppet/pull/3672 [02:17:32] PROBLEM - test151 HTTPS on test151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 7 - Failed to connect to test151.wikitide.net port 443 after 0 ms: Couldn't connect to server [02:17:34] PROBLEM - os151 APT on os151 is CRITICAL: connect to address fe80::be24:11ff:feb5:541b port 5666: Invalid argumentconnect to host fe80::be24:11ff:feb5:541b port 5666: Invalid argument [02:17:36] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/daaeb2f8a5af...c584677234de [02:17:39] [02miraheze/puppet] 07paladox 03c584677 - Update memcached.pp [02:17:40] [02puppet] 07paladox synchronize pull request 03#3672: prometheus: add support for private network in firewall - 13https://github.com/miraheze/puppet/pull/3672 [02:17:42] PROBLEM - swiftobject151 NTP time on swiftobject151 is CRITICAL: connect to address fe80::be24:11ff:fe5f:f7ba port 5666: Invalid argumentconnect to host fe80::be24:11ff:fe5f:f7ba port 5666: Invalid argument [02:17:44] PROBLEM - matomo151 Current Load on matomo151 is CRITICAL: connect to address fe80::be24:11ff:fe07:19a4 port 5666: Invalid argumentconnect to host fe80::be24:11ff:fe07:19a4 port 5666: Invalid argument [02:17:45] PROBLEM - mw152 PowerDNS Recursor on mw152 is CRITICAL: connect to address fe80::be24:11ff:feff:d7aa port 5666: Invalid argumentconnect to host fe80::be24:11ff:feff:d7aa port 5666: Invalid argument [02:17:46] PROBLEM - mw151 Current Load on mw151 is CRITICAL: connect to address fe80::be24:11ff:fe4e:5aae port 5666: Invalid argumentconnect to host fe80::be24:11ff:fe4e:5aae port 5666: Invalid argument [02:17:47] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/c584677234de...9e74948941d3 [02:17:48] PROBLEM - matomo151 PowerDNS Recursor on matomo151 is CRITICAL: connect to address fe80::be24:11ff:fe07:19a4 port 5666: Invalid argumentconnect to host fe80::be24:11ff:fe07:19a4 port 5666: Invalid argument [02:17:48] PROBLEM - test151 php-fpm on test151 is CRITICAL: connect to address fe80::be24:11ff:fed9:92ca port 5666: Invalid argumentconnect to host fe80::be24:11ff:fed9:92ca port 5666: Invalid argument [02:17:49] PROBLEM - Host matomo151 is DOWN: CRITICAL - Host Unreachable (fe80::be24:11ff:fe07:19a4) [02:17:49] PROBLEM - swiftobject151 Puppet on swiftobject151 is CRITICAL: connect to address fe80::be24:11ff:fe5f:f7ba port 5666: Invalid argumentconnect to host fe80::be24:11ff:fe5f:f7ba port 5666: Invalid argument [02:17:49] [02miraheze/puppet] 07paladox 039e74948 - Update nginx.pp [02:17:50] PROBLEM - mem151 APT on mem151 is CRITICAL: connect to address fe80::be24:11ff:fee1:68b9 port 5666: Invalid argumentconnect to host fe80::be24:11ff:fee1:68b9 port 5666: Invalid argument [02:17:50] PROBLEM - mw152 conntrack_table_size on mw152 is CRITICAL: connect to address fe80::be24:11ff:feff:d7aa port 5666: Invalid argumentconnect to host fe80::be24:11ff:feff:d7aa port 5666: Invalid argument [02:17:51] [02puppet] 07paladox synchronize pull request 03#3672: prometheus: add support for private network in firewall - 13https://github.com/miraheze/puppet/pull/3672 [02:17:51] PROBLEM - test151 PowerDNS Recursor on test151 is CRITICAL: connect to address fe80::be24:11ff:fed9:92ca port 5666: Invalid argumentconnect to host fe80::be24:11ff:fed9:92ca port 5666: Invalid argument [02:17:52] PROBLEM - swiftobject151 Disk Space on swiftobject151 is CRITICAL: connect to address fe80::be24:11ff:fe5f:f7ba port 5666: Invalid argumentconnect to host fe80::be24:11ff:fe5f:f7ba port 5666: Invalid argument [02:17:54] PROBLEM - mw151 NTP time on mw151 is CRITICAL: connect to address fe80::be24:11ff:fe4e:5aae port 5666: Invalid argumentconnect to host fe80::be24:11ff:fe4e:5aae port 5666: Invalid argument [02:17:57] PROBLEM - prometheus151 APT on prometheus151 is CRITICAL: connect to address fe80::be24:11ff:fe43:e540 port 5666: Invalid argumentconnect to host fe80::be24:11ff:fe43:e540 port 5666: Invalid argument [02:17:58] PROBLEM - mw151 Puppet on mw151 is CRITICAL: connect to address fe80::be24:11ff:fe4e:5aae port 5666: Invalid argumentconnect to host fe80::be24:11ff:fe4e:5aae port 5666: Invalid argument [02:17:58] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/9e74948941d3...7c0673902528 [02:18:01] [02miraheze/puppet] 07paladox 037c06739 - Update node.pp [02:18:02] [02puppet] 07paladox synchronize pull request 03#3672: prometheus: add support for private network in firewall - 13https://github.com/miraheze/puppet/pull/3672 [02:18:03] PROBLEM - prometheus151 Prometheus on prometheus151 is CRITICAL: connect to address fe80::be24:11ff:fe43:e540 and port 9090: Invalid argument [02:18:03] PROBLEM - mw133 Current Load on mw133 is CRITICAL: LOAD CRITICAL - total load average: 12.30, 11.61, 9.92 [02:18:03] PROBLEM - test151 Redis Process on test151 is CRITICAL: connect to address fe80::be24:11ff:fed9:92ca port 5666: Invalid argumentconnect to host fe80::be24:11ff:fed9:92ca port 5666: Invalid argument [02:18:05] PROBLEM - prometheus151 conntrack_table_size on prometheus151 is CRITICAL: connect to address fe80::be24:11ff:fe43:e540 port 5666: Invalid argumentconnect to host fe80::be24:11ff:fe43:e540 port 5666: Invalid argument [02:18:05] PROBLEM - mem151 ferm_active on mem151 is CRITICAL: connect to address fe80::be24:11ff:fee1:68b9 port 5666: Invalid argumentconnect to host fe80::be24:11ff:fee1:68b9 port 5666: Invalid argument [02:18:07] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/7c0673902528...eb31003f42c0 [02:18:07] PROBLEM - mem151 memcached on mem151 is CRITICAL: connect to address fe80::be24:11ff:fee1:68b9 and port 11211: Invalid argument [02:18:10] PROBLEM - mem151 conntrack_table_size on mem151 is CRITICAL: connect to address fe80::be24:11ff:fee1:68b9 port 5666: Invalid argumentconnect to host fe80::be24:11ff:fee1:68b9 port 5666: Invalid argument [02:18:10] [02miraheze/puppet] 07paladox 03eb31003 - Update openldap.pp [02:18:10] PROBLEM - mem151 NTP time on mem151 is CRITICAL: connect to address fe80::be24:11ff:fee1:68b9 port 5666: Invalid argumentconnect to host fe80::be24:11ff:fee1:68b9 port 5666: Invalid argument [02:18:11] PROBLEM - mw151 PowerDNS Recursor on mw151 is CRITICAL: connect to address fe80::be24:11ff:fe4e:5aae port 5666: Invalid argumentconnect to host fe80::be24:11ff:fe4e:5aae port 5666: Invalid argument [02:18:11] PROBLEM - prometheus151 NTP time on prometheus151 is CRITICAL: connect to address fe80::be24:11ff:fe43:e540 port 5666: Invalid argumentconnect to host fe80::be24:11ff:fe43:e540 port 5666: Invalid argument [02:18:12] PROBLEM - swiftobject151 ferm_active on swiftobject151 is CRITICAL: connect to address fe80::be24:11ff:fe5f:f7ba port 5666: Invalid argumentconnect to host fe80::be24:11ff:fe5f:f7ba port 5666: Invalid argument [02:18:12] [02puppet] 07paladox synchronize pull request 03#3672: prometheus: add support for private network in firewall - 13https://github.com/miraheze/puppet/pull/3672 [02:18:12] PROBLEM - mw151 php-fpm on mw151 is CRITICAL: connect to address fe80::be24:11ff:fe4e:5aae port 5666: Invalid argumentconnect to host fe80::be24:11ff:fe4e:5aae port 5666: Invalid argument [02:18:13] PROBLEM - Host prometheus151 is DOWN: CRITICAL - Host Unreachable (fe80::be24:11ff:fe43:e540) [02:18:14] PROBLEM - mem151 PowerDNS Recursor on mem151 is CRITICAL: connect to address fe80::be24:11ff:fee1:68b9 port 5666: Invalid argumentconnect to host fe80::be24:11ff:fee1:68b9 port 5666: Invalid argument [02:18:14] PROBLEM - test151 SSH on test151 is CRITICAL: connect to address fe80::be24:11ff:fed9:92ca and port 22: Invalid argument [02:18:14] PROBLEM - mw152 php-fpm on mw152 is CRITICAL: connect to address fe80::be24:11ff:feff:d7aa port 5666: Invalid argumentconnect to host fe80::be24:11ff:feff:d7aa port 5666: Invalid argument [02:18:14] PROBLEM - ping6 on mw151 is CRITICAL: CRITICAL - Host Unreachable (fe80::be24:11ff:fe4e:5aae) [02:18:16] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/eb31003f42c0...7a1ae5bb90c6 [02:18:17] [02miraheze/puppet] 07paladox 037a1ae5b - Update postfix.pp [02:18:17] PROBLEM - os151 Disk Space on os151 is CRITICAL: connect to address fe80::be24:11ff:feb5:541b port 5666: Invalid argumentconnect to host fe80::be24:11ff:feb5:541b port 5666: Invalid argument [02:18:18] PROBLEM - swiftobject151 SSH on swiftobject151 is CRITICAL: connect to address fe80::be24:11ff:fe5f:f7ba and port 22: Invalid argument [02:18:19] PROBLEM - ping6 on mem151 is CRITICAL: CRITICAL - Host Unreachable (fe80::be24:11ff:fee1:68b9) [02:18:20] [02puppet] 07paladox synchronize pull request 03#3672: prometheus: add support for private network in firewall - 13https://github.com/miraheze/puppet/pull/3672 [02:18:20] PROBLEM - os151 ferm_active on os151 is CRITICAL: connect to address fe80::be24:11ff:feb5:541b port 5666: Invalid argumentconnect to host fe80::be24:11ff:feb5:541b port 5666: Invalid argument [02:18:23] PROBLEM - mem151 Disk Space on mem151 is CRITICAL: connect to address fe80::be24:11ff:fee1:68b9 port 5666: Invalid argumentconnect to host fe80::be24:11ff:fee1:68b9 port 5666: Invalid argument [02:18:24] PROBLEM - ping6 on os151 is CRITICAL: CRITICAL - Host Unreachable (fe80::be24:11ff:feb5:541b) [02:18:25] PROBLEM - mem151 Puppet on mem151 is CRITICAL: connect to address fe80::be24:11ff:fee1:68b9 port 5666: Invalid argumentconnect to host fe80::be24:11ff:fee1:68b9 port 5666: Invalid argument [02:18:25] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/7a1ae5bb90c6...2c8cf73ca5d2 [02:18:25] PROBLEM - test151 JobRunner Service on test151 is CRITICAL: connect to address fe80::be24:11ff:fed9:92ca port 5666: Invalid argumentconnect to host fe80::be24:11ff:fed9:92ca port 5666: Invalid argument [02:18:26] PROBLEM - mw152 Puppet on mw152 is CRITICAL: connect to address fe80::be24:11ff:feff:d7aa port 5666: Invalid argumentconnect to host fe80::be24:11ff:feff:d7aa port 5666: Invalid argument [02:18:27] [02miraheze/puppet] 07paladox 032c8cf73 - Update redis.pp [02:18:28] PROBLEM - os151 conntrack_table_size on os151 is CRITICAL: connect to address fe80::be24:11ff:feb5:541b port 5666: Invalid argumentconnect to host fe80::be24:11ff:feb5:541b port 5666: Invalid argument [02:18:29] PROBLEM - test151 Disk Space on test151 is CRITICAL: connect to address fe80::be24:11ff:fed9:92ca port 5666: Invalid argumentconnect to host fe80::be24:11ff:fed9:92ca port 5666: Invalid argument [02:18:30] [02puppet] 07paladox synchronize pull request 03#3672: prometheus: add support for private network in firewall - 13https://github.com/miraheze/puppet/pull/3672 [02:18:34] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 2.34, 2.99, 2.62 [02:18:35] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/2c8cf73ca5d2...4444edad5d6e [02:18:37] PROBLEM - mw152 SSH on mw152 is CRITICAL: connect to address fe80::be24:11ff:feff:d7aa and port 22: Invalid argument [02:18:38] [02miraheze/puppet] 07paladox 034444eda - Update statsd_exporter.pp [02:18:38] PROBLEM - os151 PowerDNS Recursor on os151 is CRITICAL: connect to address fe80::be24:11ff:feb5:541b port 5666: Invalid argumentconnect to host fe80::be24:11ff:feb5:541b port 5666: Invalid argument [02:18:39] RECOVERY - mw132 Current Load on mw132 is OK: LOAD OK - total load average: 8.10, 8.71, 8.03 [02:18:40] [02puppet] 07paladox synchronize pull request 03#3672: prometheus: add support for private network in firewall - 13https://github.com/miraheze/puppet/pull/3672 [02:18:42] PROBLEM - Host os151 is DOWN: CRITICAL - Host Unreachable (fe80::be24:11ff:feb5:541b) [02:18:43] PROBLEM - swiftobject151 APT on swiftobject151 is CRITICAL: connect to address fe80::be24:11ff:fe5f:f7ba port 5666: Invalid argumentconnect to host fe80::be24:11ff:fe5f:f7ba port 5666: Invalid argument [02:18:45] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/4444edad5d6e...c476d4cde909 [02:18:46] PROBLEM - test151 ferm_active on test151 is CRITICAL: connect to address fe80::be24:11ff:fed9:92ca port 5666: Invalid argumentconnect to host fe80::be24:11ff:fed9:92ca port 5666: Invalid argument [02:18:47] [02miraheze/puppet] 07paladox 03c476d4c - Update varnish.pp [02:18:50] [02puppet] 07paladox synchronize pull request 03#3672: prometheus: add support for private network in firewall - 13https://github.com/miraheze/puppet/pull/3672 [02:18:50] PROBLEM - test151 JobChron Service on test151 is CRITICAL: connect to address fe80::be24:11ff:fed9:92ca port 5666: Invalid argumentconnect to host fe80::be24:11ff:fed9:92ca port 5666: Invalid argument [02:18:51] PROBLEM - swiftobject151 Swift Object Service on swiftobject151 is CRITICAL: connect to address fe80::be24:11ff:fe5f:f7ba and port 6000: Invalid argument [02:18:52] PROBLEM - mw152 Current Load on mw152 is CRITICAL: connect to address fe80::be24:11ff:feff:d7aa port 5666: Invalid argumentconnect to host fe80::be24:11ff:feff:d7aa port 5666: Invalid argument [02:18:55] RECOVERY - mw142 Current Load on mw142 is OK: LOAD OK - total load average: 6.32, 8.45, 7.44 [02:18:57] PROBLEM - test151 NTP time on test151 is CRITICAL: connect to address fe80::be24:11ff:fed9:92ca port 5666: Invalid argumentconnect to host fe80::be24:11ff:fed9:92ca port 5666: Invalid argument [02:19:00] PROBLEM - mw151 Disk Space on mw151 is CRITICAL: connect to address fe80::be24:11ff:fe4e:5aae port 5666: Invalid argumentconnect to host fe80::be24:11ff:fe4e:5aae port 5666: Invalid argument [02:19:03] PROBLEM - mw151 ferm_active on mw151 is CRITICAL: connect to address fe80::be24:11ff:fe4e:5aae port 5666: Invalid argumentconnect to host fe80::be24:11ff:fe4e:5aae port 5666: Invalid argument [02:19:04] PROBLEM - test151 poolcounter process on test151 is CRITICAL: connect to address fe80::be24:11ff:fed9:92ca port 5666: Invalid argumentconnect to host fe80::be24:11ff:fed9:92ca port 5666: Invalid argument [02:19:04] PROBLEM - mw152 ferm_active on mw152 is CRITICAL: connect to address fe80::be24:11ff:feff:d7aa port 5666: Invalid argumentconnect to host fe80::be24:11ff:feff:d7aa port 5666: Invalid argument [02:19:04] PROBLEM - test151 conntrack_table_size on test151 is CRITICAL: connect to address fe80::be24:11ff:fed9:92ca port 5666: Invalid argumentconnect to host fe80::be24:11ff:fed9:92ca port 5666: Invalid argument [02:19:04] PROBLEM - Host mw151 is DOWN: CRITICAL - Host Unreachable (fe80::be24:11ff:fe4e:5aae) [02:19:04] PROBLEM - swiftobject151 Current Load on swiftobject151 is CRITICAL: connect to address fe80::be24:11ff:fe5f:f7ba port 5666: Invalid argumentconnect to host fe80::be24:11ff:fe5f:f7ba port 5666: Invalid argument [02:19:06] PROBLEM - mw152 Disk Space on mw152 is CRITICAL: connect to address fe80::be24:11ff:feff:d7aa port 5666: Invalid argumentconnect to host fe80::be24:11ff:feff:d7aa port 5666: Invalid argument [02:19:11] PROBLEM - test151 memcached on test151 is CRITICAL: connect to address fe80::be24:11ff:fed9:92ca and port 11211: Invalid argument [02:19:12] PROBLEM - ping6 on test151 is CRITICAL: CRITICAL - Host Unreachable (fe80::be24:11ff:fed9:92ca) [02:19:13] PROBLEM - test151 Current Load on test151 is CRITICAL: connect to address fe80::be24:11ff:fed9:92ca port 5666: Invalid argumentconnect to host fe80::be24:11ff:fed9:92ca port 5666: Invalid argument [02:19:13] [02puppet] 07paladox closed pull request 03#3672: prometheus: add support for private network in firewall - 13https://github.com/miraheze/puppet/pull/3672 [02:19:14] PROBLEM - mem151 Current Load on mem151 is CRITICAL: connect to address fe80::be24:11ff:fee1:68b9 port 5666: Invalid argumentconnect to host fe80::be24:11ff:fee1:68b9 port 5666: Invalid argument [02:19:15] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±14] 13https://github.com/miraheze/puppet/compare/fbfc00a7480f...b22d733baab8 [02:19:16] PROBLEM - ping6 on mw152 is CRITICAL: CRITICAL - Host Unreachable (fe80::be24:11ff:feff:d7aa) [02:19:16] PROBLEM - Host mw152 is DOWN: CRITICAL - Host Unreachable (fe80::be24:11ff:feff:d7aa) [02:19:17] [02miraheze/puppet] 07paladox 03b22d733 - prometheus: add support for private network in firewall (#3672) [02:19:19] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.72, 6.73, 5.76 [02:19:19] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-12 [02:19:20] [02puppet] 07paladox deleted branch 03paladox-patch-12 - 13https://github.com/miraheze/puppet [02:19:21] PROBLEM - swiftobject151 PowerDNS Recursor on swiftobject151 is CRITICAL: connect to address fe80::be24:11ff:fe5f:f7ba port 5666: Invalid argumentconnect to host fe80::be24:11ff:fe5f:f7ba port 5666: Invalid argument [02:19:23] PROBLEM - mem151 SSH on mem151 is CRITICAL: connect to address fe80::be24:11ff:fee1:68b9 and port 22: Invalid argument [02:19:25] PROBLEM - swiftobject151 conntrack_table_size on swiftobject151 is CRITICAL: connect to address fe80::be24:11ff:fe5f:f7ba port 5666: Invalid argumentconnect to host fe80::be24:11ff:fe5f:f7ba port 5666: Invalid argument [02:19:27] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.75, 6.95, 6.30 [02:19:28] PROBLEM - Host mem151 is DOWN: CRITICAL - Host Unreachable (fe80::be24:11ff:fee1:68b9) [02:19:31] PROBLEM - Host test151 is DOWN: CRITICAL - Host Unreachable (fe80::be24:11ff:fed9:92ca) [02:21:14] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 5.17, 6.13, 5.65 [02:21:26] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.79, 6.64, 6.27 [02:21:57] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 9.06, 11.29, 10.26 [02:22:30] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/513bda5819b1 [02:22:32] [02miraheze/puppet] 07paladox 03513bda5 - monitoring: fix ip [02:22:34] [02puppet] 07paladox created branch 03paladox-patch-12 - 13https://github.com/miraheze/puppet [02:22:36] [02puppet] 07paladox opened pull request 03#3673: monitoring: fix ip - 13https://github.com/miraheze/puppet/pull/3673 [02:22:50] [02puppet] 07paladox synchronize pull request 03#3673: monitoring: fix ip - 13https://github.com/miraheze/puppet/pull/3673 [02:22:52] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/513bda5819b1...f03e5a652d99 [02:22:53] [02miraheze/puppet] 07paladox 03f03e5a6 - Update hosts.pp [02:25:51] RECOVERY - mw133 Current Load on mw133 is OK: LOAD OK - total load average: 7.50, 9.64, 9.86 [02:25:55] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/f03e5a652d99...843f97b36e92 [02:25:57] [02miraheze/puppet] 07paladox 03843f97b - Update hosts.pp [02:25:58] [02puppet] 07paladox synchronize pull request 03#3673: monitoring: fix ip - 13https://github.com/miraheze/puppet/pull/3673 [02:27:06] [02puppet] 07paladox synchronize pull request 03#3673: monitoring: fix ip - 13https://github.com/miraheze/puppet/pull/3673 [02:27:07] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/843f97b36e92...6ebf20582966 [02:27:09] [02miraheze/puppet] 07paladox 036ebf205 - Update hosts.pp [02:27:13] [02puppet] 07paladox closed pull request 03#3673: monitoring: fix ip - 13https://github.com/miraheze/puppet/pull/3673 [02:27:14] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/b22d733baab8...929ba5942710 [02:27:15] [02miraheze/puppet] 07paladox 03929ba59 - monitoring: fix ip (#3673) [02:27:18] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-12 [02:27:20] [02puppet] 07paladox deleted branch 03paladox-patch-12 - 13https://github.com/miraheze/puppet [02:29:48] PROBLEM - mw133 Current Load on mw133 is CRITICAL: LOAD CRITICAL - total load average: 12.89, 11.43, 10.51 [02:31:30] PROBLEM - bast141 Puppet on bast141 is CRITICAL: CRITICAL: Puppet has 3 failures. Last run 2 minutes ago with 3 failures. Failed resources (up to 3 shown): File[/home/void],File[/home/macfan],File[/home/oa] [02:31:35] PROBLEM - mw133 Puppet on mw133 is CRITICAL: CRITICAL: Puppet has 43 failures. Last run 2 minutes ago with 43 failures. Failed resources (up to 3 shown): File[/opt/mcrouter_2023.07.17.00-1_amd64.deb],File[/usr/local/bin/cgroup-mediawiki-clean],File[/etc/ssl/localcerts],File[/etc/ssl/private] [02:31:45] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 9.81, 10.75, 10.38 [02:33:01] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.mahdiruiz.line.pm. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [02:33:15] RECOVERY - cloud11 IPMI Sensors on cloud11 is OK: IPMI Status: OK [02:33:40] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.80, 7.23, 6.50 [02:35:38] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 6.22, 6.71, 6.39 [02:36:24] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 10.31, 10.17, 9.02 [02:37:12] PROBLEM - cloud11 IPMI Sensors on cloud11 is CRITICAL: IPMI Status: Critical [Cntlr 2 Bay 8 = Critical] [02:37:36] RECOVERY - mw133 Current Load on mw133 is OK: LOAD OK - total load average: 7.20, 9.58, 10.11 [02:38:22] RECOVERY - mw132 Current Load on mw132 is OK: LOAD OK - total load average: 7.00, 8.98, 8.72 [02:39:55] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 5.88, 3.68, 2.32 [02:42:05] PROBLEM - mw143 HTTPS on mw143 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [02:44:03] PROBLEM - mw143 HTTPS on mw143 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.582 second response time [02:45:57] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 0.81, 3.19, 2.76 [02:49:16] PROBLEM - Host swiftobject151 is DOWN: CRITICAL - Network Unreachable (10.0.15.117) [02:59:35] RECOVERY - bast141 Puppet on bast141 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [03:00:58] RECOVERY - mw133 Puppet on mw133 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [03:15:10] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.andreijiroh.uk.eu.org All nameservers failed to answer the query. [03:22:34] PROBLEM - ping6 on cloud16 is CRITICAL: PING CRITICAL - Packet loss = 100% [03:23:48] PROBLEM - cloud16 SSH on cloud16 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:24:59] PROBLEM - cp25 HTTPS on cp25 is CRITICAL: HTTP CRITICAL: HTTP/2 503 - 3363 bytes in 0.472 second response time [03:25:06] PROBLEM - cloud16 APT on cloud16 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [03:25:12] PROBLEM - cp24 HTTPS on cp24 is CRITICAL: HTTP CRITICAL: HTTP/2 503 - 3363 bytes in 0.471 second response time [03:25:18] PROBLEM - cloud16 NTP time on cloud16 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [03:25:24] PROBLEM - Host cloud16 is DOWN: PING CRITICAL - Packet loss = 100% [03:25:30] PROBLEM - mw131 Current Load on mw131 is CRITICAL: LOAD CRITICAL - total load average: 13.17, 10.63, 9.51 [03:25:43] PROBLEM - mw133 Current Load on mw133 is CRITICAL: LOAD CRITICAL - total load average: 16.08, 11.20, 9.67 [03:25:49] PROBLEM - mw134 HTTPS on mw134 is CRITICAL: HTTP CRITICAL: HTTP/2 502 - 170 bytes in 0.374 second response time [03:25:49] PROBLEM - cp34 HTTPS on cp34 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [03:25:51] PROBLEM - mw141 HTTPS on mw141 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [03:26:39] PROBLEM - mw134 Current Load on mw134 is CRITICAL: LOAD CRITICAL - total load average: 20.36, 13.63, 10.71 [03:26:40] PROBLEM - mw143 Current Load on mw143 is CRITICAL: LOAD CRITICAL - total load average: 15.83, 10.87, 7.99 [03:26:58] PROBLEM - cp25 HTTPS on cp25 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.643 second response time [03:27:11] PROBLEM - cp24 HTTPS on cp24 is WARNING: HTTP WARNING: HTTP/2 404 - 3480 bytes in 0.653 second response time [03:27:28] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 9.94, 11.29, 9.95 [03:27:38] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 10.07, 11.26, 9.93 [03:27:46] PROBLEM - mw134 HTTPS on mw134 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.542 second response time [03:27:48] PROBLEM - cp34 HTTPS on cp34 is WARNING: HTTP WARNING: HTTP/2 404 - 3480 bytes in 0.587 second response time [03:27:50] PROBLEM - mw141 HTTPS on mw141 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.510 second response time [03:28:36] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 9.60, 11.51, 10.27 [03:28:39] RECOVERY - mw143 Current Load on mw143 is OK: LOAD OK - total load average: 7.13, 9.19, 7.72 [03:29:29] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 9.37, 11.47, 9.79 [03:29:33] PROBLEM - mw133 Current Load on mw133 is CRITICAL: LOAD CRITICAL - total load average: 13.99, 12.28, 10.44 [03:30:33] RECOVERY - mw134 Current Load on mw134 is OK: LOAD OK - total load average: 7.61, 10.17, 9.92 [03:31:27] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 10.69, 11.38, 10.31 [03:33:21] RECOVERY - mw131 Current Load on mw131 is OK: LOAD OK - total load average: 8.30, 10.12, 9.95 [03:33:28] RECOVERY - mw132 Current Load on mw132 is OK: LOAD OK - total load average: 6.98, 9.27, 9.29 [03:35:17] RECOVERY - mw133 Current Load on mw133 is OK: LOAD OK - total load average: 7.70, 9.82, 9.97 [03:44:51] RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp34.miraheze.org - CNAME OK [03:50:30] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [03:56:03] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 11.81, 10.20, 9.34 [04:03:41] RECOVERY - mw133 Current Load on mw133 is OK: LOAD OK - total load average: 8.41, 9.81, 9.66 [04:05:11] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.70, 6.83, 6.25 [04:07:10] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.37, 6.55, 6.21 [04:29:15] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp34.miraheze.org - CNAME OK [04:49:34] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['NS.ANKH.FR.eu.org.', 'NS1.eu.org.', 'NS1.ERIOMEM.NET.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [05:07:20] PROBLEM - cp35 HTTPS on cp35 is CRITICAL: HTTP CRITICAL: HTTP/2 502 - 3370 bytes in 0.456 second response time [05:07:23] PROBLEM - cp34 HTTPS on cp34 is CRITICAL: HTTP CRITICAL: HTTP/2 502 - 3392 bytes in 0.454 second response time [05:07:24] PROBLEM - mw143 HTTPS on mw143 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [05:07:29] PROBLEM - mw131 HTTPS on mw131 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [05:07:55] PROBLEM - mw134 HTTPS on mw134 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [05:09:20] PROBLEM - cp35 HTTPS on cp35 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.602 second response time [05:09:21] PROBLEM - mw143 HTTPS on mw143 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.489 second response time [05:09:23] PROBLEM - cp34 HTTPS on cp34 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.609 second response time [05:09:28] PROBLEM - mw131 HTTPS on mw131 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.561 second response time [05:09:53] PROBLEM - mw134 HTTPS on mw134 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.524 second response time [05:10:18] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 11.19, 10.21, 8.67 [05:10:21] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 9.92, 10.29, 8.59 [05:12:15] RECOVERY - mw131 Current Load on mw131 is OK: LOAD OK - total load average: 9.02, 9.55, 8.60 [05:12:21] RECOVERY - mw134 Current Load on mw134 is OK: LOAD OK - total load average: 7.73, 9.66, 8.59 [05:15:22] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.85, 6.73, 6.12 [05:17:22] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.83, 6.33, 6.04 [05:20:18] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.33, 6.82, 6.20 [05:24:18] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 5.93, 6.52, 6.23 [05:33:45] [Grafana] !sre FIRING: There has been a rise in the MediaWiki exception rate https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [05:34:53] PROBLEM - cp24 Varnish Backends on cp24 is CRITICAL: 2 backends are down. mw151 mw152 [05:34:55] PROBLEM - cp35 Varnish Backends on cp35 is CRITICAL: 2 backends are down. mw151 mw152 [05:35:17] PROBLEM - cp25 Varnish Backends on cp25 is CRITICAL: 2 backends are down. mw151 mw152 [05:35:26] PROBLEM - cp34 Varnish Backends on cp34 is CRITICAL: 2 backends are down. mw151 mw152 [05:41:13] PROBLEM - mw133 Current Load on mw133 is CRITICAL: LOAD CRITICAL - total load average: 16.70, 12.52, 9.72 [05:41:52] PROBLEM - mw131 Current Load on mw131 is CRITICAL: LOAD CRITICAL - total load average: 19.24, 13.10, 9.81 [05:42:17] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.83, 6.85, 6.46 [05:42:21] PROBLEM - mw134 Current Load on mw134 is CRITICAL: LOAD CRITICAL - total load average: 19.56, 12.24, 9.22 [05:42:48] PROBLEM - mw132 Current Load on mw132 is CRITICAL: LOAD CRITICAL - total load average: 20.62, 13.02, 9.59 [05:43:22] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.70, 7.18, 6.59 [05:43:23] PROBLEM - mw141 Current Load on mw141 is CRITICAL: LOAD CRITICAL - total load average: 14.83, 9.79, 7.08 [05:44:17] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 5.23, 6.20, 6.26 [05:44:39] PROBLEM - mw143 Current Load on mw143 is CRITICAL: LOAD CRITICAL - total load average: 13.59, 10.27, 7.52 [05:45:20] PROBLEM - mw142 Current Load on mw142 is WARNING: LOAD WARNING - total load average: 11.54, 9.85, 7.39 [05:45:22] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.90, 6.76, 6.51 [05:46:38] RECOVERY - mw143 Current Load on mw143 is OK: LOAD OK - total load average: 5.92, 8.86, 7.37 [05:47:19] RECOVERY - mw142 Current Load on mw142 is OK: LOAD OK - total load average: 6.91, 8.57, 7.21 [05:47:23] RECOVERY - mw141 Current Load on mw141 is OK: LOAD OK - total load average: 6.59, 8.86, 7.41 [05:47:52] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 6.83, 11.61, 10.60 [05:48:17] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.92, 6.81, 6.52 [05:48:46] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 8.17, 11.97, 10.59 [05:49:13] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 6.95, 11.07, 10.87 [05:49:52] RECOVERY - mw131 Current Load on mw131 is OK: LOAD OK - total load average: 5.69, 9.74, 10.04 [05:50:18] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 6.62, 6.80, 6.56 [05:50:21] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 6.84, 10.33, 10.17 [05:51:22] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.82, 6.50, 6.41 [05:52:21] RECOVERY - mw134 Current Load on mw134 is OK: LOAD OK - total load average: 6.24, 8.94, 9.68 [05:52:45] RECOVERY - mw132 Current Load on mw132 is OK: LOAD OK - total load average: 6.39, 9.03, 9.70 [05:53:13] RECOVERY - mw133 Current Load on mw133 is OK: LOAD OK - total load average: 6.83, 8.94, 10.05 [05:53:22] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.15, 6.46, 6.42 [06:03:45] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1[Grafana] !sre FIRING: There has been a rise in the MediaWiki exception rate https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [06:04:33] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 4.99, 3.24, 2.62 [06:06:18] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.51, 7.14, 6.55 [06:06:28] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 2.68, 3.01, 2.61 [06:08:18] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 5.09, 6.46, 6.39 [06:08:45] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1[Grafana] !sre FIRING: The mediawiki job queue has more than 2500 unclaimed jobs https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1[Grafana] !sre FIRING: There has been a rise in the MediaWiki exception rate https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [06:21:50] RECOVERY - ns1 NTP time on ns1 is OK: NTP OK: Offset 0.009845614433 secs [06:23:28] [02miraheze/puppet] 07AgentIsai pushed 031 commit to 03AgentIsai-patch-1 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/a01adee26eb1 [06:23:29] [02miraheze/puppet] 07AgentIsai 03a01adee - provision cp26, cp36, cp41, and cp51 [06:23:32] [02puppet] 07AgentIsai created branch 03AgentIsai-patch-1 - 13https://github.com/miraheze/puppet [06:23:35] [02puppet] 07AgentIsai opened pull request 03#3674: provision cp26, cp36, cp41, and cp51 - 13https://github.com/miraheze/puppet/pull/3674 [06:35:08] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.00, 6.93, 6.55 [06:36:16] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.36, 6.87, 6.65 [06:37:06] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.42, 6.71, 6.51 [06:40:11] [02miraheze/puppet] 07AgentIsai pushed 031 commit to 03AgentIsai-patch-1 [+4/-0/±0] 13https://github.com/miraheze/puppet/compare/a01adee26eb1...5b9d584081fb [06:40:13] [02miraheze/puppet] 07AgentIsai 035b9d584 - Add to hieradata [06:40:15] [02puppet] 07AgentIsai synchronize pull request 03#3674: provision cp26, cp36, cp41, and cp51 - 13https://github.com/miraheze/puppet/pull/3674 [06:40:16] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 6.55, 6.65, 6.61 [06:40:28] [02puppet] 07AgentIsai closed pull request 03#3674: provision cp26, cp36, cp41, and cp51 - 13https://github.com/miraheze/puppet/pull/3674 [06:40:31] [02miraheze/puppet] 07AgentIsai pushed 031 commit to 03master [+4/-0/±1] 13https://github.com/miraheze/puppet/compare/929ba5942710...813de7b2983d [06:40:34] [02miraheze/puppet] 07AgentIsai 03813de7b - provision cp26, cp36, cp41, and cp51 (#3674) [06:40:36] [02miraheze/puppet] 07AgentIsai deleted branch 03AgentIsai-patch-1 [06:40:38] [02puppet] 07AgentIsai deleted branch 03AgentIsai-patch-1 - 13https://github.com/miraheze/puppet [06:47:52] PROBLEM - cp26 Nginx Backend for phab121 on cp26 is CRITICAL: connect to address 2a02:c206:2161:9253::1 port 5666: Connection refusedconnect to host 2a02:c206:2161:9253::1 port 5666: Connection refused [06:47:58] PROBLEM - cp26 Nginx Backend for mw134 on cp26 is CRITICAL: connect to address 2a02:c206:2161:9253::1 port 5666: Connection refusedconnect to host 2a02:c206:2161:9253::1 port 5666: Connection refused [06:47:58] PROBLEM - cp26 Disk Space on cp26 is CRITICAL: connect to address 2a02:c206:2161:9253::1 port 5666: Connection refusedconnect to host 2a02:c206:2161:9253::1 port 5666: Connection refused [06:48:00] PROBLEM - cp26 Varnish Backends on cp26 is CRITICAL: connect to address 2a02:c206:2161:9253::1 port 5666: Connection refusedconnect to host 2a02:c206:2161:9253::1 port 5666: Connection refused [06:48:03] PROBLEM - cp26 Nginx Backend for mw143 on cp26 is CRITICAL: connect to address 2a02:c206:2161:9253::1 port 5666: Connection refusedconnect to host 2a02:c206:2161:9253::1 port 5666: Connection refused [06:48:11] PROBLEM - cp26 Nginx Backend for mw152 on cp26 is CRITICAL: connect to address 2a02:c206:2161:9253::1 port 5666: Connection refusedconnect to host 2a02:c206:2161:9253::1 port 5666: Connection refused [06:48:12] PROBLEM - cp26 Nginx Backend for mw181 on cp26 is CRITICAL: connect to address 2a02:c206:2161:9253::1 port 5666: Connection refusedconnect to host 2a02:c206:2161:9253::1 port 5666: Connection refused [06:48:21] PROBLEM - cp26 APT on cp26 is CRITICAL: connect to address 2a02:c206:2161:9253::1 port 5666: Connection refusedconnect to host 2a02:c206:2161:9253::1 port 5666: Connection refused [06:48:22] PROBLEM - cp26 NTP time on cp26 is CRITICAL: connect to address 2a02:c206:2161:9253::1 port 5666: Connection refusedconnect to host 2a02:c206:2161:9253::1 port 5666: Connection refused [06:48:23] PROBLEM - cp26 Nginx Backend for matomo151 on cp26 is CRITICAL: connect to address 2a02:c206:2161:9253::1 port 5666: Connection refusedconnect to host 2a02:c206:2161:9253::1 port 5666: Connection refused [06:48:23] PROBLEM - cp26 Nginx Backend for mail121 on cp26 is CRITICAL: connect to address 2a02:c206:2161:9253::1 port 5666: Connection refusedconnect to host 2a02:c206:2161:9253::1 port 5666: Connection refused [06:48:23] PROBLEM - cp26 ferm_active on cp26 is CRITICAL: connect to address 2a02:c206:2161:9253::1 port 5666: Connection refusedconnect to host 2a02:c206:2161:9253::1 port 5666: Connection refused [06:48:23] PROBLEM - cp26 Nginx Backend for mw141 on cp26 is CRITICAL: connect to address 2a02:c206:2161:9253::1 port 5666: Connection refusedconnect to host 2a02:c206:2161:9253::1 port 5666: Connection refused [06:48:24] PROBLEM - cp26 conntrack_table_size on cp26 is CRITICAL: connect to address 2a02:c206:2161:9253::1 port 5666: Connection refusedconnect to host 2a02:c206:2161:9253::1 port 5666: Connection refused [06:48:26] PROBLEM - cp26 Nginx Backend for mw131 on cp26 is CRITICAL: connect to address 2a02:c206:2161:9253::1 port 5666: Connection refusedconnect to host 2a02:c206:2161:9253::1 port 5666: Connection refused [06:48:26] PROBLEM - ping6 on cp26 is WARNING: PING WARNING - Packet loss = 0%, RTA = 137.73 ms [06:48:27] PROBLEM - cp26 Nginx Backend for mw132 on cp26 is CRITICAL: connect to address 2a02:c206:2161:9253::1 port 5666: Connection refusedconnect to host 2a02:c206:2161:9253::1 port 5666: Connection refused [06:48:27] PROBLEM - cp26 Nginx Backend for test151 on cp26 is CRITICAL: connect to address 2a02:c206:2161:9253::1 port 5666: Connection refusedconnect to host 2a02:c206:2161:9253::1 port 5666: Connection refused [06:48:28] PROBLEM - cp26 Nginx Backend for mwtask181 on cp26 is CRITICAL: connect to address 2a02:c206:2161:9253::1 port 5666: Connection refusedconnect to host 2a02:c206:2161:9253::1 port 5666: Connection refused [06:48:29] PROBLEM - cp26 Nginx Backend for mw133 on cp26 is CRITICAL: connect to address 2a02:c206:2161:9253::1 port 5666: Connection refusedconnect to host 2a02:c206:2161:9253::1 port 5666: Connection refused [06:48:29] PROBLEM - cp26 HTTPS on cp26 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 7 - Failed to connect to cp26.wikitide.net port 443 after 138 ms: Couldn't connect to server [06:48:30] PROBLEM - cp26 Nginx Backend for mw182 on cp26 is CRITICAL: connect to address 2a02:c206:2161:9253::1 port 5666: Connection refusedconnect to host 2a02:c206:2161:9253::1 port 5666: Connection refused [06:49:36] PROBLEM - cp26 Puppet on cp26 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:49:40] PROBLEM - cp26 Nginx Backend for mw151 on cp26 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:49:41] PROBLEM - cp26 Nginx Backend for mon181 on cp26 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:49:42] PROBLEM - cp26 Nginx Backend for mw142 on cp26 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:49:43] PROBLEM - cp26 PowerDNS Recursor on cp26 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:49:43] PROBLEM - cp26 Nginx Backend for puppet181 on cp26 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:49:45] PROBLEM - cp26 Current Load on cp26 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:49:45] PROBLEM - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:49:47] PROBLEM - cp26 Nginx Backend for reports121 on cp26 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [06:50:30] PROBLEM - ping6 on cp26 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 137.66 ms [06:52:33] PROBLEM - ping6 on cp26 is WARNING: PING WARNING - Packet loss = 0%, RTA = 138.20 ms [06:53:25] PROBLEM - cp26 conntrack_table_size on cp26 is UNKNOWN: NRPE: Unable to read output [06:53:25] PROBLEM - cp26 ferm_active on cp26 is UNKNOWN: NRPE: Unable to read output [06:53:26] RECOVERY - cp26 Nginx Backend for matomo151 on cp26 is OK: TCP OK - 0.000 second response time on localhost port 8203 [06:53:26] RECOVERY - cp26 Nginx Backend for mw141 on cp26 is OK: TCP OK - 0.000 second response time on localhost port 8108 [06:53:26] RECOVERY - cp26 Nginx Backend for mw181 on cp26 is OK: TCP OK - 0.000 second response time on localhost port 8115 [06:53:26] RECOVERY - cp26 Nginx Backend for mw182 on cp26 is OK: TCP OK - 0.000 second response time on localhost port 8116 [06:53:26] RECOVERY - cp26 Nginx Backend for mw133 on cp26 is OK: TCP OK - 0.000 second response time on localhost port 8110 [06:53:26] RECOVERY - cp26 Nginx Backend for mwtask181 on cp26 is OK: TCP OK - 0.000 second response time on localhost port 8160 [06:53:27] RECOVERY - cp26 Nginx Backend for mw132 on cp26 is OK: TCP OK - 0.000 second response time on localhost port 8107 [06:53:27] RECOVERY - cp26 NTP time on cp26 is OK: NTP OK: Offset -0.005936771631 secs [06:53:28] PROBLEM - cp26 Varnish Backends on cp26 is WARNING: No backends detected. If this is an error, see readme.txt [06:53:28] RECOVERY - cp26 APT on cp26 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [06:53:29] RECOVERY - cp26 Nginx Backend for mail121 on cp26 is OK: TCP OK - 0.000 second response time on localhost port 8200 [06:53:30] RECOVERY - cp26 Nginx Backend for mw143 on cp26 is OK: TCP OK - 0.000 second response time on localhost port 8112 [06:53:31] RECOVERY - cp26 Nginx Backend for test151 on cp26 is OK: TCP OK - 0.000 second response time on localhost port 8181 [06:53:31] RECOVERY - cp26 Nginx Backend for mw131 on cp26 is OK: TCP OK - 0.000 second response time on localhost port 8106 [06:53:37] RECOVERY - cp26 Nginx Backend for mw152 on cp26 is OK: TCP OK - 0.000 second response time on localhost port 8114 [06:54:29] PROBLEM - cp26 Puppet on cp26 is UNKNOWN: NRPE: Unable to read output [06:54:32] RECOVERY - cp26 Nginx Backend for puppet181 on cp26 is OK: TCP OK - 0.000 second response time on localhost port 8204 [06:54:35] PROBLEM - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is UNKNOWN: UNKNOWN - NGINX Error Rate is UNKNOWN [06:54:36] RECOVERY - cp26 Current Load on cp26 is OK: LOAD OK - total load average: 0.59, 0.86, 0.50 [06:54:38] RECOVERY - cp26 Nginx Backend for mw151 on cp26 is OK: TCP OK - 0.000 second response time on localhost port 8113 [06:54:40] RECOVERY - cp26 Nginx Backend for mon181 on cp26 is OK: TCP OK - 0.000 second response time on localhost port 8201 [06:54:41] RECOVERY - cp26 Nginx Backend for mw142 on cp26 is OK: TCP OK - 0.000 second response time on localhost port 8109 [06:54:42] RECOVERY - cp26 Nginx Backend for reports121 on cp26 is OK: TCP OK - 0.001 second response time on localhost port 8205 [06:54:42] RECOVERY - cp26 PowerDNS Recursor on cp26 is OK: DNS OK: 0.084 seconds response time. miraheze.org returns 2001:41d0:801:2000::3a18,2001:41d0:801:2000::5d68,51.195.201.140,51.89.139.24 [06:55:26] PROBLEM - cp26 Varnish Backends on cp26 is CRITICAL: 11 backends are down. mw131 mw132 mw141 mw142 mw133 mw134 mw143 mw151 mw152 mw182 mediawiki [06:55:36] RECOVERY - cp26 Nginx Backend for phab121 on cp26 is OK: TCP OK - 0.000 second response time on localhost port 8202 [06:55:46] PROBLEM - cp26 Disk Space on cp26 is WARNING: DISK WARNING - free space: / 8448MiB (9% inode=98%); [06:55:47] RECOVERY - cp26 Nginx Backend for mw134 on cp26 is OK: TCP OK - 0.000 second response time on localhost port 8111 [06:56:26] [02miraheze/dns] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/a4c74e99232d...7eb2819a3610 [06:56:28] [02miraheze/dns] 07AgentIsai 037eb2819 - Add cp26, cp41, and cp51 [06:56:30] PROBLEM - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is CRITICAL: CRITICAL - NGINX Error Rate is 100% [06:57:37] PROBLEM - ns1 NTP time on ns1 is WARNING: NTP WARNING: Offset 0.1101670861 secs [06:58:28] RECOVERY - cp26 Puppet on cp26 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [06:59:19] RECOVERY - cp26 ferm_active on cp26 is OK: OK ferm input default policy is set [06:59:25] RECOVERY - cp26 conntrack_table_size on cp26 is OK: OK: nf_conntrack is 0 % full [07:00:42] PROBLEM - cp26 HTTPS on cp26 is WARNING: HTTP WARNING: HTTP/2 404 - 3476 bytes in 0.578 second response time [07:01:02] PROBLEM - mw133 Current Load on mw133 is CRITICAL: LOAD CRITICAL - total load average: 13.11, 10.26, 8.44 [07:01:37] PROBLEM - cp26 Varnish Backends on cp26 is WARNING: No backends detected. If this is an error, see readme.txt [07:03:01] RECOVERY - mw133 Current Load on mw133 is OK: LOAD OK - total load average: 8.29, 9.30, 8.31 [07:03:36] PROBLEM - cp26 Varnish Backends on cp26 is CRITICAL: 7 backends are down. mw131 mw132 mw141 mw134 mw143 mw151 mw152 [07:06:40] PROBLEM - mw141 HTTPS on mw141 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10000 milliseconds with 0 bytes received [07:08:35] PROBLEM - mw141 HTTPS on mw141 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.516 second response time [07:18:03] PROBLEM - cp41 HTTP 4xx/5xx ERROR Rate on cp41 is CRITICAL: CRITICAL - NGINX Error Rate is 100% [07:18:09] PROBLEM - cp41 Varnish Backends on cp41 is CRITICAL: 2 backends are down. mw151 mw152 [07:18:11] PROBLEM - cp51 Disk Space on cp51 is WARNING: DISK WARNING - free space: / 8444MiB (9% inode=98%); [07:18:21] PROBLEM - ping6 on cp41 is WARNING: PING WARNING - Packet loss = 0%, RTA = 129.83 ms [07:18:22] PROBLEM - cp51 HTTPS on cp51 is WARNING: HTTP WARNING: HTTP/2 404 - 3476 bytes in 1.307 second response time [07:18:24] PROBLEM - cp51 Varnish Backends on cp51 is CRITICAL: 4 backends are down. mw151 mw152 mw181 mw182 [07:18:31] PROBLEM - ping6 on cp51 is WARNING: PING WARNING - Packet loss = 0%, RTA = 162.76 ms [07:18:32] PROBLEM - cp41 HTTPS on cp41 is WARNING: HTTP WARNING: HTTP/2 404 - 3476 bytes in 1.208 second response time [07:18:49] PROBLEM - cp51 HTTP 4xx/5xx ERROR Rate on cp51 is CRITICAL: CRITICAL - NGINX Error Rate is 100% [07:18:49] PROBLEM - cp41 Disk Space on cp41 is WARNING: DISK WARNING - free space: / 8444MiB (9% inode=98%); [07:21:54] [02miraheze/dns] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/7eb2819a3610...762bdb0bc62f [07:21:56] [02miraheze/dns] 07AgentIsai 03762bdb0 - Update config [07:22:38] RECOVERY - cp51 HTTP 4xx/5xx ERROR Rate on cp51 is OK: OK - NGINX Error Rate is 24% [07:24:01] RECOVERY - cp41 HTTP 4xx/5xx ERROR Rate on cp41 is OK: OK - NGINX Error Rate is 1% [07:24:02] RECOVERY - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is OK: OK - NGINX Error Rate is 8% [07:24:46] [02miraheze/mw-config] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/94f055eb8614...8cd58587e8b1 [07:24:48] [02miraheze/mw-config] 07AgentIsai 038cd5858 - Add cp26, 41, 51 [07:25:06] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 16.27, 9.95, 7.24 [07:25:28] !log [agent@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [07:25:31] PROBLEM - mw133 Current Load on mw133 is CRITICAL: LOAD CRITICAL - total load average: 15.48, 11.18, 9.17 [07:25:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:25:42] PROBLEM - mw131 Current Load on mw131 is CRITICAL: LOAD CRITICAL - total load average: 12.01, 9.91, 8.19 [07:25:50] miraheze/mw-config - AgentIsai the build passed. [07:26:16] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 11.52, 10.34, 9.03 [07:26:43] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 14.90, 10.38, 7.58 [07:26:51] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 9.65, 7.88, 5.82 [07:27:28] PROBLEM - mw132 Current Load on mw132 is CRITICAL: LOAD CRITICAL - total load average: 12.65, 11.02, 9.07 [07:27:29] !log [agent@mwtask181] DEPLOY ABORTED: Canary check failed for publictestwiki.com@mw151.wikitide.net [07:27:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:28:13] PROBLEM - mw134 Current Load on mw134 is CRITICAL: LOAD CRITICAL - total load average: 13.76, 11.80, 9.73 [07:28:36] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 134.81 ms [07:28:47] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.96, 7.49, 5.92 [07:29:25] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 10.21, 10.73, 9.21 [07:29:39] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 10.34, 11.15, 9.16 [07:30:13] [02mw-config] 07R4356th reviewed pull request 03#5410 commit - 13https://github.com/miraheze/mw-config/pull/5410#discussion_r1464439458 [07:30:39] PROBLEM - ping6 on cp41 is WARNING: PING WARNING - Packet loss = 0%, RTA = 134.56 ms [07:30:42] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 10.31, 8.64, 6.53 [07:31:23] PROBLEM - mw132 Current Load on mw132 is CRITICAL: LOAD CRITICAL - total load average: 15.25, 12.09, 9.86 [07:33:20] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 9.50, 11.13, 9.79 [07:33:57] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.53, 3.39, 2.80 [07:34:03] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 11.10, 11.67, 10.32 [07:34:34] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.17, 7.91, 6.70 [07:35:34] RECOVERY - mw131 Current Load on mw131 is OK: LOAD OK - total load average: 6.91, 9.44, 9.13 [07:35:55] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 2.17, 2.95, 2.71 [07:37:31] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 9.08, 11.36, 10.97 [13:21:02] RECOVERY - coffeewiki.net - LetsEncrypt on sslhost is OK: OK - Certificate 'coffeewiki.net' will expire on Tue 16 Apr 2024 11:41:24 PM GMT +0000. [13:21:03] RECOVERY - wikiyri.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wikiyri.org' will expire on Mon 01 Apr 2024 01:53:01 PM GMT +0000. [13:21:03] RECOVERY - mabi.myomi.net - reverse DNS on sslhost is OK: SSL OK - mabi.myomi.net reverse DNS resolves to cp34.miraheze.org - CNAME OK [13:21:04] RECOVERY - en.clockup.wiki - reverse DNS on sslhost is OK: SSL OK - en.clockup.wiki reverse DNS resolves to cp35.miraheze.org - CNAME OK [13:21:04] RECOVERY - dragonquestwiki.com - reverse DNS on sslhost is OK: SSL OK - dragonquestwiki.com reverse DNS resolves to cp35.miraheze.org - NS RECORDS OK [13:21:05] RECOVERY - cloud17 APT on cloud17 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [13:21:05] RECOVERY - themagical.world - reverse DNS on sslhost is OK: SSL OK - themagical.world reverse DNS resolves to cp34.miraheze.org - NS RECORDS OK [13:21:06] RECOVERY - housing.wiki - reverse DNS on sslhost is OK: SSL OK - housing.wiki reverse DNS resolves to cp35.miraheze.org - CNAME FLAT [13:21:06] RECOVERY - wiki.jeremyboyd.org - reverse DNS on sslhost is OK: SSL OK - wiki.jeremyboyd.org reverse DNS resolves to cp35.miraheze.org - CNAME OK [13:21:07] RECOVERY - worldsanskrit.net - LetsEncrypt on sslhost is OK: OK - Certificate 'worldsanskrit.net' will expire on Sat 02 Mar 2024 11:32:48 AM GMT +0000. [13:21:07] RECOVERY - cloud17 conntrack_table_size on cloud17 is OK: OK: nf_conntrack is 0 % full [13:21:08] RECOVERY - cloud17 NTP time on cloud17 is OK: NTP OK: Offset -0.000430226326 secs [13:21:08] RECOVERY - wiki.overwood.xyz - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.overwood.xyz' will expire on Wed 17 Apr 2024 01:45:02 PM GMT +0000. [13:21:09] PROBLEM - isv.orain.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'orain.org' expires in 10 day(s) (Sat 03 Feb 2024 07:51:10 PM GMT +0000). [13:21:09] RECOVERY - wikigenius.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wikigenius.org' will expire on Mon 01 Apr 2024 01:28:06 PM GMT +0000. [13:21:10] RECOVERY - wiki.thesimswiki.com - reverse DNS on sslhost is OK: SSL OK - wiki.thesimswiki.com reverse DNS resolves to cp35.miraheze.org - CNAME OK [13:21:10] RECOVERY - tl.awiki.org - reverse DNS on sslhost is OK: SSL OK - tl.awiki.org reverse DNS resolves to cp34.miraheze.org - CNAME OK [13:21:11] RECOVERY - wiki.overwood.xyz - reverse DNS on sslhost is OK: SSL OK - wiki.overwood.xyz reverse DNS resolves to cp34.miraheze.org - CNAME OK [13:21:22] RECOVERY - mw141 Current Load on mw141 is OK: LOAD OK - total load average: 8.16, 9.90, 8.87 [13:21:24] RECOVERY - shiroko.us - LetsEncrypt on sslhost is OK: OK - Certificate 'shiroko.us' will expire on Thu 14 Mar 2024 07:01:48 PM GMT +0000. [13:21:25] RECOVERY - largedu.eu.org - LetsEncrypt on sslhost is OK: OK - Certificate 'largedu.eu.org' will expire on Sun 03 Mar 2024 02:54:24 PM GMT +0000. [13:21:25] RECOVERY - wiki.mxlinuxusers.de - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.mxlinuxusers.de' will expire on Fri 01 Mar 2024 01:23:29 PM GMT +0000. [13:21:26] RECOVERY - cp24 Varnish Backends on cp24 is OK: All 19 backends are healthy [13:21:26] RECOVERY - www.dariawiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'dariawiki.org' will expire on Fri 01 Mar 2024 11:09:12 AM GMT +0000. [13:21:26] RECOVERY - cloud15 Puppet on cloud15 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [13:21:27] RECOVERY - heavyironmodding.org - reverse DNS on sslhost is OK: SSL OK - heavyironmodding.org reverse DNS resolves to cp35.miraheze.org - NS RECORDS OK [13:21:28] RECOVERY - threedomwiki.pcast.site - LetsEncrypt on sslhost is OK: OK - Certificate 'threedomwiki.pcast.site' will expire on Sat 02 Mar 2024 12:34:01 PM GMT +0000. [13:21:30] RECOVERY - touhouplace.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'touhouplace.wiki' will expire on Thu 14 Mar 2024 07:29:29 PM GMT +0000. [13:21:32] RECOVERY - luxuryelevator.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Mon 18 Nov 2024 11:59:59 PM GMT +0000. [13:21:34] RECOVERY - www.dovearchives.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'www.dovearchives.wiki' will expire on Fri 22 Mar 2024 12:42:30 AM GMT +0000. [13:21:35] RECOVERY - knifepointhorror.wiki - reverse DNS on sslhost is OK: SSL OK - knifepointhorror.wiki reverse DNS resolves to cp34.miraheze.org - NS RECORDS OK [13:21:37] RECOVERY - wiki.zergmafia.top - reverse DNS on sslhost is OK: SSL OK - wiki.zergmafia.top reverse DNS resolves to cp34.miraheze.org - CNAME OK [13:21:37] RECOVERY - segawiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'segawiki.com' will expire on Sun 03 Mar 2024 06:18:42 PM GMT +0000. [13:21:38] RECOVERY - wiki.iurevar.world - reverse DNS on sslhost is OK: SSL OK - wiki.iurevar.world reverse DNS resolves to cp35.miraheze.org - CNAME OK [13:21:39] RECOVERY - wiki.oasismp.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.oasismp.com' will expire on Fri 15 Mar 2024 07:30:11 PM GMT +0000. [13:21:39] RECOVERY - cloud12 Puppet on cloud12 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [13:21:39] RECOVERY - wiki.maxshub.net - LetsEncrypt on sslhost is OK: OK - Certificate 'maxshub.net' will expire on Mon 18 Mar 2024 03:38:45 AM GMT +0000. [13:21:39] RECOVERY - wiki.denby.tech - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.denby.tech' will expire on Tue 19 Mar 2024 04:46:41 PM GMT +0000. [13:21:41] RECOVERY - bast121 Puppet on bast121 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [13:21:42] RECOVERY - removededm.com - reverse DNS on sslhost is OK: SSL OK - removededm.com reverse DNS resolves to cp34.miraheze.org - NS RECORDS OK [13:21:43] RECOVERY - wiki.funkey-project.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.funkey-project.com' will expire on Sat 02 Mar 2024 02:08:51 AM GMT +0000. [13:21:44] RECOVERY - miraheze.wiki - reverse DNS on sslhost is OK: SSL OK - miraheze.wiki reverse DNS resolves to cp34.miraheze.org - NS RECORDS OK [13:21:46] PROBLEM - mw143 HTTPS on mw143 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [13:21:46] RECOVERY - www.hebammenwiki.de - LetsEncrypt on sslhost is OK: OK - Certificate 'www.hebammenwiki.de' will expire on Wed 20 Mar 2024 01:22:39 PM GMT +0000. [13:21:47] RECOVERY - db101 Puppet on db101 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [13:21:48] RECOVERY - fallofsanctuary.com - reverse DNS on sslhost is OK: SSL OK - fallofsanctuary.com reverse DNS resolves to cp34.miraheze.org - NS RECORDS OK [13:21:48] RECOVERY - dkpedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'dkpedia.com' will expire on Sun 03 Mar 2024 03:07:28 PM GMT +0000. [13:21:48] RECOVERY - gil.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'gil.wiki' will expire on Mon 25 Mar 2024 04:02:27 PM GMT +0000. [13:21:50] RECOVERY - wiki.macc.nyc - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.macc.nyc' will expire on Fri 01 Mar 2024 12:22:35 AM GMT +0000. [13:21:50] RECOVERY - cp34 Puppet on cp34 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [13:21:54] RECOVERY - wiki.rlab.org.uk - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.rlab.org.uk' will expire on Fri 16 Feb 2024 07:23:13 PM GMT +0000. [13:21:56] RECOVERY - zhacg.wiki - reverse DNS on sslhost is OK: SSL OK - zhacg.wiki reverse DNS resolves to cp34.miraheze.org - NS RECORDS OK [13:21:56] RECOVERY - en.omniversalis.org - reverse DNS on sslhost is OK: SSL OK - en.omniversalis.org reverse DNS resolves to cp34.miraheze.org - CNAME OK [13:21:57] RECOVERY - cloud17 SSH on cloud17 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u2 (protocol 2.0) [13:22:00] RECOVERY - cloud11 Puppet on cloud11 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [13:22:01] RECOVERY - wiki.showtapeselection.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.showtapeselection.com' will expire on Fri 19 Apr 2024 12:51:15 PM GMT +0000. [13:22:02] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 11.63, 10.52, 9.76 [13:22:06] RECOVERY - wikimas.kr - reverse DNS on sslhost is OK: SSL OK - wikimas.kr reverse DNS resolves to cp35.miraheze.org - NS RECORDS OK [13:22:06] RECOVERY - thelonsdalebattalion.co.uk - LetsEncrypt on sslhost is OK: OK - Certificate 'thelonsdalebattalion.co.uk' will expire on Fri 01 Mar 2024 01:01:52 AM GMT +0000. [13:22:06] RECOVERY - dft.wikiyri.org - reverse DNS on sslhost is OK: SSL OK - dft.wikiyri.org reverse DNS resolves to cp34.miraheze.org - NS RECORDS OK [13:22:07] RECOVERY - tssm.wiki - reverse DNS on sslhost is OK: SSL OK - tssm.wiki reverse DNS resolves to cp34.miraheze.org - NS RECORDS OK [13:22:07] RECOVERY - swiftobject121 Puppet on swiftobject121 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:22:08] RECOVERY - wiki.alathramc.com - reverse DNS on sslhost is OK: SSL OK - wiki.alathramc.com reverse DNS resolves to cp35.miraheze.org - CNAME OK [13:22:10] RECOVERY - worldtobuild.wiki - reverse DNS on sslhost is OK: SSL OK - worldtobuild.wiki reverse DNS resolves to cp35.miraheze.org - NS RECORDS OK [13:22:11] RECOVERY - cnt.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Mon 18 Nov 2024 11:59:59 PM GMT +0000. [13:22:11] RECOVERY - vedopedia.witches-empire.com - LetsEncrypt on sslhost is OK: OK - Certificate 'vedopedia.witches-empire.com' will expire on Fri 01 Mar 2024 06:20:13 PM GMT +0000. [13:22:12] PROBLEM - os141 Puppet on os141 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:22:13] RECOVERY - repository.archiopedia.org - reverse DNS on sslhost is OK: SSL OK - repository.archiopedia.org reverse DNS resolves to cp35.miraheze.org - CNAME OK [13:22:17] RECOVERY - cp24 Puppet on cp24 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [13:22:21] RECOVERY - mockgovernments.com - reverse DNS on sslhost is OK: SSL OK - mockgovernments.com reverse DNS resolves to cp35.miraheze.org - NS RECORDS OK [13:22:22] RECOVERY - www.pomologia.org - reverse DNS on sslhost is OK: SSL OK - www.pomologia.org reverse DNS resolves to cp35.miraheze.org - NS RECORDS OK [13:22:22] RECOVERY - cloud17 PowerDNS Recursor on cloud17 is OK: DNS OK: 0.025 seconds response time. miraheze.org returns 2607:5300:205:200::1c93,2607:5300:205:200::3121,51.222.12.133,51.222.14.30 [13:22:22] RECOVERY - ping6 on cloud17 is OK: PING OK - Packet loss = 0%, RTA = 0.16 ms [13:22:23] RECOVERY - cp25 Puppet on cp25 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [13:22:23] RECOVERY - cp35 Puppet on cp35 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [13:22:23] RECOVERY - cp26 Puppet on cp26 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [13:22:24] RECOVERY - cloud10 Puppet on cloud10 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [13:22:25] RECOVERY - os141 PowerDNS Recursor on os141 is OK: DNS OK: 8.305 seconds response time. miraheze.org returns 2a02:c206:2161:9253::1,84.247.188.12 [13:22:27] RECOVERY - cloud17 Current Load on cloud17 is OK: LOAD OK - total load average: 0.18, 0.24, 0.11 [13:22:30] RECOVERY - db112 Puppet on db112 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:22:32] RECOVERY - cloud17 Puppet on cloud17 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:22:32] RECOVERY - cloud17 ferm_active on cloud17 is OK: OK ferm input default policy is set [13:22:33] RECOVERY - jobchron121 Puppet on jobchron121 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [13:22:33] RECOVERY - www.hoolehistoryheritagesociety.org.uk - LetsEncrypt on sslhost is OK: OK - Certificate 'www.hoolehistoryheritagesociety.org.uk' will expire on Sat 02 Mar 2024 02:00:17 AM GMT +0000. [13:22:34] RECOVERY - infectowiki.com - reverse DNS on sslhost is OK: SSL OK - infectowiki.com reverse DNS resolves to cp34.miraheze.org - NS RECORDS OK [13:22:35] RECOVERY - dlfm-wiki.top - LetsEncrypt on sslhost is OK: OK - Certificate 'dlfm-wiki.top' will expire on Mon 18 Mar 2024 01:57:56 PM GMT +0000. [13:22:35] RECOVERY - cloud18 Puppet on cloud18 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:22:36] RECOVERY - private.yahyabd.xyz - reverse DNS on sslhost is OK: SSL OK - private.yahyabd.xyz reverse DNS resolves to cp35.miraheze.org - CNAME OK [13:22:37] RECOVERY - yokaiwatch.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Mon 18 Nov 2024 11:59:59 PM GMT +0000. [13:22:37] RECOVERY - wiki.pulsus.cc - reverse DNS on sslhost is OK: SSL OK - wiki.pulsus.cc reverse DNS resolves to cp34.miraheze.org - CNAME OK [13:22:38] RECOVERY - wiki.starship.digital - reverse DNS on sslhost is OK: SSL OK - wiki.starship.digital reverse DNS resolves to cp35.miraheze.org - CNAME OK [13:22:40] RECOVERY - wiki.kachow.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.kachow.org' will expire on Sun 03 Mar 2024 01:42:41 AM GMT +0000. [13:22:43] RECOVERY - equestripedia.org - reverse DNS on sslhost is OK: SSL OK - equestripedia.org reverse DNS resolves to cp35.miraheze.org - CNAME FLAT [13:22:44] RECOVERY - pwiki.drydraytonvillagehall.org.uk - reverse DNS on sslhost is OK: SSL OK - pwiki.drydraytonvillagehall.org.uk reverse DNS resolves to cp35.miraheze.org - CNAME OK [13:22:45] RECOVERY - wonderfuleveryday.org - reverse DNS on sslhost is OK: SSL OK - wonderfuleveryday.org reverse DNS resolves to cp35.miraheze.org - NS RECORDS OK [13:22:45] RECOVERY - you.r-fit.cc - reverse DNS on sslhost is OK: SSL OK - you.r-fit.cc reverse DNS resolves to cp35.miraheze.org - CNAME OK [13:22:46] RECOVERY - pornwiki.org - reverse DNS on sslhost is OK: SSL OK - pornwiki.org reverse DNS resolves to cp35.miraheze.org - NS RECORDS OK [13:22:46] RECOVERY - wiki.meregos.com - reverse DNS on sslhost is OK: SSL OK - wiki.meregos.com reverse DNS resolves to cp34.miraheze.org - CNAME OK [13:22:48] RECOVERY - hr.petrawiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'hr.petrawiki.org' will expire on Mon 04 Mar 2024 02:59:19 PM GMT +0000. [13:22:52] RECOVERY - cp51 Puppet on cp51 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:22:54] RECOVERY - reviwiki.info - LetsEncrypt on sslhost is OK: OK - Certificate 'reviwiki.info' will expire on Sun 03 Mar 2024 08:00:29 PM GMT +0000. [13:22:55] RECOVERY - data.nonbinary.wiki - reverse DNS on sslhost is OK: SSL OK - data.nonbinary.wiki reverse DNS resolves to cp34.miraheze.org - CNAME OK [13:22:57] RECOVERY - www.pyramidgames.wiki - reverse DNS on sslhost is OK: SSL OK - www.pyramidgames.wiki reverse DNS resolves to cp34.miraheze.org - NS RECORDS OK [13:22:57] RECOVERY - miraheze.gq - LetsEncrypt on sslhost is OK: OK - Certificate 'miraheze.gq' will expire on Sun 03 Mar 2024 08:52:18 PM GMT +0000. [13:22:59] RECOVERY - cloud.neptune.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'cloud.neptune.wiki' will expire on Sun 24 Mar 2024 02:48:46 AM GMT +0000. [13:23:02] RECOVERY - wiki.yuanpi.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.yuanpi.eu.org reverse DNS resolves to cp34.miraheze.org - CNAME OK [13:23:03] RECOVERY - ie.unigon.net - LetsEncrypt on sslhost is OK: OK - Certificate 'ie.unigon.net' will expire on Thu 29 Feb 2024 09:33:00 AM GMT +0000. [13:23:05] RECOVERY - wiki.walkscape.app - reverse DNS on sslhost is OK: SSL OK - wiki.walkscape.app reverse DNS resolves to cp34.miraheze.org - CNAME OK [13:23:06] RECOVERY - kirisame-kissaten.cf - reverse DNS on sslhost is OK: SSL OK - kirisame-kissaten.cf reverse DNS resolves to cp35.miraheze.org - CNAME FLAT [13:23:08] RECOVERY - rodzinka.wiki - reverse DNS on sslhost is OK: SSL OK - rodzinka.wiki reverse DNS resolves to cp35.miraheze.org - NS RECORDS OK [13:23:09] RECOVERY - wiki.mcsoft.org - reverse DNS on sslhost is OK: SSL OK - wiki.mcsoft.org reverse DNS resolves to cp35.miraheze.org - CNAME OK [13:23:16] RECOVERY - wiki.rosestulipsandliberty.com - reverse DNS on sslhost is OK: SSL OK - wiki.rosestulipsandliberty.com reverse DNS resolves to cp34.miraheze.org - CNAME OK [13:23:17] RECOVERY - wiki.gab.pt.eu.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.gab.pt.eu.org' will expire on Tue 09 Apr 2024 05:31:08 PM GMT +0000. [13:23:20] RECOVERY - cloud14 Puppet on cloud14 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:23:22] RECOVERY - www.rothwell-leeds.co.uk - LetsEncrypt on sslhost is OK: OK - Certificate 'www.rothwell-leeds.co.uk' will expire on Sat 02 Mar 2024 01:49:08 AM GMT +0000. [13:23:23] RECOVERY - electowiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'electowiki.org' will expire on Fri 01 Mar 2024 02:06:02 PM GMT +0000. [13:23:25] RECOVERY - isv.orain.org - reverse DNS on sslhost is OK: SSL OK - isv.orain.org reverse DNS resolves to cp34.miraheze.org - NS RECORDS OK [13:23:26] RECOVERY - www.publictestwiki.com - reverse DNS on sslhost is OK: SSL OK - www.publictestwiki.com reverse DNS resolves to cp34.miraheze.org - CNAME OK [13:23:28] RECOVERY - lcn.zfc.id.lv - reverse DNS on sslhost is OK: SSL OK - lcn.zfc.id.lv reverse DNS resolves to cp34.miraheze.org - CNAME OK [13:23:29] RECOVERY - wiki.aclevo.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.aclevo.com' will expire on Fri 16 Feb 2024 07:18:12 PM GMT +0000. [13:23:31] RECOVERY - lab.wikimimo.com - reverse DNS on sslhost is OK: SSL OK - lab.wikimimo.com reverse DNS resolves to cp34.miraheze.org - CNAME OK [13:23:32] RECOVERY - db131 Puppet on db131 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:23:33] RECOVERY - gil.wiki - reverse DNS on sslhost is OK: SSL OK - gil.wiki reverse DNS resolves to cp35.miraheze.org - NS RECORDS OK [13:23:33] RECOVERY - small.polandballwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'small.polandballwiki.com' will expire on Sat 02 Mar 2024 02:08:27 PM GMT +0000. [13:23:34] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [13:23:34] RECOVERY - graylog131 Puppet on graylog131 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [13:23:35] PROBLEM - cp41 HTTPS on cp41 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [13:23:39] RECOVERY - wiki.queenscourt.games - reverse DNS on sslhost is OK: SSL OK - wiki.queenscourt.games reverse DNS resolves to cp35.miraheze.org - CNAME OK [13:23:39] RECOVERY - wiki.gesamtschule-nordkirchen.de - reverse DNS on sslhost is OK: SSL OK - wiki.gesamtschule-nordkirchen.de reverse DNS resolves to cp35.miraheze.org - CNAME OK [13:23:40] RECOVERY - cloud16 Puppet on cloud16 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:23:41] PROBLEM - cp25 HTTPS on cp25 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [13:23:42] RECOVERY - mem141 Puppet on mem141 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:23:43] RECOVERY - history.estill.org - reverse DNS on sslhost is OK: SSL OK - history.estill.org reverse DNS resolves to cp34.miraheze.org - CNAME OK [13:23:44] RECOVERY - zomvilles.pixiemeta.com - reverse DNS on sslhost is OK: SSL OK - zomvilles.pixiemeta.com reverse DNS resolves to cp35.miraheze.org - CNAME OK [13:23:44] RECOVERY - wiki.dwarvesrp.com - reverse DNS on sslhost is OK: SSL OK - wiki.dwarvesrp.com reverse DNS resolves to cp35.miraheze.org - CNAME OK [13:23:45] RECOVERY - persist.wiki - reverse DNS on sslhost is OK: SSL OK - persist.wiki reverse DNS resolves to cp35.miraheze.org - NS RECORDS OK [13:23:46] PROBLEM - mw132 Current Load on mw132 is CRITICAL: LOAD CRITICAL - total load average: 13.29, 11.18, 10.23 [13:23:48] RECOVERY - www.programming.red - LetsEncrypt on sslhost is OK: OK - Certificate 'programming.red' will expire on Fri 01 Mar 2024 11:41:00 AM GMT +0000. [13:23:48] RECOVERY - wiki.apico.buzz - reverse DNS on sslhost is OK: SSL OK - wiki.apico.buzz reverse DNS resolves to cp34.miraheze.org - CNAME OK [13:23:48] RECOVERY - miraheze.com - reverse DNS on sslhost is OK: SSL OK - miraheze.com reverse DNS resolves to cp34.miraheze.org - NS RECORDS OK [13:23:49] RECOVERY - storytime.jdstroy.cf - LetsEncrypt on sslhost is OK: OK - Certificate 'storytime.jdstroy.cf' will expire on Sat 02 Mar 2024 05:11:34 PM GMT +0000. [13:23:51] RECOVERY - talumn.neptune.wiki - reverse DNS on sslhost is OK: SSL OK - talumn.neptune.wiki reverse DNS resolves to cp35.miraheze.org - CNAME OK [13:23:51] RECOVERY - cp41 Puppet on cp41 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:23:51] RECOVERY - revi.wiki - PositiveSSLDV on sslhost is OK: OK - Certificate 'revi.wiki' will expire on Sun 29 Dec 2024 11:59:59 PM GMT +0000. [13:23:52] RECOVERY - wiki.candelabrem.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.candelabrem.com' will expire on Sun 07 Apr 2024 08:24:56 PM GMT +0000. [13:23:52] RECOVERY - os131 Puppet on os131 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [13:23:52] RECOVERY - quadball.wiki - reverse DNS on sslhost is OK: SSL OK - quadball.wiki reverse DNS resolves to cp35.miraheze.org - NS RECORDS OK [13:23:53] RECOVERY - mw131 Disk Space on mw131 is OK: DISK OK - free space: / 3803MiB (16% inode=70%); [13:23:56] RECOVERY - mariopedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'mariopedia.org' will expire on Sun 03 Mar 2024 02:19:40 AM GMT +0000. [13:23:57] RECOVERY - wiki.scvo.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.scvo.org' will expire on Fri 01 Mar 2024 10:55:32 AM GMT +0000. [13:23:58] RECOVERY - inourownwords.online - reverse DNS on sslhost is OK: SSL OK - inourownwords.online reverse DNS resolves to cp35.miraheze.org - NS RECORDS OK [13:23:59] RECOVERY - wiki.milkywayidle.com - reverse DNS on sslhost is OK: SSL OK - wiki.milkywayidle.com reverse DNS resolves to cp35.miraheze.org - CNAME OK [13:23:59] RECOVERY - wiki.geoparkcorumbatai.com.br - reverse DNS on sslhost is OK: SSL OK - wiki.geoparkcorumbatai.com.br reverse DNS resolves to cp34.miraheze.org - CNAME OK [13:23:59] RECOVERY - fc.songcontests.eu - reverse DNS on sslhost is OK: SSL OK - fc.songcontests.eu reverse DNS resolves to cp34.miraheze.org - CNAME OK [13:24:00] RECOVERY - ldap141 Puppet on ldap141 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [13:24:01] RECOVERY - metroid.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Mon 18 Nov 2024 11:59:59 PM GMT +0000. [13:24:02] PROBLEM - mw131 Current Load on mw131 is CRITICAL: LOAD CRITICAL - total load average: 13.50, 12.20, 10.51 [13:24:02] RECOVERY - internalwiki.coolstation.space - LetsEncrypt on sslhost is OK: OK - Certificate 'internalwiki.coolstation.space' will expire on Wed 13 Mar 2024 07:50:30 AM GMT +0000. [13:24:02] RECOVERY - mail121 Puppet on mail121 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [13:24:02] RECOVERY - wiki.showtapeselection.com - reverse DNS on sslhost is OK: SSL OK - wiki.showtapeselection.com reverse DNS resolves to cp35.miraheze.org - CNAME OK [13:24:14] RECOVERY - www.mh142.com - LetsEncrypt on sslhost is OK: OK - Certificate 'mh142.com' will expire on Fri 01 Mar 2024 11:05:13 AM GMT +0000. [13:24:19] RECOVERY - dkwiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'dkwiki.org' will expire on Sat 02 Mar 2024 07:01:51 PM GMT +0000. [13:24:20] RECOVERY - test2.specialchat.org - reverse DNS on sslhost is OK: SSL OK - test2.specialchat.org reverse DNS resolves to cp34.miraheze.org - CNAME OK [13:24:21] RECOVERY - storytime.jdstroy.cf - reverse DNS on sslhost is OK: SSL OK - storytime.jdstroy.cf reverse DNS resolves to cp34.miraheze.org - CNAME OK [13:24:21] RECOVERY - mw131 Puppet on mw131 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [13:24:22] RECOVERY - eng.archiopedia.org - reverse DNS on sslhost is OK: SSL OK - eng.archiopedia.org reverse DNS resolves to cp35.miraheze.org - CNAME OK [13:24:25] RECOVERY - mw132 Puppet on mw132 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [13:24:25] RECOVERY - history.sdtef.org - reverse DNS on sslhost is OK: SSL OK - history.sdtef.org reverse DNS resolves to cp35.miraheze.org - CNAME OK [13:24:27] RECOVERY - emmytherobot.wiki - reverse DNS on sslhost is OK: SSL OK - emmytherobot.wiki reverse DNS resolves to cp35.miraheze.org - NS RECORDS OK [13:24:31] RECOVERY - pwiki.arkcls.com - reverse DNS on sslhost is OK: SSL OK - pwiki.arkcls.com reverse DNS resolves to cp35.miraheze.org - CNAME OK [13:24:31] RECOVERY - donkeykong.miraheze.org - reverse DNS on sslhost is OK: SSL OK - donkeykong.miraheze.org reverse DNS resolves to cp35.miraheze.org - NS RECORDS OK [13:24:35] RECOVERY - yokaiwatchwiki.com - reverse DNS on sslhost is OK: SSL OK - yokaiwatchwiki.com reverse DNS resolves to cp34.miraheze.org - NS RECORDS OK [13:24:35] RECOVERY - cities.simulz.kr - LetsEncrypt on sslhost is OK: OK - Certificate 'cities.simulz.kr' will expire on Tue 16 Apr 2024 06:33:38 PM GMT +0000. [13:24:36] PROBLEM - allthetropes.orain.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'orain.org' expires in 10 day(s) (Sat 03 Feb 2024 07:51:10 PM GMT +0000). [13:24:37] RECOVERY - wiki.auraxis.co - reverse DNS on sslhost is OK: SSL OK - wiki.auraxis.co reverse DNS resolves to cp34.miraheze.org - CNAME OK [13:24:37] RECOVERY - crocwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'crocwiki.com' will expire on Sat 02 Mar 2024 06:40:15 PM GMT +0000. [13:24:37] RECOVERY - progressbar95.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'progressbar95.wiki' will expire on Fri 16 Feb 2024 07:21:06 PM GMT +0000. [13:24:38] PROBLEM - db182 ferm_active on db182 is WARNING: Could not resolve hostname : Name or service not known [13:24:40] RECOVERY - wiki.potabi.com - reverse DNS on sslhost is OK: SSL OK - wiki.potabi.com reverse DNS resolves to cp34.miraheze.org - CNAME OK [13:24:41] PROBLEM - db181 NTP time on db181 is WARNING: Could not resolve hostname : Name or service not known [13:24:41] RECOVERY - hr.petrawiki.org - reverse DNS on sslhost is OK: SSL OK - hr.petrawiki.org reverse DNS resolves to cp34.miraheze.org - NS RECORDS OK [13:24:45] RECOVERY - kb.nena.org - reverse DNS on sslhost is OK: SSL OK - kb.nena.org reverse DNS resolves to cp34.miraheze.org - CNAME OK [13:24:46] RECOVERY - ao90.pinho.org - LetsEncrypt on sslhost is OK: OK - Certificate 'ao90.pinho.org' will expire on Fri 01 Mar 2024 12:04:13 PM GMT +0000. [13:24:46] PROBLEM - mon181 Check correctness of the icinga configuration on mon181 is WARNING: Could not resolve hostname : Name or service not known [13:24:49] RECOVERY - mem131 Puppet on mem131 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:24:49] RECOVERY - phab121 Puppet on phab121 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:24:49] PROBLEM - db181 Disk Space on db181 is WARNING: Could not resolve hostname : Name or service not known [13:24:50] RECOVERY - rockrevolution.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'rockrevolution.wiki' will expire on Mon 19 Feb 2024 02:25:53 AM GMT +0000. [13:24:51] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.88, 6.84, 5.84 [13:24:52] PROBLEM - mon181 NTP time on mon181 is WARNING: Could not resolve hostname : Name or service not known [13:24:56] RECOVERY - swiftac111 Puppet on swiftac111 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:24:58] RECOVERY - abeyancewiki.bumblebee-studios.se - reverse DNS on sslhost is OK: SSL OK - abeyancewiki.bumblebee-studios.se reverse DNS resolves to cp35.miraheze.org - CNAME OK [13:25:00] PROBLEM - bast181 APT on bast181 is WARNING: Could not resolve hostname : Name or service not known [13:25:01] RECOVERY - pandorastale.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Mon 18 Nov 2024 11:59:59 PM GMT +0000. [13:25:02] RECOVERY - reports121 Puppet on reports121 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:25:03] RECOVERY - bast141 Puppet on bast141 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [13:25:03] RECOVERY - wiki.beergeeks.co.il - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.beergeeks.co.il' will expire on Sat 02 Mar 2024 12:52:53 PM GMT +0000. [13:25:04] RECOVERY - legacygt.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'legacygt.wiki' will expire on Sun 03 Mar 2024 12:47:38 AM GMT +0000. [13:25:06] RECOVERY - echoes-wiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'echoes-wiki.com' will expire on Thu 14 Mar 2024 07:43:47 PM GMT +0000. [13:25:06] PROBLEM - mon181 Disk Space on mon181 is WARNING: Could not resolve hostname : Name or service not known [13:25:07] RECOVERY - comprehensibleinputwiki.org - reverse DNS on sslhost is OK: SSL OK - comprehensibleinputwiki.org reverse DNS resolves to cp35.miraheze.org - NS RECORDS OK [13:25:08] RECOVERY - marinebiodiversitymatrix.org - reverse DNS on sslhost is OK: SSL OK - marinebiodiversitymatrix.org reverse DNS resolves to cp34.miraheze.org - NS RECORDS OK [13:25:08] RECOVERY - hardcore.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'hardcore.wiki' will expire on Sun 18 Feb 2024 07:26:36 PM GMT +0000. [13:25:11] RECOVERY - mw134 Puppet on mw134 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:25:11] PROBLEM - mon181 IRCEcho on mon181 is WARNING: Could not resolve hostname : Name or service not known [13:25:12] PROBLEM - db181 APT on db181 is WARNING: Could not resolve hostname : Name or service not known [13:25:12] RECOVERY - thelangyalist.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Mon 18 Nov 2024 11:59:59 PM GMT +0000. [13:25:12] RECOVERY - wiki.songngu.xyz - reverse DNS on sslhost is OK: SSL OK - wiki.songngu.xyz reverse DNS resolves to cp34.miraheze.org - CNAME OK [13:25:13] RECOVERY - wiki.coolstation.space - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.coolstation.space' will expire on Wed 13 Mar 2024 07:35:23 AM GMT +0000. [13:25:14] RECOVERY - steamdecklinux.wiki - reverse DNS on sslhost is OK: SSL OK - steamdecklinux.wiki reverse DNS resolves to cp34.miraheze.org - CNAME OK [13:25:15] RECOVERY - mw133 Puppet on mw133 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:25:18] RECOVERY - wikislamica.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wikislamica.org' will expire on Wed 06 Mar 2024 02:18:30 AM GMT +0000. [13:25:19] RECOVERY - www.pastport.org - reverse DNS on sslhost is OK: SSL OK - www.pastport.org reverse DNS resolves to cp34.miraheze.org - NS RECORDS OK [13:25:19] RECOVERY - mirabeta.org - LetsEncrypt on sslhost is OK: OK - Certificate 'mirabeta.org' will expire on Mon 08 Apr 2024 11:37:28 AM GMT +0000. [13:25:20] PROBLEM - mon181 IRC RC Bot on mon181 is WARNING: Could not resolve hostname : Name or service not known [13:25:21] RECOVERY - wiki.potabi.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.potabi.com' will expire on Sat 02 Mar 2024 11:24:38 AM GMT +0000. [13:25:21] RECOVERY - swiftproxy131 Puppet on swiftproxy131 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [13:25:22] PROBLEM - db181 conntrack_table_size on db181 is WARNING: Could not resolve hostname : Name or service not known [13:25:24] RECOVERY - wiki.ooer.ooo - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.ooer.ooo' will expire on Sun 24 Mar 2024 03:34:42 PM GMT +0000. [13:25:26] RECOVERY - swiftproxy111 Puppet on swiftproxy111 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:25:27] RECOVERY - wiki.seamly.io - reverse DNS on sslhost is OK: SSL OK - wiki.seamly.io reverse DNS resolves to cp34.miraheze.org - CNAME OK [13:25:27] PROBLEM - Host db181 is DOWN: check_ping: Invalid hostname/address - Usage:check_ping -H -w ,% -c ,% [-p packets] [-t timeout] [-4|-6] [13:25:30] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 11.86, 9.48, 7.96 [13:25:30] PROBLEM - cp41 HTTPS on cp41 is WARNING: HTTP WARNING: HTTP/2 404 - 3480 bytes in 1.391 second response time [13:25:30] RECOVERY - tep.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'tep.wiki' will expire on Fri 01 Mar 2024 12:11:58 PM GMT +0000. [13:25:31] PROBLEM - mon181 PowerDNS Recursor on mon181 is WARNING: Could not resolve hostname : Name or service not known [13:25:32] RECOVERY - smafallfest.org - LetsEncrypt on sslhost is OK: OK - Certificate 'smafallfest.org' will expire on Sun 03 Mar 2024 09:00:43 PM GMT +0000. [13:25:33] PROBLEM - bast181 Current Load on bast181 is WARNING: Could not resolve hostname : Name or service not known [13:25:34] PROBLEM - wiki.geoparkcorumbatai.com.br - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.geoparkcorumbatai.com.br' expires in 15 day(s) (Fri 09 Feb 2024 08:02:28 AM GMT +0000). [13:25:34] RECOVERY - wiki.moores.tech - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.moores.tech' will expire on Thu 21 Mar 2024 06:05:31 PM GMT +0000. [13:25:34] PROBLEM - db182 Disk Space on db182 is WARNING: Could not resolve hostname : Name or service not known [13:25:34] RECOVERY - wiki.fbpml.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.fbpml.org' will expire on Sat 02 Mar 2024 04:05:17 AM GMT +0000. [13:25:35] RECOVERY - swiftobject111 Puppet on swiftobject111 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [13:25:37] PROBLEM - cp25 HTTPS on cp25 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 1.317 second response time [13:25:39] RECOVERY - schizoidnightmares.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'schizoidnightmares.wiki' will expire on Sun 03 Mar 2024 07:25:37 PM GMT +0000. [13:25:39] PROBLEM - db182 Backups SQL reports on db182 is WARNING: Could not resolve hostname : Name or service not known [13:25:43] PROBLEM - bast181 conntrack_table_size on bast181 is WARNING: Could not resolve hostname : Name or service not known [13:25:43] PROBLEM - mon181 IRC Log Server Bot on mon181 is WARNING: Could not resolve hostname : Name or service not known [13:25:43] PROBLEM - db182 Backups SQL roundcubemail on db182 is WARNING: Could not resolve hostname : Name or service not known [13:25:44] RECOVERY - equestripedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'equestripedia.org' will expire on Sat 02 Mar 2024 04:36:06 AM GMT +0000. [13:25:44] PROBLEM - mon181 icinga.miraheze.org HTTPS on mon181 is CRITICAL: Name or service not knownHTTP CRITICAL - Unable to open TCP socket [13:25:45] RECOVERY - wiki.evscope.org - reverse DNS on sslhost is OK: SSL OK - wiki.evscope.org reverse DNS resolves to cp34.miraheze.org - CNAME OK [13:25:46] PROBLEM - db182 Current Load on db182 is WARNING: Could not resolve hostname : Name or service not known [13:25:47] RECOVERY - wiki.jagtek.org - reverse DNS on sslhost is OK: SSL OK - wiki.jagtek.org reverse DNS resolves to cp34.miraheze.org - CNAME OK [13:25:48] RECOVERY - wiki.leapit.no - reverse DNS on sslhost is OK: SSL OK - wiki.leapit.no reverse DNS resolves to cp35.miraheze.org - CNAME OK [13:25:50] PROBLEM - mw143 HTTPS on mw143 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 1.295 second response time [13:25:53] PROBLEM - mon181 APT on mon181 is WARNING: Could not resolve hostname : Name or service not known [13:25:53] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/4c6f979d954e...056df779dc06 [13:25:54] PROBLEM - mon181 IRC Log Bot on mon181 is WARNING: Could not resolve hostname : Name or service not known [13:25:54] PROBLEM - Host mon181 is DOWN: check_ping: Invalid hostname/address - Usage:check_ping -H -w ,% -c ,% [-p packets] [-t timeout] [-4|-6] [13:25:54] PROBLEM - bast181 NTP time on bast181 is WARNING: Could not resolve hostname : Name or service not known [13:25:54] [02miraheze/ssl] 07MirahezeSSLBot 03056df77 - Bot: Update SSL cert for wiki.geoparkcorumbatai.com.br [13:25:55] RECOVERY - rockrevolution.wiki - reverse DNS on sslhost is OK: SSL OK - rockrevolution.wiki reverse DNS resolves to cp35.miraheze.org - CNAME OK [13:25:57] PROBLEM - bast181 SSH on bast181 is UNKNOWN: Usage:check_ssh [-4|-6] [-t ] [-r ] [-p ] [13:25:59] RECOVERY - kalons-reverie.com - reverse DNS on sslhost is OK: SSL OK - kalons-reverie.com reverse DNS resolves to cp34.miraheze.org - NS RECORDS OK [13:26:00] RECOVERY - db142 Puppet on db142 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:26:00] RECOVERY - steem.wiki - reverse DNS on sslhost is OK: SSL OK - steem.wiki reverse DNS resolves to cp34.miraheze.org - CNAME OK [13:26:01] PROBLEM - db182 APT on db182 is WARNING: Could not resolve hostname : Name or service not known [13:26:02] PROBLEM - db182 conntrack_table_size on db182 is WARNING: Could not resolve hostname : Name or service not known [13:26:03] PROBLEM - db182 Backups SQL icinga on db182 is WARNING: Could not resolve hostname : Name or service not known [13:26:03] PROBLEM - Host db182 is DOWN: check_ping: Invalid hostname/address - Usage:check_ping -H -w ,% -c ,% [-p packets] [-t timeout] [-4|-6] [13:26:04] RECOVERY - sarovia.graalmilitary.com - reverse DNS on sslhost is OK: SSL OK - sarovia.graalmilitary.com reverse DNS resolves to cp35.miraheze.org - CNAME OK [13:26:11] RECOVERY - iol.wiki - reverse DNS on sslhost is OK: SSL OK - iol.wiki reverse DNS resolves to cp35.miraheze.org - NS RECORDS OK [13:26:11] RECOVERY - hastursnotebook.org - LetsEncrypt on sslhost is OK: OK - Certificate 'hastursnotebook.org' will expire on Mon 04 Mar 2024 11:06:34 AM GMT +0000. [13:26:12] RECOVERY - cp41 Current Load on cp41 is OK: LOAD OK - total load average: 5.30, 5.78, 4.53 [13:26:13] PROBLEM - bast181 ferm_active on bast181 is WARNING: Could not resolve hostname : Name or service not known [13:26:13] RECOVERY - swiftobject112 Puppet on swiftobject112 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:26:13] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.68, 6.91, 6.74 [13:26:15] RECOVERY - wiki.metaiitgn.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.metaiitgn.org' will expire on Sun 17 Mar 2024 04:33:23 PM GMT +0000. [13:26:15] PROBLEM - bast181 Disk Space on bast181 is WARNING: Could not resolve hostname : Name or service not known [13:26:19] PROBLEM - bast181 Puppet on bast181 is WARNING: Could not resolve hostname : Name or service not known [13:26:19] RECOVERY - looneypyramids.wiki - reverse DNS on sslhost is OK: SSL OK - looneypyramids.wiki reverse DNS resolves to cp34.miraheze.org - NS RECORDS OK [13:26:19] RECOVERY - www.winenjoy.net - LetsEncrypt on sslhost is OK: OK - Certificate 'www.winenjoy.net' will expire on Sat 02 Mar 2024 03:55:15 AM GMT +0000. [13:26:20] RECOVERY - puritwiki.p-e.kr - LetsEncrypt on sslhost is OK: OK - Certificate 'puritwiki.p-e.kr' will expire on Sun 03 Mar 2024 04:03:07 PM GMT +0000. [13:26:23] RECOVERY - otherkin.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'otherkin.wiki' will expire on Mon 01 Apr 2024 05:37:31 PM GMT +0000. [13:26:23] RECOVERY - replication-ops.com - LetsEncrypt on sslhost is OK: OK - Certificate 'replication-ops.com' will expire on Sun 03 Mar 2024 04:29:40 PM GMT +0000. [13:26:24] RECOVERY - beidipedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'beidipedia.com' will expire on Mon 01 Apr 2024 05:35:47 PM GMT +0000. [13:26:24] PROBLEM - Host bast181 is DOWN: check_ping: Invalid hostname/address - Usage:check_ping -H -w ,% -c ,% [-p packets] [-t timeout] [-4|-6] [13:26:25] RECOVERY - wiki.kirbygang.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.kirbygang.com' will expire on Wed 13 Mar 2024 06:56:49 AM GMT +0000. [13:26:27] RECOVERY - swiftobject101 Puppet on swiftobject101 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:26:29] RECOVERY - ru.countryhumans.polandball.wiki - reverse DNS on sslhost is OK: SSL OK - ru.countryhumans.polandball.wiki reverse DNS resolves to cp34.miraheze.org - CNAME OK [13:26:29] RECOVERY - en.petrawiki.org - reverse DNS on sslhost is OK: SSL OK - en.petrawiki.org reverse DNS resolves to cp35.miraheze.org - NS RECORDS OK [13:26:29] RECOVERY - wiki.ricochetuniverse.com - reverse DNS on sslhost is OK: SSL OK - wiki.ricochetuniverse.com reverse DNS resolves to cp34.miraheze.org - CNAME OK [13:26:30] RECOVERY - vise.dayid.org - reverse DNS on sslhost is OK: SSL OK - vise.dayid.org reverse DNS resolves to cp34.miraheze.org - CNAME OK [13:26:36] RECOVERY - wiki.mikrodev.com - reverse DNS on sslhost is OK: SSL OK - wiki.mikrodev.com reverse DNS resolves to cp35.miraheze.org - CNAME OK [13:26:37] RECOVERY - swiftobject113 Puppet on swiftobject113 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:26:39] RECOVERY - airlineinsider.org - LetsEncrypt on sslhost is OK: OK - Certificate 'airlineinsider.org' will expire on Fri 05 Apr 2024 07:18:27 AM GMT +0000. [13:26:42] RECOVERY - wiki.pixlies.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.pixlies.net' will expire on Sat 02 Mar 2024 06:14:09 PM GMT +0000. [13:26:50] RECOVERY - wiki.strangereons.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.strangereons.com' will expire on Tue 16 Apr 2024 06:15:07 PM GMT +0000. [13:26:55] RECOVERY - kirisame-kissaten.cf - LetsEncrypt on sslhost is OK: OK - Certificate 'kirisame-kissaten.cf' will expire on Sun 10 Mar 2024 01:53:38 PM GMT +0000. [13:26:55] RECOVERY - rarewarewiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'rarewarewiki.com' will expire on Sun 03 Mar 2024 07:39:38 PM GMT +0000. [13:26:55] RECOVERY - www.hoolehistoryheritagesociety.org.uk - reverse DNS on sslhost is OK: SSL OK - www.hoolehistoryheritagesociety.org.uk reverse DNS resolves to cp34.miraheze.org - CNAME OK [13:26:56] RECOVERY - wiki.creaturathegame.com - reverse DNS on sslhost is OK: SSL OK - wiki.creaturathegame.com reverse DNS resolves to cp35.miraheze.org - CNAME OK [13:27:01] RECOVERY - persist.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'persist.wiki' will expire on Sun 03 Mar 2024 02:12:41 AM GMT +0000. [13:27:01] RECOVERY - wiki.corgicam.tv - reverse DNS on sslhost is OK: SSL OK - wiki.corgicam.tv reverse DNS resolves to cp34.miraheze.org - CNAME OK [13:27:03] RECOVERY - apocrypha.wiki - reverse DNS on sslhost is OK: SSL OK - apocrypha.wiki reverse DNS resolves to cp34.miraheze.org - NS RECORDS OK [13:27:03] RECOVERY - www.istpcomputing.com - reverse DNS on sslhost is OK: SSL OK - www.istpcomputing.com reverse DNS resolves to cp34.miraheze.org - NS RECORDS OK [13:27:07] RECOVERY - miraheze.gq - reverse DNS on sslhost is OK: SSL OK - miraheze.gq reverse DNS resolves to cp34.miraheze.org - NS RECORDS OK [13:27:09] RECOVERY - wiki.luemir.xyz - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.luemir.xyz' will expire on Sat 02 Mar 2024 06:52:08 PM GMT +0000. [13:27:09] RECOVERY - wiki.tmyt105.leyhp.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.tmyt105.leyhp.com' will expire on Wed 17 Apr 2024 02:04:14 PM GMT +0000. [13:27:09] RECOVERY - www.pyramidgames.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'pyramidgames.wiki' will expire on Sat 13 Apr 2024 01:26:13 PM GMT +0000. [13:27:10] RECOVERY - chobots.wiki - reverse DNS on sslhost is OK: SSL OK - chobots.wiki reverse DNS resolves to cp35.miraheze.org - CNAME FLAT [13:27:11] RECOVERY - 321nailswiki.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Mon 18 Nov 2024 11:59:59 PM GMT +0000. [13:27:12] RECOVERY - gogigantic.wiki - reverse DNS on sslhost is OK: SSL OK - gogigantic.wiki reverse DNS resolves to cp34.miraheze.org - CNAME OK [13:27:12] RECOVERY - uk.religiononfire.mar.in.ua - LetsEncrypt on sslhost is OK: OK - Certificate 'uk.religiononfire.mar.in.ua' will expire on Sat 23 Mar 2024 04:20:14 PM GMT +0000. [13:27:13] RECOVERY - fanon.polandballwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'fanon.polandballwiki.com' will expire on Sat 02 Mar 2024 04:21:02 PM GMT +0000. [13:27:16] RECOVERY - sims.miraheze.org - reverse DNS on sslhost is OK: SSL OK - sims.miraheze.org reverse DNS resolves to cp34.miraheze.org - NS RECORDS OK [13:27:19] RECOVERY - metroidpedia.com - reverse DNS on sslhost is OK: SSL OK - metroidpedia.com reverse DNS resolves to cp34.miraheze.org - NS RECORDS OK [13:27:20] RECOVERY - fid.koymi.net - LetsEncrypt on sslhost is OK: OK - Certificate 'fid.koymi.net' will expire on Fri 16 Feb 2024 07:19:57 PM GMT +0000. [13:27:21] RECOVERY - www.pastport.org - LetsEncrypt on sslhost is OK: OK - Certificate 'pastport.org' will expire on Sat 02 Mar 2024 07:28:11 PM GMT +0000. [13:27:24] PROBLEM - cp34 Varnish Backends on cp34 is CRITICAL: 2 backends are down. mw141 mw143 [13:27:26] RECOVERY - archives.nsgov.org - reverse DNS on sslhost is OK: SSL OK - archives.nsgov.org reverse DNS resolves to cp35.miraheze.org - CNAME OK [13:27:28] RECOVERY - wiki.strangereons.com - reverse DNS on sslhost is OK: SSL OK - wiki.strangereons.com reverse DNS resolves to cp35.miraheze.org - CNAME OK [13:27:30] PROBLEM - mw141 HTTPS on mw141 is CRITICAL: HTTP CRITICAL: HTTP/2 502 - 170 bytes in 9.342 second response time [13:27:34] RECOVERY - data.wikiyri.org - reverse DNS on sslhost is OK: SSL OK - data.wikiyri.org reverse DNS resolves to cp34.miraheze.org - NS RECORDS OK [13:27:34] RECOVERY - encyclopediarobotica.org - LetsEncrypt on sslhost is OK: OK - Certificate 'encyclopediarobotica.org' will expire on Thu 18 Apr 2024 05:05:13 PM GMT +0000. [13:27:35] RECOVERY - wiki.triplescripts.org - reverse DNS on sslhost is OK: SSL OK - wiki.triplescripts.org reverse DNS resolves to cp35.miraheze.org - CNAME OK [13:27:35] RECOVERY - istpcomputing.com - LetsEncrypt on sslhost is OK: OK - Certificate 'istpcomputing.com' will expire on Sun 03 Mar 2024 04:19:48 PM GMT +0000. [13:27:40] RECOVERY - null-cpu.emudev.org - LetsEncrypt on sslhost is OK: OK - Certificate 'null-cpu.emudev.org' will expire on Wed 13 Mar 2024 07:14:41 AM GMT +0000. [13:27:41] RECOVERY - Host matomo151 is UP: PING OK - Packet loss = 0%, RTA = 0.19 ms [13:27:43] RECOVERY - worldtobuild.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'worldtobuild.wiki' will expire on Tue 26 Mar 2024 09:24:34 PM GMT +0000. [13:27:43] RECOVERY - wiki.meower.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.meower.org' will expire on Sat 02 Mar 2024 07:00:21 PM GMT +0000. [13:27:44] RECOVERY - enc.for.uz - reverse DNS on sslhost is OK: SSL OK - enc.for.uz reverse DNS resolves to cp35.miraheze.org - NS RECORDS OK [13:27:47] RECOVERY - Host db151 is UP: PING OK - Packet loss = 0%, RTA = 0.90 ms [13:27:49] RECOVERY - sonicpedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'sonicpedia.org' will expire on Sat 06 Apr 2024 11:41:38 AM GMT +0000. [13:27:51] RECOVERY - wiki.k2cho.me - reverse DNS on sslhost is OK: SSL OK - wiki.k2cho.me reverse DNS resolves to cp34.miraheze.org - CNAME OK [13:27:51] RECOVERY - www.cgradegames.net - reverse DNS on sslhost is OK: SSL OK - www.cgradegames.net reverse DNS resolves to cp35.miraheze.org - CNAME OK [13:27:55] RECOVERY - projectsekai.miraheze.org - reverse DNS on sslhost is OK: SSL OK - projectsekai.miraheze.org reverse DNS resolves to cp34.miraheze.org - NS RECORDS OK [13:28:00] RECOVERY - wiki.aetherexplorers.science - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.aetherexplorers.science' will expire on Wed 10 Apr 2024 12:07:55 PM GMT +0000. [13:28:02] PROBLEM - mw143 HTTPS on mw143 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [13:28:15] RECOVERY - nonciclopedia.org - reverse DNS on sslhost is OK: SSL OK - nonciclopedia.org reverse DNS resolves to cp35.miraheze.org - NS RECORDS OK [13:28:17] RECOVERY - Host mw152 is UP: PING OK - Packet loss = 0%, RTA = 0.29 ms [13:28:24] PROBLEM - ping6 on matomo151 is UNKNOWN: check_ping: Invalid hostname/address - 10.0.15.112Usage:check_ping -H -w ,% -c ,% [-p packets] [-t timeout] [-4|-6] [13:28:34] RECOVERY - crocwiki.com - reverse DNS on sslhost is OK: SSL OK - crocwiki.com reverse DNS resolves to cp34.miraheze.org - NS RECORDS OK [13:28:36] RECOVERY - wiki.ivy.cm - reverse DNS on sslhost is OK: SSL OK - wiki.ivy.cm reverse DNS resolves to cp34.miraheze.org - CNAME OK [13:28:39] RECOVERY - evil.wiki - reverse DNS on sslhost is OK: SSL OK - evil.wiki reverse DNS resolves to cp34.miraheze.org - CNAME OK [13:28:40] RECOVERY - www.christipedia.nl - LetsEncrypt on sslhost is OK: OK - Certificate 'www.christipedia.nl' will expire on Sat 02 Mar 2024 01:52:01 AM GMT +0000. [13:28:41] RECOVERY - wiki.esnmilanostatale.it - reverse DNS on sslhost is OK: SSL OK - wiki.esnmilanostatale.it reverse DNS resolves to cp35.miraheze.org - CNAME OK [13:28:42] PROBLEM - mw142 HTTPS on mw142 is CRITICAL: HTTP CRITICAL: HTTP/2 502 - 170 bytes in 0.715 second response time [13:28:46] RECOVERY - apeirology.com - LetsEncrypt on sslhost is OK: OK - Certificate 'apeirology.com' will expire on Sun 03 Mar 2024 07:27:09 PM GMT +0000. [13:28:49] PROBLEM - matomo151 Redis Process on matomo151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:28:51] PROBLEM - with.cpt-ra.bid - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - with.cpt-ra.bid All nameservers failed to answer the query. [13:28:51] RECOVERY - www.moiswiki.net - reverse DNS on sslhost is OK: SSL OK - www.moiswiki.net reverse DNS resolves to cp34.miraheze.org - CNAME OK [13:28:54] PROBLEM - matomo151 SSH on matomo151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:28:58] RECOVERY - crashspyro.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Mon 18 Nov 2024 11:59:59 PM GMT +0000. [13:28:59] PROBLEM - db151 SSH on db151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:28:59] RECOVERY - en.wikiyri.org - LetsEncrypt on sslhost is OK: OK - Certificate 'en.wikiyri.org' will expire on Thu 14 Mar 2024 06:58:38 PM GMT +0000. [13:29:02] RECOVERY - Host mem151 is UP: PING OK - Packet loss = 0%, RTA = 0.26 ms [13:29:04] PROBLEM - db151 Puppet on db151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:29:07] RECOVERY - Host prometheus151 is UP: PING OK - Packet loss = 0%, RTA = 0.32 ms [13:29:08] RECOVERY - wiki.otir.nl - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.otir.nl' will expire on Sun 03 Mar 2024 02:28:30 PM GMT +0000. [13:29:10] RECOVERY - wiki.insideearth.info - reverse DNS on sslhost is OK: SSL OK - wiki.insideearth.info reverse DNS resolves to cp34.miraheze.org - CNAME OK [13:29:13] RECOVERY - www.rothwell-leeds.co.uk - reverse DNS on sslhost is OK: SSL OK - www.rothwell-leeds.co.uk reverse DNS resolves to cp35.miraheze.org - NS RECORDS OK [13:29:13] RECOVERY - wiki.worldbuilding.network - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.worldbuilding.network' will expire on Sat 02 Mar 2024 11:53:08 AM GMT +0000. [13:29:14] RECOVERY - wiki.starshipsailing.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.starshipsailing.net' will expire on Sat 02 Mar 2024 07:38:35 PM GMT +0000. [13:29:14] RECOVERY - stablestate.org - LetsEncrypt on sslhost is OK: OK - Certificate 'stablestate.org' will expire on Fri 01 Mar 2024 10:54:06 AM GMT +0000. [13:29:16] RECOVERY - wiki.yumeka.icu - reverse DNS on sslhost is OK: SSL OK - wiki.yumeka.icu reverse DNS resolves to cp35.miraheze.org - CNAME OK [13:29:16] RECOVERY - Host os151 is UP: PING OK - Packet loss = 0%, RTA = 0.26 ms [13:29:17] RECOVERY - infectowiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'infectowiki.com' will expire on Fri 01 Mar 2024 06:29:41 PM GMT +0000. [13:29:20] RECOVERY - portalsofphereon.com - reverse DNS on sslhost is OK: SSL OK - portalsofphereon.com reverse DNS resolves to cp34.miraheze.org - NS RECORDS OK [13:29:20] RECOVERY - hastursnotebook.org - reverse DNS on sslhost is OK: SSL OK - hastursnotebook.org reverse DNS resolves to cp35.miraheze.org - NS RECORDS OK [13:29:20] RECOVERY - monarchists.wiki - reverse DNS on sslhost is OK: SSL OK - monarchists.wiki reverse DNS resolves to cp34.miraheze.org - NS RECORDS OK [13:29:21] RECOVERY - journeytheword.wiki - reverse DNS on sslhost is OK: SSL OK - journeytheword.wiki reverse DNS resolves to cp35.miraheze.org - NS RECORDS OK [13:29:23] RECOVERY - wiki.knowledgerevolution.eu - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.knowledgerevolution.eu' will expire on Thu 14 Mar 2024 06:20:44 PM GMT +0000. [13:29:25] RECOVERY - Host mw151 is UP: PING OK - Packet loss = 0%, RTA = 0.64 ms [13:29:25] RECOVERY - files.petrawiki.org - reverse DNS on sslhost is OK: SSL OK - files.petrawiki.org reverse DNS resolves to cp34.miraheze.org - NS RECORDS OK [13:29:27] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/813de7b2983d...dfeab5b92c3f [13:29:27] RECOVERY - miraheze.cf - LetsEncrypt on sslhost is OK: OK - Certificate 'miraheze.cf' will expire on Sun 03 Mar 2024 07:46:34 PM GMT +0000. [13:29:28] RECOVERY - wiki.themarkside.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.themarkside.com' will expire on Fri 16 Feb 2024 07:22:49 PM GMT +0000. [13:29:30] [02miraheze/puppet] 07paladox 03dfeab5b - Fix [13:29:31] RECOVERY - patternarchive.online - reverse DNS on sslhost is OK: SSL OK - patternarchive.online reverse DNS resolves to cp34.miraheze.org - NS RECORDS OK [13:29:31] RECOVERY - www.bluepageswiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'www.bluepageswiki.org' will expire on Fri 01 Mar 2024 06:34:16 PM GMT +0000. [13:29:33] RECOVERY - podpedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'podpedia.org' will expire on Fri 01 Mar 2024 12:29:40 PM GMT +0000. [13:29:34] PROBLEM - matomo151 conntrack_table_size on matomo151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:29:34] PROBLEM - ping6 on mw151 is UNKNOWN: check_ping: Invalid hostname/address - 10.0.15.114Usage:check_ping -H -w ,% -c ,% [-p packets] [-t timeout] [-4|-6] [13:29:38] RECOVERY - wiki.closai.com - reverse DNS on sslhost is OK: SSL OK - wiki.closai.com reverse DNS resolves to cp34.miraheze.org - CNAME OK [13:29:38] RECOVERY - cities.simulz.kr - reverse DNS on sslhost is OK: SSL OK - cities.simulz.kr reverse DNS resolves to cp35.miraheze.org - CNAME OK [13:29:38] RECOVERY - wiki.walkscape.app - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.walkscape.app' will expire on Thu 21 Mar 2024 01:13:18 PM GMT +0000. [13:29:38] RECOVERY - witchingworld.org - LetsEncrypt on sslhost is OK: OK - Certificate 'witchingworld.org' will expire on Sun 03 Mar 2024 09:11:26 PM GMT +0000. [13:29:39] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 6.36, 6.76, 6.10 [13:29:39] PROBLEM - ping6 on mw152 is UNKNOWN: check_ping: Invalid hostname/address - 10.0.15.115Usage:check_ping -H -w ,% -c ,% [-p packets] [-t timeout] [-4|-6] [13:29:42] RECOVERY - civwiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'civwiki.org' will expire on Sat 02 Mar 2024 02:36:54 AM GMT +0000. [13:29:43] RECOVERY - wiki.aoemods.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.aoemods.com' will expire on Sun 03 Mar 2024 03:01:04 AM GMT +0000. [13:29:44] PROBLEM - matomo151 php-fpm on matomo151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:29:45] RECOVERY - wiki.gestaltonline.ca - reverse DNS on sslhost is OK: SSL OK - wiki.gestaltonline.ca reverse DNS resolves to cp35.miraheze.org - CNAME OK [13:29:46] RECOVERY - wiki.mcjones.gay - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.mcjones.gay' will expire on Wed 06 Mar 2024 02:20:55 AM GMT +0000. [13:29:46] PROBLEM - mw181 MediaWiki Rendering on mw181 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:29:47] RECOVERY - squareenixwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'squareenixwiki.com' will expire on Sun 03 Mar 2024 03:25:04 AM GMT +0000. [13:29:49] PROBLEM - matomo151 APT on matomo151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:29:49] RECOVERY - wiki.tmyt105.leyhp.com - reverse DNS on sslhost is OK: SSL OK - wiki.tmyt105.leyhp.com reverse DNS resolves to cp34.miraheze.org - CNAME OK [13:29:50] RECOVERY - Host test151 is UP: PING OK - Packet loss = 0%, RTA = 0.44 ms [13:29:51] RECOVERY - www.winenjoy.net - reverse DNS on sslhost is OK: SSL OK - www.winenjoy.net reverse DNS resolves to cp34.miraheze.org - CNAME OK [13:29:51] RECOVERY - www.tintinpedia.fr - reverse DNS on sslhost is OK: SSL OK - www.tintinpedia.fr reverse DNS resolves to cp34.miraheze.org - NS RECORDS OK [13:29:51] RECOVERY - pubwiki.lab.co.pl - LetsEncrypt on sslhost is OK: OK - Certificate 'pubwiki.lab.co.pl' will expire on Fri 01 Mar 2024 08:39:02 PM GMT +0000. [13:29:54] PROBLEM - ping6 on db151 is UNKNOWN: check_ping: Invalid hostname/address - 10.0.15.110Usage:check_ping -H -w ,% -c ,% [-p packets] [-t timeout] [-4|-6] [13:29:54] PROBLEM - ping6 on mem151 is UNKNOWN: check_ping: Invalid hostname/address - 10.0.15.113Usage:check_ping -H -w ,% -c ,% [-p packets] [-t timeout] [-4|-6] [13:29:55] RECOVERY - wiki.manos.pt - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.manos.pt' will expire on Mon 26 Feb 2024 11:06:06 PM GMT +0000. [13:29:56] RECOVERY - wiki.bikinibottomdiscord.org - reverse DNS on sslhost is OK: SSL OK - wiki.bikinibottomdiscord.org reverse DNS resolves to cp34.miraheze.org - CNAME OK [13:29:57] RECOVERY - wmworld.sktz.live - LetsEncrypt on sslhost is OK: OK - Certificate 'wmworld.sktz.live' will expire on Sat 02 Mar 2024 04:27:41 AM GMT +0000. [13:29:58] RECOVERY - www.thesimswiki.com - reverse DNS on sslhost is OK: SSL OK - www.thesimswiki.com reverse DNS resolves to cp34.miraheze.org - CNAME OK [13:29:59] PROBLEM - db151 PowerDNS Recursor on db151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:30:00] RECOVERY - pastport.org - reverse DNS on sslhost is OK: SSL OK - pastport.org reverse DNS resolves to cp34.miraheze.org - NS RECORDS OK [13:30:01] RECOVERY - www.documentopublico.com.br - reverse DNS on sslhost is OK: SSL OK - www.documentopublico.com.br reverse DNS resolves to cp35.miraheze.org - CNAME OK [13:30:04] RECOVERY - mw141 Puppet on mw141 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [13:30:05] RECOVERY - staratlaswiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'staratlaswiki.org' will expire on Sat 02 Mar 2024 11:29:11 AM GMT +0000. [13:30:06] RECOVERY - kingdomdeath.wiki - reverse DNS on sslhost is OK: SSL OK - kingdomdeath.wiki reverse DNS resolves to cp35.miraheze.org - NS RECORDS OK [13:30:06] RECOVERY - encyclopediarobotica.org - reverse DNS on sslhost is OK: SSL OK - encyclopediarobotica.org reverse DNS resolves to cp35.miraheze.org - NS RECORDS OK [13:30:06] RECOVERY - reviwiki.info - reverse DNS on sslhost is OK: SSL OK - reviwiki.info reverse DNS resolves to cp35.miraheze.org - CNAME FLAT [13:30:12] RECOVERY - icclopedia.org - reverse DNS on sslhost is OK: SSL OK - icclopedia.org reverse DNS resolves to cp34.miraheze.org - NS RECORDS OK [13:30:12] RECOVERY - wiki.thesimswiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.thesimswiki.com' will expire on Mon 19 Feb 2024 02:16:01 AM GMT +0000. [13:30:13] PROBLEM - mw141 HTTPS on mw141 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 9.961 second response time [13:30:14] PROBLEM - matomo151 Disk Space on matomo151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:30:14] RECOVERY - wiki.cyberfurs.org - reverse DNS on sslhost is OK: SSL OK - wiki.cyberfurs.org reverse DNS resolves to cp34.miraheze.org - CNAME OK [13:30:16] RECOVERY - wiki.cityrp.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.cityrp.org' will expire on Sun 17 Mar 2024 04:34:46 PM GMT +0000. [13:30:16] RECOVERY - wiki.cityrp.org - reverse DNS on sslhost is OK: SSL OK - wiki.cityrp.org reverse DNS resolves to cp34.miraheze.org - CNAME OK [13:30:16] RECOVERY - dlfm-wiki.top - reverse DNS on sslhost is OK: SSL OK - dlfm-wiki.top reverse DNS resolves to cp35.miraheze.org - CNAME OK [13:30:17] RECOVERY - vise.dayid.org - LetsEncrypt on sslhost is OK: OK - Certificate 'vise.dayid.org' will expire on Fri 01 Mar 2024 01:51:54 PM GMT +0000. [13:30:17] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 9.17, 11.66, 10.99 [13:30:18] RECOVERY - familiacorsi.com - reverse DNS on sslhost is OK: SSL OK - familiacorsi.com reverse DNS resolves to cp35.miraheze.org - NS RECORDS OK [13:30:19] PROBLEM - ping6 on os151 is UNKNOWN: check_ping: Invalid hostname/address - 10.0.15.111Usage:check_ping -H -w ,% -c ,% [-p packets] [-t timeout] [-4|-6] [13:30:21] PROBLEM - cp24 Varnish Backends on cp24 is CRITICAL: 2 backends are down. mw141 mw143 [13:30:22] RECOVERY - creativecommons.id - LetsEncrypt on sslhost is OK: OK - Certificate 'creativecommons.id' will expire on Sun 03 Mar 2024 09:12:54 PM GMT +0000. [13:30:22] RECOVERY - fc.songcontests.eu - LetsEncrypt on sslhost is OK: OK - Certificate 'fc.songcontests.eu' will expire on Fri 01 Mar 2024 08:20:19 PM GMT +0000. [13:30:23] RECOVERY - crustypedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'crustypedia.org' will expire on Sat 30 Mar 2024 07:56:17 AM GMT +0000. [13:30:23] RECOVERY - www.mcpk.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'www.mcpk.wiki' will expire on Fri 01 Mar 2024 06:15:15 PM GMT +0000. [13:30:24] PROBLEM - os151 SSH on os151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:30:24] PROBLEM - matomo151 ferm_active on matomo151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:30:24] PROBLEM - os151 Current Load on os151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:30:25] RECOVERY - antiguabarbudacalypso.com - LetsEncrypt on sslhost is OK: OK - Certificate 'antiguabarbudacalypso.com' will expire on Sat 02 Mar 2024 09:33:49 PM GMT +0000. [13:30:26] RECOVERY - wiki.aridia.space - reverse DNS on sslhost is OK: SSL OK - wiki.aridia.space reverse DNS resolves to cp34.miraheze.org - CNAME OK [13:30:26] RECOVERY - nijigen.wiki - reverse DNS on sslhost is OK: SSL OK - nijigen.wiki reverse DNS resolves to cp35.miraheze.org - CNAME OK [13:30:26] PROBLEM - mw141 Current Load on mw141 is CRITICAL: LOAD CRITICAL - total load average: 16.24, 15.47, 11.70 [13:30:39] PROBLEM - db151 APT on db151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:30:40] PROBLEM - cp35 Varnish Backends on cp35 is CRITICAL: 2 backends are down. mw141 mw143 [13:30:42] RECOVERY - iceria.org - LetsEncrypt on sslhost is OK: OK - Certificate 'www.iceria.org' will expire on Fri 01 Mar 2024 07:52:36 PM GMT +0000. [13:30:42] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.20, 7.43, 6.99 [13:30:43] RECOVERY - fanonpedia.com - LetsEncrypt on sslhost is OK: OK - Certificate 'fanonpedia.com' will expire on Mon 25 Mar 2024 04:05:42 PM GMT +0000. [13:30:44] PROBLEM - matomo151 Puppet on matomo151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:30:45] PROBLEM - itadminwiki.pl - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'itadminwiki.pl' expires in 8 day(s) (Fri 02 Feb 2024 09:29:29 AM GMT +0000). [13:30:48] PROBLEM - mw143 MediaWiki Rendering on mw143 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:30:49] PROBLEM - os151 NTP time on os151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:31:19] PROBLEM - matomo151 NTP time on matomo151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:31:24] PROBLEM - db151 Current Load on db151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:31:34] PROBLEM - db151 conntrack_table_size on db151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:35:45] RECOVERY - wiki.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Mon 18 Nov 2024 11:59:59 PM GMT +0000. [13:35:45] RECOVERY - sona.pona.la - reverse DNS on sslhost is OK: SSL OK - sona.pona.la reverse DNS resolves to cp34.miraheze.org - CNAME OK [13:35:46] RECOVERY - wiki.songngu.xyz - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.songngu.xyz' will expire on Tue 02 Apr 2024 01:21:12 PM GMT +0000. [13:35:46] RECOVERY - wiki.yuanpi.eu.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.yuanpi.eu.org' will expire on Sun 18 Feb 2024 10:10:37 AM GMT +0000. [13:35:46] RECOVERY - evilgeniuswiki.com - reverse DNS on sslhost is OK: SSL OK - evilgeniuswiki.com reverse DNS resolves to cp35.miraheze.org - NS RECORDS OK [13:35:46] RECOVERY - www.programming.red - reverse DNS on sslhost is OK: SSL OK - www.programming.red reverse DNS resolves to cp34.miraheze.org - NS RECORDS OK [13:35:46] RECOVERY - www.widedata.org - reverse DNS on sslhost is OK: SSL OK - www.widedata.org reverse DNS resolves to cp35.miraheze.org - NS RECORDS OK [13:35:47] RECOVERY - wiki.teessidehackspace.org.uk - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.teessidehackspace.org.uk' will expire on Sat 02 Mar 2024 05:56:22 PM GMT +0000. [13:35:47] RECOVERY - wiki.zamnhacking.net - reverse DNS on sslhost is OK: SSL OK - wiki.zamnhacking.net reverse DNS resolves to cp35.miraheze.org - CNAME OK [13:35:48] RECOVERY - private.yahyabd.xyz - LetsEncrypt on sslhost is OK: OK - Certificate 'private.yahyabd.xyz' will expire on Sun 03 Mar 2024 09:50:49 PM GMT +0000. [13:40:07] RECOVERY - data.nonbinary.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'data.nonbinary.wiki' will expire on Fri 01 Mar 2024 12:44:09 PM GMT +0000. [13:40:08] RECOVERY - segawiki.com - reverse DNS on sslhost is OK: SSL OK - segawiki.com reverse DNS resolves to cp34.miraheze.org - NS RECORDS OK [13:40:08] RECOVERY - www.lab612.at - reverse DNS on sslhost is OK: SSL OK - www.lab612.at reverse DNS resolves to cp34.miraheze.org - CNAME OK [13:40:08] RECOVERY - wiki.rosestulipsandliberty.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.rosestulipsandliberty.com' will expire on Mon 01 Apr 2024 05:40:02 PM GMT +0000. [13:40:08] RECOVERY - www.iceria.org - LetsEncrypt on sslhost is OK: OK - Certificate 'www.iceria.org' will expire on Fri 01 Mar 2024 07:52:36 PM GMT +0000. [13:40:08] RECOVERY - wiki.veloren.net - reverse DNS on sslhost is OK: SSL OK - wiki.veloren.net reverse DNS resolves to cp34.miraheze.org - CNAME OK [13:40:08] RECOVERY - ru-teirailway.f5.si - LetsEncrypt on sslhost is OK: OK - Certificate 'ru-teirailway.f5.si' will expire on Wed 03 Apr 2024 05:01:28 PM GMT +0000. [13:40:09] RECOVERY - acgn.sfdev.eu.org - reverse DNS on sslhost is OK: SSL OK - acgn.sfdev.eu.org reverse DNS resolves to cp35.miraheze.org - CNAME OK [13:40:09] RECOVERY - archive.a2b2.org - LetsEncrypt on sslhost is OK: OK - Certificate 'archive.a2b2.org' will expire on Sat 02 Mar 2024 03:53:54 AM GMT +0000. [13:40:10] RECOVERY - ao90.pinho.org - reverse DNS on sslhost is OK: SSL OK - ao90.pinho.org reverse DNS resolves to cp34.miraheze.org - CNAME OK [13:40:10] RECOVERY - trollpasta.com - LetsEncrypt on sslhost is OK: OK - Certificate 'trollpasta.com' will expire on Sat 02 Mar 2024 02:16:33 AM GMT +0000. [13:40:11] RECOVERY - wiki.worldbuilding.network - reverse DNS on sslhost is OK: SSL OK - wiki.worldbuilding.network reverse DNS resolves to cp34.miraheze.org - CNAME OK [13:41:01] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/dfeab5b92c3f...948661c62b07 [13:41:02] [02miraheze/puppet] 07paladox 03948661c - nginx::site: support private network [13:43:57] PROBLEM - bast181 Disk Space on bast181 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:43:57] RECOVERY - robloxapi.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'robloxapi.wiki' will expire on Sat 02 Mar 2024 11:19:24 AM GMT +0000. [13:44:08] PROBLEM - db181 MariaDB Connections on db181 is UNKNOWN: PHP Fatal error: Uncaught mysqli_sql_exception: Connection timed out in /usr/lib/nagios/plugins/check_mysql_connections.php:66Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(66): mysqli_real_connect(Object(mysqli), 'db181.wikitide....', 'icinga', Object(SensitiveParameterValue), NULL, NULL, NULL, true)#1 {main} thrown in /usr/lib/nagios/plugins/check_mysql_connection [13:44:08] on line 66Fatal error: Uncaught mysqli_sql_exception: Connection timed out in /usr/lib/nagios/plugins/check_mysql_connections.php:66Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(66): mysqli_real_connect(Object(mysqli), 'db181.wikitide....', 'icinga', Object(SensitiveParameterValue), NULL, NULL, NULL, true)#1 {main} thrown in /usr/lib/nagios/plugins/check_mysql_connections.php on line 66 [13:44:11] PROBLEM - db182 Backups SQL roundcubemail on db182 is CRITICAL: connect to address 10.0.18.103 port 5666: Connection refusedconnect to host 10.0.18.103 port 5666: Connection refused [13:44:13] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: connect to address 10.0.18.106 port 5666: Connection refusedconnect to host 10.0.18.106 port 5666: Connection refused [13:44:16] PROBLEM - db182 Puppet on db182 is CRITICAL: connect to address 10.0.18.103 port 5666: Connection refusedconnect to host 10.0.18.103 port 5666: Connection refused [13:44:16] PROBLEM - db181 PowerDNS Recursor on db181 is CRITICAL: connect to address 10.0.18.102 port 5666: Connection refusedconnect to host 10.0.18.102 port 5666: Connection refused [13:44:16] PROBLEM - mw151 conntrack_table_size on mw151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:44:21] PROBLEM - db181 Backups SQL mhglobal on db181 is CRITICAL: connect to address 10.0.18.102 port 5666: Connection refusedconnect to host 10.0.18.102 port 5666: Connection refused [13:44:26] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 0.79, 2.34, 3.51 [13:44:39] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.57, 7.89, 7.43 [13:45:13] PROBLEM - mw182 MediaWiki Rendering on mw182 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:45:17] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/948661c62b07...a9444231c0ee [13:45:18] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 7.83, 8.35, 7.93 [13:45:20] [02miraheze/puppet] 07paladox 03a944423 - bastion: add support for private network [13:45:21] PROBLEM - db182 Current Load on db182 is CRITICAL: connect to address 10.0.18.103 port 5666: Connection refusedconnect to host 10.0.18.103 port 5666: Connection refused [13:45:21] PROBLEM - bast181 NTP time on bast181 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:45:21] PROBLEM - db181 conntrack_table_size on db181 is CRITICAL: connect to address 10.0.18.102 port 5666: Connection refusedconnect to host 10.0.18.102 port 5666: Connection refused [13:45:25] PROBLEM - ping6 on bast181 is UNKNOWN: check_ping: Invalid hostname/address - 10.0.18.101Usage:check_ping -H -w ,% -c ,% [-p packets] [-t timeout] [-4|-6] [13:45:26] PROBLEM - bast181 Current Load on bast181 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:45:32] RECOVERY - db151 SSH on db151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u2 (protocol 2.0) [13:45:36] PROBLEM - mwtask181 Disk Space on mwtask181 is CRITICAL: connect to address 10.0.18.106 port 5666: Connection refusedconnect to host 10.0.18.106 port 5666: Connection refused [13:45:36] PROBLEM - db181 Current Load on db181 is CRITICAL: connect to address 10.0.18.102 port 5666: Connection refusedconnect to host 10.0.18.102 port 5666: Connection refused [13:45:36] RECOVERY - prometheus151 Prometheus on prometheus151 is OK: TCP OK - 0.000 second response time on 10.0.15.116 port 9090 [13:45:36] PROBLEM - ping6 on swiftobject181 is UNKNOWN: check_ping: Invalid hostname/address - 10.0.18.107Usage:check_ping -H -w ,% -c ,% [-p packets] [-t timeout] [-4|-6] [13:45:36] PROBLEM - swiftproxy181 Current Load on swiftproxy181 is WARNING: Could not resolve hostname : Name or service not known [13:45:36] PROBLEM - swiftproxy181 PowerDNS Recursor on swiftproxy181 is WARNING: Could not resolve hostname : Name or service not known [13:45:37] PROBLEM - puppet181 MirahezeRenewSsl on puppet181 is CRITICAL: connect to address fe80::be24:11ff:fe6f:ef62 and port 5000: Invalid argument [13:45:37] PROBLEM - mwtask181 MediaWiki Rendering on mwtask181 is CRITICAL: Name or service not knownHTTP CRITICAL - Unable to open TCP socket [13:45:38] PROBLEM - db142 MariaDB Connections on db142 is UNKNOWN: PHP Fatal error: Uncaught mysqli_sql_exception: Connection timed out in /usr/lib/nagios/plugins/check_mysql_connections.php:66Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(66): mysqli_real_connect(Object(mysqli), 'db142.miraheze....', 'icinga', Object(SensitiveParameterValue), NULL, NULL, NULL, true)#1 {main} thrown in /usr/lib/nagios/plugins/check_mysql_connection [13:45:38] on line 66Fatal error: Uncaught mysqli_sql_exception: Connection timed out in /usr/lib/nagios/plugins/check_mysql_connections.php:66Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(66): mysqli_real_connect(Object(mysqli), 'db142.miraheze....', 'icinga', Object(SensitiveParameterValue), NULL, NULL, NULL, true)#1 {main} thrown in /usr/lib/nagios/plugins/check_mysql_connections.php on line 66 [13:45:46] PROBLEM - bast181 conntrack_table_size on bast181 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:45:50] PROBLEM - phab121 phab.miraheze.wiki HTTPS on phab121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:45:50] PROBLEM - graylog131 HTTPS on graylog131 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Connection timed out after 10003 milliseconds [13:45:50] PROBLEM - mon181 HTTPS on mon181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 7 - Failed to connect to mon181.wikitide.net port 443 after 0 ms: Couldn't connect to server [13:45:50] PROBLEM - mwtask181 conntrack_table_size on mwtask181 is CRITICAL: connect to address 10.0.18.106 port 5666: Connection refusedconnect to host 10.0.18.106 port 5666: Connection refused [13:45:50] PROBLEM - phab121 phabricator.miraheze.org HTTPS on phab121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:45:51] PROBLEM - swiftproxy131 Swift Proxy on swiftproxy131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:45:51] PROBLEM - mw141 MediaWiki Rendering on mw141 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:45:51] PROBLEM - swiftproxy181 Disk Space on swiftproxy181 is WARNING: Could not resolve hostname : Name or service not known [13:45:54] PROBLEM - swiftproxy181 APT on swiftproxy181 is WARNING: Could not resolve hostname : Name or service not known [13:45:56] PROBLEM - db181 Disk Space on db181 is CRITICAL: connect to address 10.0.18.102 port 5666: Connection refusedconnect to host 10.0.18.102 port 5666: Connection refused [13:45:59] PROBLEM - mwtask181 JobRunner Service on mwtask181 is CRITICAL: connect to address 10.0.18.106 port 5666: Connection refusedconnect to host 10.0.18.106 port 5666: Connection refused [13:46:00] PROBLEM - db182 MariaDB Connections on db182 is UNKNOWN: PHP Fatal error: Uncaught mysqli_sql_exception: Connection timed out in /usr/lib/nagios/plugins/check_mysql_connections.php:66Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(66): mysqli_real_connect(Object(mysqli), 'db182.wikitide....', 'icinga', Object(SensitiveParameterValue), NULL, NULL, NULL, true)#1 {main} thrown in /usr/lib/nagios/plugins/check_mysql_connection [13:46:00] on line 66Fatal error: Uncaught mysqli_sql_exception: Connection timed out in /usr/lib/nagios/plugins/check_mysql_connections.php:66Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(66): mysqli_real_connect(Object(mysqli), 'db182.wikitide....', 'icinga', Object(SensitiveParameterValue), NULL, NULL, NULL, true)#1 {main} thrown in /usr/lib/nagios/plugins/check_mysql_connections.php on line 66 [13:46:01] PROBLEM - db181 Backups SQL on db181 is CRITICAL: connect to address 10.0.18.102 port 5666: Connection refusedconnect to host 10.0.18.102 port 5666: Connection refused [13:46:01] PROBLEM - db181 Puppet on db181 is CRITICAL: connect to address 10.0.18.102 port 5666: Connection refusedconnect to host 10.0.18.102 port 5666: Connection refused [13:46:01] RECOVERY - bast181 SSH on bast181 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u2 (protocol 2.0) [13:46:06] PROBLEM - ping6 on mw182 is UNKNOWN: check_ping: Invalid hostname/address - 10.0.18.105Usage:check_ping -H -w ,% -c ,% [-p packets] [-t timeout] [-4|-6] [13:46:08] PROBLEM - swiftobject111 Swift Object Service on swiftobject111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:46:08] PROBLEM - swiftproxy181 Puppet on swiftproxy181 is WARNING: Could not resolve hostname : Name or service not known [13:46:09] PROBLEM - ping6 on puppet181 is UNKNOWN: check_ping: Invalid hostname/address - 10.0.18.100Usage:check_ping -H -w ,% -c ,% [-p packets] [-t timeout] [-4|-6] [13:46:11] PROBLEM - bast181 ferm_active on bast181 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:46:11] PROBLEM - swiftproxy131 HTTPS on swiftproxy131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:46:12] PROBLEM - swiftproxy181 conntrack_table_size on swiftproxy181 is WARNING: Could not resolve hostname : Name or service not known [13:46:22] PROBLEM - mwtask181 PowerDNS Recursor on mwtask181 is CRITICAL: connect to address 10.0.18.106 port 5666: Connection refusedconnect to host 10.0.18.106 port 5666: Connection refused [13:46:25] PROBLEM - Host swiftproxy181 is DOWN: check_ping: Invalid hostname/address - Usage:check_ping -H -w ,% -c ,% [-p packets] [-t timeout] [-4|-6] [13:46:25] PROBLEM - mw181 SSH on mw181 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:46:26] PROBLEM - db112 MariaDB Connections on db112 is UNKNOWN: PHP Fatal error: Uncaught mysqli_sql_exception: Connection timed out in /usr/lib/nagios/plugins/check_mysql_connections.php:66Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(66): mysqli_real_connect(Object(mysqli), 'db112.miraheze....', 'icinga', Object(SensitiveParameterValue), NULL, NULL, NULL, true)#1 {main} thrown in /usr/lib/nagios/plugins/check_mysql_connection [13:46:27] on line 66Fatal error: Uncaught mysqli_sql_exception: Connection timed out in /usr/lib/nagios/plugins/check_mysql_connections.php:66Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(66): mysqli_real_connect(Object(mysqli), 'db112.miraheze....', 'icinga', Object(SensitiveParameterValue), NULL, NULL, NULL, true)#1 {main} thrown in /usr/lib/nagios/plugins/check_mysql_connections.php on line 66 [13:46:27] PROBLEM - mwtask181 NTP time on mwtask181 is CRITICAL: connect to address 10.0.18.106 port 5666: Connection refusedconnect to host 10.0.18.106 port 5666: Connection refused [13:46:30] PROBLEM - mw132 MediaWiki Rendering on mw132 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:46:31] PROBLEM - db181 APT on db181 is CRITICAL: connect to address 10.0.18.102 port 5666: Connection refusedconnect to host 10.0.18.102 port 5666: Connection refused [13:46:35] PROBLEM - mw131 MediaWiki Rendering on mw131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:46:37] PROBLEM - swiftobject101 Swift Object Service on swiftobject101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:46:44] PROBLEM - puppet181 puppetdb on puppet181 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:46:47] RECOVERY - mw141 Current Load on mw141 is OK: LOAD OK - total load average: 6.43, 8.04, 9.90 [13:46:56] PROBLEM - bast181 Puppet on bast181 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:47:01] PROBLEM - mwtask181 php-fpm on mwtask181 is CRITICAL: connect to address 10.0.18.106 port 5666: Connection refusedconnect to host 10.0.18.106 port 5666: Connection refused [13:47:24] PROBLEM - puppet181 SSH on puppet181 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:47:31] RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u2 (protocol 2.0) [13:47:31] PROBLEM - mwtask181 ferm_active on mwtask181 is CRITICAL: connect to address 10.0.18.106 port 5666: Connection refusedconnect to host 10.0.18.106 port 5666: Connection refused [13:47:35] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 1.06, 1.77, 3.06 [13:47:39] PROBLEM - swiftobject181 Swift Object Service on swiftobject181 is CRITICAL: connect to address fe80::be24:11ff:fec8:da05 and port 6000: Invalid argument [13:47:45] PROBLEM - swiftobject181 SSH on swiftobject181 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:47:46] PROBLEM - bast181 PowerDNS Recursor on bast181 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:47:46] PROBLEM - puppet181 NTP time on puppet181 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:47:48] PROBLEM - mw182 Current Load on mw182 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:47:49] PROBLEM - mw182 SSH on mw182 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:47:53] PROBLEM - mw181 ferm_active on mw181 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:47:56] PROBLEM - puppet181 APT on puppet181 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:47:59] PROBLEM - mw182 NTP time on mw182 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:47:59] PROBLEM - puppet181 conntrack_table_size on puppet181 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:48:03] PROBLEM - mw181 Puppet on mw181 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:48:05] PROBLEM - mw181 Disk Space on mw181 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:48:06] PROBLEM - db182 MariaDB on db182 is UNKNOWN: [13:48:06] PROBLEM - swiftobject181 PowerDNS Recursor on swiftobject181 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:48:14] PROBLEM - puppet181 Disk Space on puppet181 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:48:20] PROBLEM - mw182 ferm_active on mw182 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:48:22] PROBLEM - puppet181 Backups SSLKeys on puppet181 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:48:26] PROBLEM - puppet181 Backups Private on puppet181 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:48:32] PROBLEM - mw181 PowerDNS Recursor on mw181 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:48:37] PROBLEM - mw181 NTP time on mw181 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:48:58] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/a9444231c0ee...9d547a982541 [13:48:59] [02miraheze/puppet] 07paladox 039d547a9 - nrpe: allow mon181 private ip [13:51:25] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/9d547a982541...27b5ee8b85e0 [13:51:28] [02miraheze/puppet] 07paladox 0327b5ee8 - db: support private network in firewall [13:54:08] PROBLEM - puppet181 ferm_active on puppet181 is CRITICAL: connect to address 10.0.18.100 port 5666: Connection refusedconnect to host 10.0.18.100 port 5666: Connection refused [13:54:12] PROBLEM - mw181 conntrack_table_size on mw181 is CRITICAL: connect to address 10.0.18.104 port 5666: Connection refusedconnect to host 10.0.18.104 port 5666: Connection refused [13:54:15] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/27b5ee8b85e0...894487df97f0 [13:54:16] [02miraheze/puppet] 07paladox 03894487d - graylog: support private network in firewall [13:54:16] RECOVERY - Host swiftproxy181 is UP: PING OK - Packet loss = 0%, RTA = 0.79 ms [13:54:18] RECOVERY - mw152 SSH on mw152 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u2 (protocol 2.0) [13:54:22] PROBLEM - mw134 MediaWiki Rendering on mw134 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:54:23] PROBLEM - cp34 HTTPS on cp34 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [13:54:26] PROBLEM - mw181 Current Load on mw181 is CRITICAL: connect to address 10.0.18.104 port 5666: Connection refusedconnect to host 10.0.18.104 port 5666: Connection refused [13:54:27] PROBLEM - swiftproxy111 memcached on swiftproxy111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:54:30] PROBLEM - swiftproxy111 Swift Proxy on swiftproxy111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:54:36] PROBLEM - puppet181 Puppet on puppet181 is CRITICAL: connect to address 10.0.18.100 port 5666: Connection refusedconnect to host 10.0.18.100 port 5666: Connection refused [13:54:37] PROBLEM - ldap141 LDAP on ldap141 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:54:37] RECOVERY - puppet181 puppetdb on puppet181 is OK: TCP OK - 0.000 second response time on 10.0.18.100 port 8081 [13:54:42] PROBLEM - swiftproxy181 Puppet on swiftproxy181 is CRITICAL: connect to address 10.0.18.108 port 5666: Connection refusedconnect to host 10.0.18.108 port 5666: Connection refused [13:54:42] RECOVERY - mw143 Current Load on mw143 is OK: LOAD OK - total load average: 4.70, 6.43, 8.82 [13:54:47] PROBLEM - swiftproxy181 ferm_active on swiftproxy181 is CRITICAL: connect to address 10.0.18.108 port 5666: Connection refusedconnect to host 10.0.18.108 port 5666: Connection refused [13:54:50] RECOVERY - db182 MariaDB on db182 is OK: Uptime: 2265 Threads: 3 Questions: 151824 Slow queries: 0 Opens: 129 Open tables: 122 Queries per second avg: 67.030 [13:54:55] RECOVERY - puppet181 SSH on puppet181 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u2 (protocol 2.0) [13:55:01] RECOVERY - matomo151 SSH on matomo151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u2 (protocol 2.0) [13:55:05] PROBLEM - swiftproxy181 Swift Proxy on swiftproxy181 is CRITICAL: connect to address fe80::be24:11ff:fefc:dce and port 80: Invalid argument [13:55:06] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.73, 7.62, 7.44 [13:55:06] PROBLEM - mw182 Disk Space on mw182 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:55:06] PROBLEM - mw181 php-fpm on mw181 is CRITICAL: connect to address 10.0.18.104 port 5666: Connection refusedconnect to host 10.0.18.104 port 5666: Connection refused [13:55:06] PROBLEM - puppet181 Current Load on puppet181 is CRITICAL: connect to address 10.0.18.100 port 5666: Connection refusedconnect to host 10.0.18.100 port 5666: Connection refused [13:55:09] PROBLEM - db112 MariaDB on db112 is UNKNOWN: [13:55:11] PROBLEM - swiftproxy181 APT on swiftproxy181 is CRITICAL: connect to address 10.0.18.108 port 5666: Connection refusedconnect to host 10.0.18.108 port 5666: Connection refused [13:55:12] PROBLEM - puppet181 PowerDNS Recursor on puppet181 is CRITICAL: connect to address 10.0.18.100 port 5666: Connection refusedconnect to host 10.0.18.100 port 5666: Connection refused [13:55:14] RECOVERY - test151 SSH on test151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u2 (protocol 2.0) [13:55:22] PROBLEM - swiftproxy111 HTTPS on swiftproxy111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:55:24] PROBLEM - swiftproxy131 HTTP on swiftproxy131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:55:25] RECOVERY - db182 MariaDB Connections on db182 is OK: OK connection usage: 0.4%Current connections: 2 [13:55:33] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/894487df97f0...dfcbd2c13e77 [13:55:33] PROBLEM - swiftobject181 ferm_active on swiftobject181 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:55:34] PROBLEM - swiftproxy181 HTTPS on swiftproxy181 is CRITICAL: connect to address fe80::be24:11ff:fefc:dce and port 443: Invalid argumentHTTP CRITICAL - Unable to open TCP socket [13:55:35] [02miraheze/puppet] 07paladox 03dfcbd2c - opensearch: support private network in firewall [13:55:37] PROBLEM - swiftproxy181 PowerDNS Recursor on swiftproxy181 is CRITICAL: connect to address 10.0.18.108 port 5666: Connection refusedconnect to host 10.0.18.108 port 5666: Connection refused [13:55:38] PROBLEM - mw182 PowerDNS Recursor on mw182 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:55:38] PROBLEM - swiftproxy181 Disk Space on swiftproxy181 is CRITICAL: connect to address 10.0.18.108 port 5666: Connection refusedconnect to host 10.0.18.108 port 5666: Connection refused [13:55:39] PROBLEM - mw182 Puppet on mw182 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:55:39] PROBLEM - ping6 on mw181 is UNKNOWN: check_ping: Invalid hostname/address - 10.0.18.104Usage:check_ping -H -w ,% -c ,% [-p packets] [-t timeout] [-4|-6] [13:55:42] PROBLEM - swiftproxy181 NTP time on swiftproxy181 is CRITICAL: connect to address 10.0.18.108 port 5666: Connection refusedconnect to host 10.0.18.108 port 5666: Connection refused [13:55:46] RECOVERY - db181 MariaDB Connections on db181 is OK: OK connection usage: 0.2%Current connections: 1 [13:55:48] PROBLEM - swiftproxy181 conntrack_table_size on swiftproxy181 is CRITICAL: connect to address 10.0.18.108 port 5666: Connection refusedconnect to host 10.0.18.108 port 5666: Connection refused [13:55:51] PROBLEM - swiftproxy181 Current Load on swiftproxy181 is CRITICAL: connect to address 10.0.18.108 port 5666: Connection refusedconnect to host 10.0.18.108 port 5666: Connection refused [13:55:52] RECOVERY - mw181 SSH on mw181 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u2 (protocol 2.0) [13:55:53] PROBLEM - mail121 webmail.miraheze.org HTTPS on mail121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:55:56] PROBLEM - swiftproxy181 HTTP on swiftproxy181 is CRITICAL: connect to address fe80::be24:11ff:fefc:dce and port 80: Invalid argumentHTTP CRITICAL - Unable to open TCP socket [13:55:57] PROBLEM - db151 MariaDB Connections on db151 is UNKNOWN: PHP Fatal error: Uncaught mysqli_sql_exception: Connection timed out in /usr/lib/nagios/plugins/check_mysql_connections.php:66Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(66): mysqli_real_connect(Object(mysqli), 'db151.wikitide....', 'icinga', Object(SensitiveParameterValue), NULL, NULL, NULL, true)#1 {main} thrown in /usr/lib/nagios/plugins/check_mysql_connection [13:55:57] on line 66Fatal error: Uncaught mysqli_sql_exception: Connection timed out in /usr/lib/nagios/plugins/check_mysql_connections.php:66Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(66): mysqli_real_connect(Object(mysqli), 'db151.wikitide....', 'icinga', Object(SensitiveParameterValue), NULL, NULL, NULL, true)#1 {main} thrown in /usr/lib/nagios/plugins/check_mysql_connections.php on line 66 [13:55:57] PROBLEM - mw182 conntrack_table_size on mw182 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:56:01] PROBLEM - mw182 php-fpm on mw182 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:56:03] PROBLEM - swiftproxy111 HTTP on swiftproxy111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:56:03] PROBLEM - db142 MariaDB on db142 is UNKNOWN: [13:56:06] PROBLEM - swiftobject113 Swift Object Service on swiftobject113 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:56:06] PROBLEM - swiftobject112 Swift Object Service on swiftobject112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:56:09] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/dfcbd2c13e77...72d1be6b776f [13:56:11] [02miraheze/puppet] 07paladox 0372d1be6 - puppetdb: support private network in firewall [13:56:18] RECOVERY - test151 memcached on test151 is OK: TCP OK - 0.000 second response time on 10.0.15.118 port 11211 [13:56:18] PROBLEM - cp34 HTTPS on cp34 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.642 second response time [13:56:24] PROBLEM - swiftobject181 NTP time on swiftobject181 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:56:29] PROBLEM - mem141 memcached on mem141 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:56:32] PROBLEM - swiftobject181 Puppet on swiftobject181 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:56:35] PROBLEM - swiftobject181 Disk Space on swiftobject181 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:56:47] PROBLEM - swiftobject181 Current Load on swiftobject181 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:56:49] PROBLEM - swiftobject181 conntrack_table_size on swiftobject181 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:56:52] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/72d1be6b776f...86672b35bc7d [13:56:53] PROBLEM - swiftobject181 APT on swiftobject181 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:56:55] PROBLEM - db101 MariaDB Connections on db101 is UNKNOWN: PHP Fatal error: Uncaught mysqli_sql_exception: Connection timed out in /usr/lib/nagios/plugins/check_mysql_connections.php:66Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(66): mysqli_real_connect(Object(mysqli), 'db101.miraheze....', 'icinga', Object(SensitiveParameterValue), NULL, NULL, NULL, true)#1 {main} thrown in /usr/lib/nagios/plugins/check_mysql_connection [13:56:55] [02miraheze/puppet] 07paladox 0386672b3 - openldap: support private network in firewall [13:56:55] on line 66Fatal error: Uncaught mysqli_sql_exception: Connection timed out in /usr/lib/nagios/plugins/check_mysql_connections.php:66Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(66): mysqli_real_connect(Object(mysqli), 'db101.miraheze....', 'icinga', Object(SensitiveParameterValue), NULL, NULL, NULL, true)#1 {main} thrown in /usr/lib/nagios/plugins/check_mysql_connections.php on line 66 [13:57:01] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 9.22, 8.15, 7.65 [13:57:05] PROBLEM - mail121 HTTPS on mail121 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Connection timed out after 10003 milliseconds [13:57:12] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/86672b35bc7d...c3333f127648 [13:57:15] [02miraheze/puppet] 07paladox 03c3333f1 - redis: support private network in firewall [13:57:15] PROBLEM - swiftobject121 Swift Object Service on swiftobject121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:57:22] RECOVERY - graylog131 HTTPS on graylog131 is OK: HTTP OK: HTTP/2 200 - 646 bytes in 0.624 second response time [13:57:23] RECOVERY - swiftobject181 SSH on swiftobject181 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u2 (protocol 2.0) [13:57:39] RECOVERY - mw182 SSH on mw182 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u2 (protocol 2.0) [13:57:44] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/c3333f127648...6833e5ce040a [13:57:45] [02miraheze/puppet] 07paladox 036833e5c - swift: support private network in firewall [13:57:46] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.69, 7.39, 7.77 [13:57:48] PROBLEM - mem131 memcached on mem131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:57:52] PROBLEM - db121 MariaDB Connections on db121 is UNKNOWN: PHP Fatal error: Uncaught mysqli_sql_exception: Connection timed out in /usr/lib/nagios/plugins/check_mysql_connections.php:66Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(66): mysqli_real_connect(Object(mysqli), 'db121.miraheze....', 'icinga', Object(SensitiveParameterValue), NULL, NULL, NULL, true)#1 {main} thrown in /usr/lib/nagios/plugins/check_mysql_connection [13:57:52] on line 66Fatal error: Uncaught mysqli_sql_exception: Connection timed out in /usr/lib/nagios/plugins/check_mysql_connections.php:66Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(66): mysqli_real_connect(Object(mysqli), 'db121.miraheze....', 'icinga', Object(SensitiveParameterValue), NULL, NULL, NULL, true)#1 {main} thrown in /usr/lib/nagios/plugins/check_mysql_connections.php on line 66 [13:57:57] PROBLEM - swiftac111 Swift Container Service on swiftac111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:57:59] PROBLEM - swiftobject122 Swift Object Service on swiftobject122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:57:59] PROBLEM - swiftac111 Swift Account Service on swiftac111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:58:07] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/6833e5ce040a...fe383adff0a3 [13:58:10] [02miraheze/puppet] 07paladox 03fe383ad - ssl: support private network in firewall [13:58:16] RECOVERY - with.cpt-ra.bid - reverse DNS on sslhost is OK: SSL OK - with.cpt-ra.bid reverse DNS resolves to cp35.miraheze.org - CNAME OK [13:58:29] PROBLEM - db151 MariaDB on db151 is UNKNOWN: [13:58:32] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/fe383adff0a3...a0ab2038ab70 [13:58:33] [02miraheze/puppet] 07paladox 03a0ab203 - cloud: support private network in firewall [13:58:57] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 5.80, 7.17, 7.34 [13:59:09] PROBLEM - db121 MariaDB on db121 is UNKNOWN: [13:59:13] PROBLEM - db101 MariaDB on db101 is UNKNOWN: [13:59:16] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/a0ab2038ab70...8e75e9d589ca [13:59:18] [02miraheze/puppet] 07paladox 038e75e9d - poolcounter: support private network in firewall [13:59:35] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/8e75e9d589ca...59d7be3c1104 [13:59:38] [02miraheze/puppet] 07paladox 0359d7be3 - Phabricator: support private network in firewall [13:59:40] PROBLEM - reports121 HTTPS on reports121 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Connection timed out after 10002 milliseconds [13:59:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.90, 7.98, 7.93 [13:59:56] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/59d7be3c1104...bbbf5ea12f2c [13:59:57] [02miraheze/puppet] 07paladox 03bbbf5ea - mathoid: support private network in firewall [14:00:14] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/bbbf5ea12f2c...76e39bd0acb8 [14:00:16] [02miraheze/puppet] 07paladox 0376e39bd - memcached: support private network in firewall [14:00:37] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/76e39bd0acb8...4aefae33e5aa [14:00:39] [02miraheze/puppet] 07paladox 034aefae3 - mediawiki: support private network in firewall [14:00:59] PROBLEM - mw133 HTTPS on mw133 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Connection timed out after 10002 milliseconds [14:01:05] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/4aefae33e5aa...707661872233 [14:01:08] [02miraheze/puppet] 07paladox 037076618 - irc: support private network in firewall [14:01:24] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/707661872233...e74213a88602 [14:01:26] [02miraheze/puppet] 07paladox 03e74213a - matomo: support private network in firewall [14:01:42] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/e74213a88602...fa9ada19a6b0 [14:01:43] [02miraheze/puppet] 07paladox 03fa9ada1 - reports: support private network in firewall [14:02:07] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/fa9ada19a6b0...91d93b05dafd [14:02:09] [02miraheze/puppet] 07paladox 0391d93b0 - PostgreSQL: support private network in firewall [14:02:32] RECOVERY - swiftobject101 Swift Object Service on swiftobject101 is OK: TCP OK - 0.122 second response time on 2a10:6740::6:103 port 6000 [14:02:35] PROBLEM - mw142 HTTPS on mw142 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Connection timed out after 10004 milliseconds [14:02:38] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/91d93b05dafd...7353b919c152 [14:02:39] [02miraheze/puppet] 07paladox 037353b91 - varnish: support private network in firewall [14:02:48] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.14, 7.57, 7.45 [14:02:50] RECOVERY - db112 MariaDB Connections on db112 is OK: OK connection usage: 5.4%Current connections: 27 [14:02:54] PROBLEM - mw133 MediaWiki Rendering on mw133 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:02:58] RECOVERY - db112 MariaDB on db112 is OK: Uptime: 4348238 Threads: 27 Questions: 2775155076 Slow queries: 251701 Opens: 1091382 Open tables: 2246 Queries per second avg: 638.225 [14:03:22] PROBLEM - cp41 HTTPS on cp41 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [14:03:39] PROBLEM - cp51 HTTPS on cp51 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [14:03:52] RECOVERY - db142 MariaDB Connections on db142 is OK: OK connection usage: 7.2%Current connections: 36 [14:03:54] RECOVERY - db142 MariaDB on db142 is OK: Uptime: 6876 Threads: 26 Questions: 2311535 Slow queries: 367 Opens: 37166 Open tables: 37109 Queries per second avg: 336.174 [14:03:54] RECOVERY - db151 MariaDB on db151 is OK: Uptime: 3205 Threads: 1 Questions: 1256 Slow queries: 0 Opens: 19 Open tables: 13 Queries per second avg: 0.391 [14:04:15] RECOVERY - db101 MariaDB Connections on db101 is OK: OK connection usage: 3.4%Current connections: 17 [14:04:16] PROBLEM - mw142 MediaWiki Rendering on mw142 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:04:27] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.59, 7.48, 7.45 [14:04:37] RECOVERY - ldap141 LDAP on ldap141 is OK: LDAP OK - 0.513 seconds response time [14:04:38] RECOVERY - db121 MariaDB on db121 is OK: Uptime: 1591983 Threads: 72 Questions: 1071792727 Slow queries: 22317 Opens: 688340 Open tables: 385029 Queries per second avg: 673.243 [14:04:45] RECOVERY - mem141 memcached on mem141 is OK: TCP OK - 0.122 second response time on 2a10:6740::6:510 port 11211 [14:04:51] RECOVERY - db101 MariaDB on db101 is OK: Uptime: 2524705 Threads: 25 Questions: 1506563751 Slow queries: 2159 Opens: 1106620 Open tables: 200000 Queries per second avg: 596.728 [14:04:57] RECOVERY - mw151 SSH on mw151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u2 (protocol 2.0) [14:05:16] PROBLEM - cp41 Varnish Backends on cp41 is CRITICAL: 1 backends are down. mw143 [14:05:20] PROBLEM - cp41 HTTPS on cp41 is WARNING: HTTP WARNING: HTTP/2 404 - 3480 bytes in 1.168 second response time [14:05:30] RECOVERY - swiftobject111 Swift Object Service on swiftobject111 is OK: TCP OK - 0.122 second response time on 2a10:6740::6:203 port 6000 [14:05:36] RECOVERY - db151 MariaDB Connections on db151 is OK: OK connection usage: 0.2%Current connections: 1 [14:05:40] RECOVERY - db121 MariaDB Connections on db121 is OK: OK connection usage: 9%Current connections: 45 [14:05:44] PROBLEM - cp51 HTTPS on cp51 is WARNING: HTTP WARNING: HTTP/2 404 - 3478 bytes in 6.268 second response time [14:05:48] RECOVERY - reports121 HTTPS on reports121 is OK: HTTP OK: HTTP/2 200 - 1139 bytes in 0.496 second response time [14:05:52] RECOVERY - swiftproxy111 HTTPS on swiftproxy111 is OK: HTTP OK: Status line output matched "HTTP/1.1 404" - 352 bytes in 0.614 second response time [14:05:55] PROBLEM - mw131 HTTPS on mw131 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 7.000 second response time [14:05:57] PROBLEM - cp26 Varnish Backends on cp26 is CRITICAL: 1 backends are down. mw134 [14:06:22] RECOVERY - mw134 MediaWiki Rendering on mw134 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 1.583 second response time [14:06:22] RECOVERY - phab121 phabricator.miraheze.org HTTPS on phab121 is OK: HTTP OK: HTTP/1.1 200 OK - 20294 bytes in 0.785 second response time [14:06:27] PROBLEM - mw143 Current Load on mw143 is WARNING: LOAD WARNING - total load average: 10.14, 10.67, 9.77 [14:06:28] RECOVERY - phab121 phab.miraheze.wiki HTTPS on phab121 is OK: HTTP OK: Status line output matched "HTTP/1.1 200" - 17683 bytes in 0.667 second response time [14:06:30] PROBLEM - mw134 HTTPS on mw134 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 1.941 second response time [14:06:32] RECOVERY - swiftac111 Swift Container Service on swiftac111 is OK: TCP OK - 0.122 second response time on 2a10:6740::6:202 port 6001 [14:06:44] RECOVERY - swiftobject112 Swift Object Service on swiftobject112 is OK: TCP OK - 0.122 second response time on 2a10:6740::6:204 port 6000 [14:06:46] RECOVERY - swiftac111 Swift Account Service on swiftac111 is OK: TCP OK - 0.123 second response time on 2a10:6740::6:202 port 6002 [14:06:47] RECOVERY - mw133 MediaWiki Rendering on mw133 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 2.814 second response time [14:06:49] RECOVERY - swiftproxy131 HTTPS on swiftproxy131 is OK: HTTP OK: Status line output matched "HTTP/1.1 404" - 352 bytes in 0.518 second response time [14:06:50] RECOVERY - swiftobject113 Swift Object Service on swiftobject113 is OK: TCP OK - 0.122 second response time on 2a10:6740::6:205 port 6000 [14:06:52] RECOVERY - swiftproxy111 HTTP on swiftproxy111 is OK: HTTP OK: Status line output matched "HTTP/1.1 404" - 352 bytes in 0.263 second response time [14:06:56] PROBLEM - mw143 HTTPS on mw143 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.909 second response time [14:06:57] RECOVERY - swiftproxy131 Swift Proxy on swiftproxy131 is OK: TCP OK - 0.122 second response time on 2a10:6740::6:315 port 80 [14:06:57] RECOVERY - swiftproxy111 memcached on swiftproxy111 is OK: TCP OK - 0.122 second response time on 2a10:6740::6:201 port 11211 [14:06:58] RECOVERY - mw151 MediaWiki Rendering on mw151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 8.157 second response time [14:06:59] PROBLEM - mw133 HTTPS on mw133 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.566 second response time [14:06:59] PROBLEM - mw142 Current Load on mw142 is CRITICAL: LOAD CRITICAL - total load average: 13.99, 13.54, 10.56 [14:07:03] RECOVERY - swiftproxy111 Swift Proxy on swiftproxy111 is OK: TCP OK - 0.122 second response time on 2a10:6740::6:201 port 80 [14:07:03] PROBLEM - mw141 HTTPS on mw141 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 8.132 second response time [14:07:03] RECOVERY - mem131 memcached on mem131 is OK: TCP OK - 0.122 second response time on 2a10:6740::6:405 port 11211 [14:07:12] RECOVERY - mw131 MediaWiki Rendering on mw131 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 6.207 second response time [14:07:13] RECOVERY - cp41 Varnish Backends on cp41 is OK: All 19 backends are healthy [14:07:13] PROBLEM - mw132 HTTPS on mw132 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 1.244 second response time [14:07:13] PROBLEM - mw142 HTTPS on mw142 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.670 second response time [14:07:15] RECOVERY - mw141 MediaWiki Rendering on mw141 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 1.137 second response time [14:07:29] RECOVERY - mw132 MediaWiki Rendering on mw132 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 3.719 second response time [14:07:35] RECOVERY - mw143 MediaWiki Rendering on mw143 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 4.093 second response time [14:07:39] PROBLEM - mw141 Current Load on mw141 is WARNING: LOAD WARNING - total load average: 11.09, 10.46, 9.34 [14:07:40] RECOVERY - swiftproxy131 HTTP on swiftproxy131 is OK: HTTP OK: Status line output matched "HTTP/1.1 404" - 352 bytes in 0.267 second response time [14:07:46] RECOVERY - swiftobject121 Swift Object Service on swiftobject121 is OK: TCP OK - 0.122 second response time on 2a10:6740::6:316 port 6000 [14:07:52] RECOVERY - cp26 Varnish Backends on cp26 is OK: All 19 backends are healthy [14:08:25] RECOVERY - swiftobject122 Swift Object Service on swiftobject122 is OK: TCP OK - 0.122 second response time on 2a10:6740::6:317 port 6000 [14:08:25] RECOVERY - mw142 MediaWiki Rendering on mw142 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 1.032 second response time [14:08:25] PROBLEM - mw143 Current Load on mw143 is CRITICAL: LOAD CRITICAL - total load average: 13.50, 11.02, 9.95 [14:10:22] RECOVERY - mw143 Current Load on mw143 is OK: LOAD OK - total load average: 7.48, 9.66, 9.59 [14:10:49] RECOVERY - mw141 Current Load on mw141 is OK: LOAD OK - total load average: 7.86, 10.03, 9.45 [14:12:55] PROBLEM - mw142 Current Load on mw142 is WARNING: LOAD WARNING - total load average: 8.48, 10.77, 10.37 [14:12:58] PROBLEM - cp26 Disk Space on cp26 is CRITICAL: DISK CRITICAL - free space: / 5588MiB (5% inode=98%); [14:13:07] RECOVERY - swiftproxy181 PowerDNS Recursor on swiftproxy181 is OK: DNS OK: 0.204 seconds response time. miraheze.org returns 2607:5300:205:200::1c93,2607:5300:205:200::3121,51.222.12.133,51.222.14.30 [14:13:31] RECOVERY - swiftproxy181 Current Load on swiftproxy181 is OK: LOAD OK - total load average: 0.12, 0.07, 0.02 [14:13:33] PROBLEM - mw151 MediaWiki Rendering on mw151 is CRITICAL: Name or service not knownHTTP CRITICAL - Unable to open TCP socket [14:13:50] RECOVERY - swiftproxy181 NTP time on swiftproxy181 is OK: NTP OK: Offset 0.001493364573 secs [14:13:54] RECOVERY - swiftproxy181 conntrack_table_size on swiftproxy181 is OK: OK: nf_conntrack is 0 % full [14:14:06] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/7353b919c152...eb87e758532a [14:14:07] [02miraheze/puppet] 07paladox 03eb87e75 - base::icinga: listen on all interfaces for nrpe [14:14:14] RECOVERY - swiftproxy181 APT on swiftproxy181 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [14:14:23] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 10.14, 8.38, 7.70 [14:14:37] RECOVERY - bast181 PowerDNS Recursor on bast181 is OK: DNS OK: 0.101 seconds response time. miraheze.org returns 2607:5300:205:200::1c93,2607:5300:205:200::3121,51.222.12.133,51.222.14.30 [14:14:38] RECOVERY - swiftproxy181 ferm_active on swiftproxy181 is OK: OK ferm input default policy is set [14:14:39] RECOVERY - swiftproxy181 Disk Space on swiftproxy181 is OK: DISK OK - free space: / 23356MiB (87% inode=94%); [14:14:40] RECOVERY - swiftproxy181 Puppet on swiftproxy181 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [14:14:47] RECOVERY - db151 conntrack_table_size on db151 is OK: OK: nf_conntrack is 0 % full [14:14:49] RECOVERY - db151 NTP time on db151 is OK: NTP OK: Offset 0.03348687291 secs [14:14:51] RECOVERY - mw142 Current Load on mw142 is OK: LOAD OK - total load average: 8.64, 9.80, 10.04 [14:14:55] RECOVERY - db151 Disk Space on db151 is OK: DISK OK - free space: / 884275MiB (99% inode=99%); [14:14:58] RECOVERY - db151 ferm_active on db151 is OK: OK ferm input default policy is set [14:15:03] RECOVERY - bast181 Current Load on bast181 is OK: LOAD OK - total load average: 0.08, 0.03, 0.01 [14:15:05] RECOVERY - db182 ferm_active on db182 is OK: OK ferm input default policy is set [14:15:13] RECOVERY - db182 NTP time on db182 is OK: NTP OK: Offset 0.03072792292 secs [14:15:14] RECOVERY - prometheus151 Puppet on prometheus151 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [14:15:17] RECOVERY - db181 Current Load on db181 is OK: LOAD OK - total load average: 0.04, 0.02, 0.00 [14:15:17] RECOVERY - db181 PowerDNS Recursor on db181 is OK: DNS OK: 0.104 seconds response time. miraheze.org returns 2607:5300:205:200::1c93,2607:5300:205:200::3121,51.222.12.133,51.222.14.30 [14:15:19] RECOVERY - bast181 ferm_active on bast181 is OK: OK ferm input default policy is set [14:15:22] RECOVERY - matomo151 PowerDNS Recursor on matomo151 is OK: DNS OK: 0.115 seconds response time. miraheze.org returns 2607:5300:205:200::1c93,2607:5300:205:200::3121,51.222.12.133,51.222.14.30 [14:15:25] RECOVERY - mem151 ferm_active on mem151 is OK: OK ferm input default policy is set [14:15:25] RECOVERY - matomo151 Disk Space on matomo151 is OK: DISK OK - free space: / 11626MiB (64% inode=90%); [14:15:27] RECOVERY - matomo151 php-fpm on matomo151 is OK: PROCS OK: 37 processes with command name 'php-fpm8.2' [14:15:29] RECOVERY - db151 PowerDNS Recursor on db151 is OK: DNS OK: 0.128 seconds response time. miraheze.org returns 2607:5300:205:200::1c93,2607:5300:205:200::3121,51.222.12.133,51.222.14.30 [14:15:29] RECOVERY - db182 conntrack_table_size on db182 is OK: OK: nf_conntrack is 0 % full [14:15:30] RECOVERY - mem151 APT on mem151 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [14:15:33] RECOVERY - db181 Puppet on db181 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [14:15:35] RECOVERY - mw181 Disk Space on mw181 is OK: DISK OK - free space: / 39351MiB (73% inode=89%); [14:15:35] RECOVERY - db151 Puppet on db151 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [14:15:36] RECOVERY - db181 Disk Space on db181 is OK: DISK OK - free space: / 884219MiB (99% inode=99%); [14:15:36] RECOVERY - db181 Backups SQL mhglobal on db181 is OK: FILE_AGE OK: /var/log/sql-mhglobal-backup-weekly.log is 292502 seconds old and 209 bytes [14:15:41] RECOVERY - matomo151 NTP time on matomo151 is OK: NTP OK: Offset 0.03447270393 secs [14:15:42] RECOVERY - mon181 conntrack_table_size on mon181 is OK: OK: nf_conntrack is 0 % full [14:15:44] RECOVERY - mem151 PowerDNS Recursor on mem151 is OK: DNS OK: 0.121 seconds response time. miraheze.org returns 2607:5300:205:200::1c93,2607:5300:205:200::3121,51.222.12.133,51.222.14.30 [14:15:45] RECOVERY - db182 APT on db182 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [14:15:45] RECOVERY - mw182 NTP time on mw182 is OK: NTP OK: Offset 0.03159958124 secs [14:15:46] RECOVERY - mon181 Current Load on mon181 is OK: LOAD OK - total load average: 0.73, 0.51, 0.54 [14:15:49] RECOVERY - mw152 conntrack_table_size on mw152 is OK: OK: nf_conntrack is 0 % full [14:15:49] RECOVERY - mon181 PowerDNS Recursor on mon181 is OK: DNS OK: 0.152 seconds response time. miraheze.org returns 2607:5300:205:200::1c93,2607:5300:205:200::3121,51.222.12.133,51.222.14.30 [14:15:50] RECOVERY - db181 Backups SQL on db181 is OK: FILE_AGE OK: /var/log/sql-backup.log is 472548 seconds old and 93 bytes [14:15:51] RECOVERY - mon181 IRC RC Bot on mon181 is OK: PROCS OK: 1 process with args 'ircrcbot.py' [14:15:52] RECOVERY - os151 ferm_active on os151 is OK: OK ferm input default policy is set [14:15:52] RECOVERY - db181 conntrack_table_size on db181 is OK: OK: nf_conntrack is 0 % full [14:15:54] RECOVERY - bast181 NTP time on bast181 is OK: NTP OK: Offset 0.0319827795 secs [14:15:55] RECOVERY - bast181 conntrack_table_size on bast181 is OK: OK: nf_conntrack is 0 % full [14:15:57] RECOVERY - db181 NTP time on db181 is OK: NTP OK: Offset 0.0315926075 secs [14:16:00] RECOVERY - os151 Current Load on os151 is OK: LOAD OK - total load average: 0.17, 0.07, 0.01 [14:16:00] RECOVERY - bast181 Disk Space on bast181 is OK: DISK OK - free space: / 5843MiB (66% inode=86%); [14:16:00] RECOVERY - mwtask181 ferm_active on mwtask181 is OK: OK ferm input default policy is set [14:16:00] RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 0.49, 0.17, 0.05 [14:16:01] RECOVERY - matomo151 Redis Process on matomo151 is OK: PROCS OK: 1 process with args 'redis-server' [14:16:01] RECOVERY - mon181 Check correctness of the icinga configuration on mon181 is OK: Icinga configuration is correct [14:16:02] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 0.96, 0.71, 0.47 [14:16:03] RECOVERY - mwtask181 NTP time on mwtask181 is OK: NTP OK: Offset 0.03175449371 secs [14:16:04] RECOVERY - db151 Backups SQL mhglobal on db151 is OK: FILE_AGE OK: /var/log/sql-mhglobal-backup-weekly.log is 292529 seconds old and 208 bytes [14:16:05] RECOVERY - mem151 Puppet on mem151 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [14:16:06] RECOVERY - mw151 Puppet on mw151 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [14:16:06] RECOVERY - db151 Current Load on db151 is OK: LOAD OK - total load average: 0.20, 0.10, 0.03 [14:16:08] RECOVERY - puppet181 Disk Space on puppet181 is OK: DISK OK - free space: / 22320MiB (73% inode=92%); [14:16:09] RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.168 seconds response time. miraheze.org returns 2607:5300:205:200::1c93,2607:5300:205:200::3121,51.222.12.133,51.222.14.30 [14:16:09] RECOVERY - puppet181 PowerDNS Recursor on puppet181 is OK: DNS OK: 0.120 seconds response time. miraheze.org returns 2607:5300:205:200::1c93,2607:5300:205:200::3121,51.222.12.133,51.222.14.30 [14:16:09] RECOVERY - db181 ferm_active on db181 is OK: OK ferm input default policy is set [14:16:10] RECOVERY - mon181 IRC Log Bot on mon181 is OK: PROCS OK: 1 process with args 'adminlogbot.py' [14:16:11] RECOVERY - db151 APT on db151 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [14:16:11] RECOVERY - db182 Backups SQL roundcubemail on db182 is OK: FILE_AGE OK: /var/log/sql-roundcubemail-backup-fortnightly.log is 810959 seconds old and 219 bytes [14:16:12] RECOVERY - prometheus151 APT on prometheus151 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [14:16:13] RECOVERY - mw181 NTP time on mw181 is OK: NTP OK: Offset 0.03247320652 secs [14:16:14] RECOVERY - matomo151 conntrack_table_size on matomo151 is OK: OK: nf_conntrack is 0 % full [14:16:15] RECOVERY - db182 Disk Space on db182 is OK: DISK OK - free space: / 441094MiB (96% inode=99%); [14:16:16] RECOVERY - swiftobject181 Disk Space on swiftobject181 is OK: DISK OK - free space: / 1332285MiB (99% inode=99%); [14:16:16] RECOVERY - puppet181 APT on puppet181 is OK: APT OK: 4 packages available for upgrade (0 critical updates). [14:16:17] RECOVERY - puppet181 Current Load on puppet181 is OK: LOAD OK - total load average: 5.14, 2.64, 1.73 [14:16:18] RECOVERY - db182 PowerDNS Recursor on db182 is OK: DNS OK: 0.102 seconds response time. miraheze.org returns 2607:5300:205:200::1c93,2607:5300:205:200::3121,51.222.12.133,51.222.14.30 [14:16:19] RECOVERY - mw151 Disk Space on mw151 is OK: DISK OK - free space: / 39393MiB (73% inode=89%); [14:16:19] RECOVERY - db182 Backups SQL reports on db182 is OK: FILE_AGE OK: /var/log/sql-reports-backup-weekly.log is 292545 seconds old and 206 bytes [14:16:20] RECOVERY - db181 APT on db181 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [14:16:21] RECOVERY - os151 NTP time on os151 is OK: NTP OK: Offset 0.0349932611 secs [14:16:32] RECOVERY - mw182 Disk Space on mw182 is OK: DISK OK - free space: / 39350MiB (73% inode=89%); [14:16:32] RECOVERY - mw181 ferm_active on mw181 is OK: OK ferm input default policy is set [14:16:33] RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 0.10, 0.05, 0.01 [14:16:33] RECOVERY - mw151 ferm_active on mw151 is OK: OK ferm input default policy is set [14:16:34] RECOVERY - matomo151 Puppet on matomo151 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [14:16:34] RECOVERY - mw152 php-fpm on mw152 is OK: PROCS OK: 31 processes with command name 'php-fpm8.2' [14:16:35] RECOVERY - swiftobject181 Puppet on swiftobject181 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [14:16:35] RECOVERY - mem151 Disk Space on mem151 is OK: DISK OK - free space: / 5887MiB (66% inode=86%); [14:16:36] RECOVERY - swiftobject181 Current Load on swiftobject181 is OK: LOAD OK - total load average: 0.10, 0.04, 0.01 [14:16:36] RECOVERY - swiftobject181 conntrack_table_size on swiftobject181 is OK: OK: nf_conntrack is 0 % full [14:16:37] RECOVERY - puppet181 conntrack_table_size on puppet181 is OK: OK: nf_conntrack is 0 % full [14:16:37] RECOVERY - db182 Backups SQL icingaweb2 on db182 is OK: FILE_AGE OK: /var/log/sql-icingaweb2-backup-fortnightly.log is 810983 seconds old and 125 bytes [14:16:38] RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 0.13, 0.08, 0.03 [14:16:38] RECOVERY - mwtask181 PowerDNS Recursor on mwtask181 is OK: DNS OK: 0.101 seconds response time. miraheze.org returns 2607:5300:205:200::1c93,2607:5300:205:200::3121,51.222.12.133,51.222.14.30 [14:16:39] RECOVERY - mw181 php-fpm on mw181 is OK: PROCS OK: 31 processes with command name 'php-fpm8.2' [14:16:40] RECOVERY - db182 Puppet on db182 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [14:16:43] RECOVERY - mem151 conntrack_table_size on mem151 is OK: OK: nf_conntrack is 0 % full [14:16:44] RECOVERY - prometheus151 ferm_active on prometheus151 is OK: OK ferm input default policy is set [14:16:44] RECOVERY - mw151 NTP time on mw151 is OK: NTP OK: Offset 0.03544119 secs [14:16:45] RECOVERY - mw151 conntrack_table_size on mw151 is OK: OK: nf_conntrack is 0 % full [14:16:48] RECOVERY - db182 Backups SQL icinga on db182 is OK: FILE_AGE OK: /var/log/sql-icinga-backup-fortnightly.log is 810931 seconds old and 120 bytes [14:16:50] RECOVERY - mon181 IRC-Discord Relay Bot on mon181 is OK: PROCS OK: 2 processes with args 'relaybot' [14:16:52] RECOVERY - swiftobject181 PowerDNS Recursor on swiftobject181 is OK: DNS OK: 0.107 seconds response time. miraheze.org returns 2607:5300:205:200::1c93,2607:5300:205:200::3121,51.222.12.133,51.222.14.30 [14:16:53] RECOVERY - prometheus151 conntrack_table_size on prometheus151 is OK: OK: nf_conntrack is 0 % full [14:16:54] RECOVERY - os151 PowerDNS Recursor on os151 is OK: DNS OK: 0.122 seconds response time. miraheze.org returns 2607:5300:205:200::1c93,2607:5300:205:200::3121,51.222.12.133,51.222.14.30 [14:16:55] RECOVERY - mw181 PowerDNS Recursor on mw181 is OK: DNS OK: 0.193 seconds response time. miraheze.org returns 2607:5300:205:200::1c93,2607:5300:205:200::3121,51.222.12.133,51.222.14.30 [14:16:55] RECOVERY - prometheus151 Disk Space on prometheus151 is OK: DISK OK - free space: / 127130MiB (95% inode=99%); [14:16:56] PROBLEM - cloud11 Puppet on cloud11 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [14:16:56] RECOVERY - puppet181 ferm_active on puppet181 is OK: OK ferm input default policy is set [14:16:59] RECOVERY - mw182 Puppet on mw182 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [14:16:59] RECOVERY - os151 conntrack_table_size on os151 is OK: OK: nf_conntrack is 0 % full [14:17:01] RECOVERY - mw182 PowerDNS Recursor on mw182 is OK: DNS OK: 0.186 seconds response time. miraheze.org returns 2607:5300:205:200::1c93,2607:5300:205:200::3121,51.222.12.133,51.222.14.30 [14:17:03] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 0.04, 0.03, 0.01 [14:17:03] PROBLEM - mon181 Backups Grafana on mon181 is WARNING: FILE_AGE WARNING: /var/log/grafana-backup.log is 1163800 seconds old and 93 bytes [14:17:03] RECOVERY - mon181 IRCEcho on mon181 is OK: PROCS OK: 1 process with args '/usr/local/bin/ircecho' [14:17:04] RECOVERY - mw182 ferm_active on mw182 is OK: OK ferm input default policy is set [14:17:06] RECOVERY - mw152 PowerDNS Recursor on mw152 is OK: DNS OK: 0.199 seconds response time. miraheze.org returns 2607:5300:205:200::1c93,2607:5300:205:200::3121,51.222.12.133,51.222.14.30 [14:17:07] RECOVERY - mw182 conntrack_table_size on mw182 is OK: OK: nf_conntrack is 0 % full [14:17:13] RECOVERY - puppet181 Backups Private on puppet181 is OK: FILE_AGE OK: /var/log/private-backup.log is 298924 seconds old and 92 bytes [14:17:14] RECOVERY - mon181 IRC Log Server Bot on mon181 is OK: PROCS OK: 1 process with args 'irclogserverbot.py' [14:17:16] RECOVERY - mw181 conntrack_table_size on mw181 is OK: OK: nf_conntrack is 0 % full [14:17:16] RECOVERY - mon181 php-fpm on mon181 is OK: PROCS OK: 21 processes with command name 'php-fpm8.2' [14:17:16] RECOVERY - mw181 Puppet on mw181 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [14:17:20] RECOVERY - mon181 Puppet on mon181 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [14:17:20] RECOVERY - mwtask181 JobRunner Service on mwtask181 is OK: PROCS OK: 1 process with args 'redisJobRunnerService' [14:17:22] RECOVERY - test151 php-fpm on test151 is OK: PROCS OK: 13 processes with command name 'php-fpm8.2' [14:17:24] RECOVERY - mon181 NTP time on mon181 is OK: NTP OK: Offset 0.01442375779 secs [14:17:25] RECOVERY - mw182 php-fpm on mw182 is OK: PROCS OK: 31 processes with command name 'php-fpm8.2' [14:17:25] RECOVERY - swiftobject181 ferm_active on swiftobject181 is OK: OK ferm input default policy is set [14:17:25] RECOVERY - mon181 APT on mon181 is OK: APT OK: 2 packages available for upgrade (0 critical updates). [14:17:26] RECOVERY - mw152 Puppet on mw152 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [14:17:27] RECOVERY - test151 poolcounter process on test151 is OK: PROCS OK: 1 process with UID = 999 (poolcounter), command name 'poolcounterd' [14:17:28] RECOVERY - mwtask181 conntrack_table_size on mwtask181 is OK: OK: nf_conntrack is 0 % full [14:17:31] RECOVERY - puppet181 NTP time on puppet181 is OK: NTP OK: Offset 0.09084442258 secs [14:17:38] RECOVERY - puppet181 Puppet on puppet181 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [14:17:39] RECOVERY - test151 JobRunner Service on test151 is OK: PROCS OK: 1 process with args 'redisJobRunnerService' [14:17:40] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 140.21 ms [14:17:44] RECOVERY - test151 PowerDNS Recursor on test151 is OK: DNS OK: 0.112 seconds response time. miraheze.org returns 2607:5300:205:200::1c93,2607:5300:205:200::3121,51.222.12.133,51.222.14.30 [14:18:01] RECOVERY - test151 conntrack_table_size on test151 is OK: OK: nf_conntrack is 0 % full [14:18:13] RECOVERY - swiftobject181 NTP time on swiftobject181 is OK: NTP OK: Offset 0.01984232664 secs [14:18:16] RECOVERY - test151 ferm_active on test151 is OK: OK ferm input default policy is set [14:18:20] RECOVERY - test151 JobChron Service on test151 is OK: PROCS OK: 1 process with args 'redisJobChronService' [14:18:24] RECOVERY - test151 Current Load on test151 is OK: LOAD OK - total load average: 0.30, 0.18, 0.11 [14:19:13] RECOVERY - test151 Redis Process on test151 is OK: PROCS OK: 1 process with args 'redis-server' [14:19:18] RECOVERY - test151 Disk Space on test151 is OK: DISK OK - free space: / 61051MiB (69% inode=85%); [14:19:19] RECOVERY - test151 NTP time on test151 is OK: NTP OK: Offset -0.006741046906 secs [14:19:40] PROBLEM - ping6 on cp41 is WARNING: PING WARNING - Packet loss = 0%, RTA = 139.45 ms [14:20:49] RECOVERY - cloud11 Puppet on cloud11 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [14:21:46] PROBLEM - cp24 HTTPS on cp24 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [14:22:09] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.65, 7.99, 7.94 [14:23:09] RECOVERY - wiki.geoparkcorumbatai.com.br - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.geoparkcorumbatai.com.br' will expire on Tue 23 Apr 2024 12:25:43 PM GMT +0000. [14:23:43] PROBLEM - cp24 HTTPS on cp24 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 1.752 second response time [14:24:05] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.02, 7.97, 7.93 [14:25:32] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mahdiruiz.line.pm All nameservers failed to answer the query. [14:25:34] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/a3266d0f3e52 [14:25:37] [02miraheze/puppet] 07paladox 03a3266d0 - mediawiki::monitoring: support private network [14:25:38] [02puppet] 07paladox created branch 03paladox-patch-12 - 13https://github.com/miraheze/puppet [14:25:39] [02puppet] 07paladox opened pull request 03#3675: mediawiki::monitoring: support private network - 13https://github.com/miraheze/puppet/pull/3675 [14:25:59] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/a3266d0f3e52...cca84d94bb45 [14:26:02] [02miraheze/puppet] 07paladox 03cca84d9 - Update commands.conf [14:26:04] [02puppet] 07paladox synchronize pull request 03#3675: mediawiki::monitoring: support private network - 13https://github.com/miraheze/puppet/pull/3675 [14:26:34] [02puppet] 07paladox closed pull request 03#3675: mediawiki::monitoring: support private network - 13https://github.com/miraheze/puppet/pull/3675 [14:26:36] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/puppet/compare/eb87e758532a...8b0042c29874 [14:26:39] [02miraheze/puppet] 07paladox 038b0042c - mediawiki::monitoring: support private network (#3675) [14:26:41] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-12 [14:26:42] [02puppet] 07paladox deleted branch 03paladox-patch-12 - 13https://github.com/miraheze/puppet [14:28:08] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.74, 7.86, 7.94 [14:29:06] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/8b0042c29874...aaf754354602 [14:29:09] [02miraheze/puppet] 07paladox 03aaf7543 - swift::proxy: support private network [14:30:08] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 0%, RTA = 201.97 ms [14:30:12] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.92, 7.35, 7.99 [14:31:43] RECOVERY - mw151 MediaWiki Rendering on mw151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 7.948 second response time [14:31:52] PROBLEM - ping6 on cp41 is WARNING: PING WARNING - Packet loss = 0%, RTA = 153.95 ms [14:32:17] RECOVERY - mw152 MediaWiki Rendering on mw152 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 8.090 second response time [14:32:53] PROBLEM - mw143 Current Load on mw143 is WARNING: LOAD WARNING - total load average: 11.65, 11.01, 9.77 [14:32:55] PROBLEM - mw141 Current Load on mw141 is WARNING: LOAD WARNING - total load average: 11.32, 9.95, 9.00 [14:33:15] RECOVERY - mwtask181 Puppet on mwtask181 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [14:33:21] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/aaf754354602...fb2790b9087b [14:33:23] [02miraheze/puppet] 07paladox 03fb2790b - cp26: decrease storage to 70g [14:33:45] PROBLEM - mw142 Current Load on mw142 is CRITICAL: LOAD CRITICAL - total load average: 12.68, 11.34, 9.97 [14:34:15] RECOVERY - mw181 MediaWiki Rendering on mw181 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 8.061 second response time [14:34:51] PROBLEM - mw143 Current Load on mw143 is CRITICAL: LOAD CRITICAL - total load average: 17.45, 12.72, 10.50 [14:34:51] PROBLEM - mw141 Current Load on mw141 is CRITICAL: LOAD CRITICAL - total load average: 17.09, 12.01, 9.83 [14:35:09] PROBLEM - cp26 HTTPS on cp26 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [14:35:14] RECOVERY - mwtask181 MediaWiki Rendering on mwtask181 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 5.465 second response time [14:35:14] RECOVERY - mw182 MediaWiki Rendering on mw182 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 7.698 second response time [14:35:28] PROBLEM - cp41 HTTPS on cp41 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [14:35:45] PROBLEM - cp35 HTTPS on cp35 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [14:35:57] PROBLEM - cp51 HTTPS on cp51 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [14:36:04] PROBLEM - ping6 on cp26 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 156.23 ms [14:36:04] PROBLEM - mw142 HTTPS on mw142 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [14:36:05] PROBLEM - cp34 HTTPS on cp34 is CRITICAL: HTTP CRITICAL: HTTP/2 502 - 3370 bytes in 0.987 second response time [14:36:13] PROBLEM - cp24 HTTPS on cp24 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [14:36:14] PROBLEM - mw133 HTTPS on mw133 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [14:37:06] PROBLEM - cp41 Varnish Backends on cp41 is CRITICAL: 3 backends are down. mw131 mw133 mw134 [14:37:18] PROBLEM - cp51 Varnish Backends on cp51 is CRITICAL: 3 backends are down. mw131 mw141 mw134 [14:37:26] PROBLEM - cp25 Varnish Backends on cp25 is CRITICAL: 4 backends are down. mw131 mw132 mw141 mw134 [14:37:30] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/0626938b7953 [14:37:31] [02miraheze/puppet] 07paladox 030626938 - swift: support private network [14:37:32] [02puppet] 07paladox created branch 03paladox-patch-12 - 13https://github.com/miraheze/puppet [14:37:34] [02puppet] 07paladox opened pull request 03#3676: swift: support private network - 13https://github.com/miraheze/puppet/pull/3676 [14:37:41] PROBLEM - cp35 HTTPS on cp35 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.662 second response time [14:37:42] PROBLEM - cp35 Varnish Backends on cp35 is CRITICAL: 4 backends are down. mw131 mw132 mw141 mw143 [14:37:59] PROBLEM - cp24 Varnish Backends on cp24 is CRITICAL: 4 backends are down. mw132 mw141 mw142 mw143 [14:38:07] PROBLEM - ping6 on cp26 is WARNING: PING WARNING - Packet loss = 0%, RTA = 164.54 ms [14:38:09] PROBLEM - mw142 HTTPS on mw142 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 6.817 second response time [14:38:12] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/0626938b7953...8321f2f04b41 [14:38:13] [02miraheze/puppet] 07paladox 038321f2f - Update ac.pp [14:38:14] [02puppet] 07paladox synchronize pull request 03#3676: swift: support private network - 13https://github.com/miraheze/puppet/pull/3676 [14:38:16] PROBLEM - mw133 HTTPS on mw133 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 7.333 second response time [14:38:43] PROBLEM - cp34 Varnish Backends on cp34 is CRITICAL: 3 backends are down. mw142 mw133 mw134 [14:38:44] PROBLEM - cp26 Varnish Backends on cp26 is CRITICAL: 3 backends are down. mw131 mw133 mw134 [14:38:45] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/8321f2f04b41...58c0546274d9 [14:38:47] [02miraheze/puppet] 07paladox 0358c0546 - Update storage.pp [14:38:50] [02puppet] 07paladox synchronize pull request 03#3676: swift: support private network - 13https://github.com/miraheze/puppet/pull/3676 [14:39:02] PROBLEM - mw132 HTTPS on mw132 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [14:39:04] [02puppet] 07paladox closed pull request 03#3676: swift: support private network - 13https://github.com/miraheze/puppet/pull/3676 [14:39:06] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-12 [14:39:08] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±3] 13https://github.com/miraheze/puppet/compare/fb2790b9087b...5243fddb5218 [14:39:11] [02miraheze/puppet] 07paladox 035243fdd - swift: support private network (#3676) [14:39:13] PROBLEM - cp26 HTTPS on cp26 is WARNING: HTTP WARNING: HTTP/2 404 - 3480 bytes in 6.732 second response time [14:39:13] [02puppet] 07paladox deleted branch 03paladox-patch-12 - 13https://github.com/miraheze/puppet [14:39:51] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.33, 7.79, 7.91 [14:39:54] PROBLEM - mw134 HTTPS on mw134 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [14:40:13] PROBLEM - cp34 HTTPS on cp34 is WARNING: HTTP WARNING: HTTP/2 404 - 3480 bytes in 0.573 second response time [14:40:19] PROBLEM - mw132 HTTPS on mw132 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.516 second response time [14:40:43] PROBLEM - cp24 HTTPS on cp24 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 2.605 second response time [14:40:45] RECOVERY - cp34 Varnish Backends on cp34 is OK: All 19 backends are healthy [14:40:47] RECOVERY - swiftproxy181 HTTP on swiftproxy181 is OK: HTTP OK: Status line output matched "HTTP/1.1 404" - 352 bytes in 0.875 second response time [14:40:48] RECOVERY - swiftproxy181 Swift Proxy on swiftproxy181 is OK: TCP OK - 0.000 second response time on 10.0.18.108 port 80 [14:41:00] PROBLEM - mw134 HTTPS on mw134 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 1.782 second response time [14:41:13] RECOVERY - cp51 Varnish Backends on cp51 is OK: All 19 backends are healthy [14:41:14] RECOVERY - swiftproxy181 HTTPS on swiftproxy181 is OK: HTTP OK: Status line output matched "HTTP/1.1 404" - 352 bytes in 0.882 second response time [14:41:22] RECOVERY - cp25 Varnish Backends on cp25 is OK: All 19 backends are healthy [14:41:30] PROBLEM - cp41 HTTPS on cp41 is WARNING: HTTP WARNING: HTTP/2 404 - 3480 bytes in 1.573 second response time [14:41:38] RECOVERY - cp35 Varnish Backends on cp35 is OK: All 19 backends are healthy [14:41:48] RECOVERY - cp24 Varnish Backends on cp24 is OK: All 19 backends are healthy [14:42:02] PROBLEM - cp51 HTTPS on cp51 is WARNING: HTTP WARNING: HTTP/2 404 - 3479 bytes in 1.350 second response time [14:42:12] RECOVERY - cp41 Varnish Backends on cp41 is OK: All 19 backends are healthy [14:42:33] RECOVERY - cp26 Varnish Backends on cp26 is OK: All 19 backends are healthy [14:43:11] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.69, 7.53, 7.32 [14:43:21] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/5243fddb5218...dd0cb742f9d2 [14:43:24] [02miraheze/puppet] 07paladox 03dd0cb74 - Fix [14:45:03] PROBLEM - ping6 on cp26 is CRITICAL: PING CRITICAL - Packet loss = 0%, RTA = 243.90 ms [14:45:07] PROBLEM - mw142 Current Load on mw142 is WARNING: LOAD WARNING - total load average: 8.82, 11.26, 11.07 [14:45:26] RECOVERY - cp26 Disk Space on cp26 is OK: DISK OK - free space: / 15575MiB (16% inode=98%); [14:46:38] RECOVERY - mon181 HTTPS on mon181 is OK: HTTP OK: HTTP/2 200 - 336 bytes in 0.014 second response time [14:46:43] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 11.20, 8.82, 6.97 [14:47:05] PROBLEM - ping6 on cp26 is WARNING: PING WARNING - Packet loss = 0%, RTA = 181.17 ms [14:47:11] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/dd0cb742f9d2...6438bed170f3 [14:47:12] [02miraheze/puppet] 07paladox 036438bed - cp41: decrease storage size to 70g [14:48:02] PROBLEM - puppet181 NTP time on puppet181 is WARNING: NTP WARNING: Offset 0.1008801162 secs [14:48:13] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/6438bed170f3...ded348f2bbfb [14:48:16] [02miraheze/puppet] 07paladox 03ded348f - cp51: decrease storage size to 70g [14:48:37] PROBLEM - mw141 Current Load on mw141 is WARNING: LOAD WARNING - total load average: 7.15, 10.68, 11.11 [14:48:39] PROBLEM - mw143 Current Load on mw143 is WARNING: LOAD WARNING - total load average: 8.95, 11.53, 11.91 [14:49:47] RECOVERY - cp41 Disk Space on cp41 is OK: DISK OK - free space: / 15186MiB (16% inode=98%); [14:50:14] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 0%, RTA = 219.51 ms [14:50:45] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/6da84ad07522 [14:50:48] [02miraheze/puppet] 07paladox 036da84ad - monitoring::services add exemption for cp26 ping [14:50:50] [02puppet] 07paladox created branch 03paladox-patch-12 - 13https://github.com/miraheze/puppet [14:50:53] [02puppet] 07paladox opened pull request 03#3677: monitoring::services add exemption for cp26 ping - 13https://github.com/miraheze/puppet/pull/3677 [14:50:55] [02puppet] 07paladox closed pull request 03#3677: monitoring::services add exemption for cp26 ping - 13https://github.com/miraheze/puppet/pull/3677 [14:50:58] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/ded348f2bbfb...f8d4ee9de755 [14:51:01] [02miraheze/puppet] 07paladox 03f8d4ee9 - monitoring::services add exemption for cp26 ping (#3677) [14:51:04] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-12 [14:51:07] [02puppet] 07paladox deleted branch 03paladox-patch-12 - 13https://github.com/miraheze/puppet [14:52:10] RECOVERY - cp51 Disk Space on cp51 is OK: DISK OK - free space: / 18354MiB (19% inode=98%); [14:52:33] PROBLEM - ping6 on cp41 is WARNING: PING WARNING - Packet loss = 0%, RTA = 137.06 ms [14:52:39] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/f8d4ee9de755...a642214f9492 [14:52:40] [02miraheze/puppet] 07paladox 03a642214 - monitoring::services: add exemption for cp41 ping [14:53:14] RECOVERY - ping6 on cp26 is OK: PING OK - Packet loss = 0%, RTA = 141.75 ms [14:53:35] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp35.miraheze.org - CNAME OK [14:53:55] RECOVERY - mw141 Current Load on mw141 is OK: LOAD OK - total load average: 6.44, 8.09, 9.85 [14:54:26] PROBLEM - mw182 HTTPS on mw182 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.035 second response time [14:54:44] PROBLEM - test151 HTTPS on test151 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.065 second response time [14:54:47] RECOVERY - mw142 Current Load on mw142 is OK: LOAD OK - total load average: 7.30, 9.01, 10.09 [14:54:49] PROBLEM - mw152 HTTPS on mw152 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.066 second response time [14:55:18] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 138.39 ms [14:55:54] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/28053a539f36 [14:55:57] [02miraheze/puppet] 07paladox 0328053a5 - ssl::web: support private network [14:56:00] [02puppet] 07paladox created branch 03paladox-patch-12 - 13https://github.com/miraheze/puppet [14:56:04] [02puppet] 07paladox opened pull request 03#3678: ssl::web: support private network - 13https://github.com/miraheze/puppet/pull/3678 [14:56:05] [02puppet] 07paladox closed pull request 03#3678: ssl::web: support private network - 13https://github.com/miraheze/puppet/pull/3678 [14:56:08] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/a642214f9492...0d0485591dfb [14:56:09] [02miraheze/puppet] 07paladox 030d04855 - ssl::web: support private network (#3678) [14:56:12] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-12 [14:56:15] [02puppet] 07paladox deleted branch 03paladox-patch-12 - 13https://github.com/miraheze/puppet [14:56:16] PROBLEM - mw181 HTTPS on mw181 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.035 second response time [14:57:30] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [14:57:54] RECOVERY - matomo151 HTTPS on matomo151 is OK: HTTP OK: HTTP/2 200 - 552 bytes in 0.138 second response time [14:58:21] RECOVERY - test151 MediaWiki Rendering on test151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 9.797 second response time [14:58:25] RECOVERY - mw143 Current Load on mw143 is OK: LOAD OK - total load average: 8.18, 8.60, 10.18 [14:58:34] RECOVERY - puppet181 MirahezeRenewSsl on puppet181 is OK: TCP OK - 0.000 second response time on 10.0.18.100 port 5000 [14:58:49] RECOVERY - swiftobject181 Swift Object Service on swiftobject181 is OK: TCP OK - 0.000 second response time on 10.0.18.107 port 6000 [14:58:57] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/0d0485591dfb...1ab53c70621c [14:58:58] [02miraheze/puppet] 07paladox 031ab53c7 - grafana: support private ip in monitoring [15:00:54] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/1ab53c70621c...5065625e19a6 [15:00:55] [02miraheze/puppet] 07paladox 035065625 - icingaweb2: support private ip in monitoring [15:01:07] RECOVERY - puppet181 NTP time on puppet181 is OK: NTP OK: Offset 0.08501785994 secs [15:02:13] RECOVERY - mon181 grafana.miraheze.org HTTPS on mon181 is OK: HTTP OK: HTTP/1.1 200 OK - 45233 bytes in 0.017 second response time [15:02:19] RECOVERY - mon181 icinga.miraheze.org HTTPS on mon181 is OK: HTTP OK: HTTP/1.1 302 Found - 297 bytes in 0.009 second response time [15:04:25] PROBLEM - mwtask181 HTTPS on mwtask181 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.038 second response time [15:10:27] PROBLEM - mw134 HTTPS on mw134 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [15:11:20] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/5065625e19a6...e90429b486e5 [15:11:21] RECOVERY - Host swiftobject151 is UP: PING OK - Packet loss = 0%, RTA = 0.19 ms [15:11:23] [02miraheze/puppet] 07paladox 03e90429b - roundcubemail: support private ip in firewall [15:11:43] RECOVERY - swiftobject151 ferm_active on swiftobject151 is OK: OK ferm input default policy is set [15:11:43] RECOVERY - swiftobject151 PowerDNS Recursor on swiftobject151 is OK: DNS OK: 0.035 seconds response time. miraheze.org returns 2607:5300:205:200::1c93,2607:5300:205:200::3121,51.222.12.133,51.222.14.30 [15:11:44] RECOVERY - swiftobject151 conntrack_table_size on swiftobject151 is OK: OK: nf_conntrack is 0 % full [15:11:48] RECOVERY - swiftobject151 Disk Space on swiftobject151 is OK: DISK OK - free space: / 1332772MiB (99% inode=99%); [15:11:48] RECOVERY - swiftobject151 Current Load on swiftobject151 is OK: LOAD OK - total load average: 0.13, 0.05, 0.01 RECOVERY - swiftobject151 Swift Object Service on swiftobject151 is OK: TCP OK - 0.000 second response time on 10.0.15.117 port 6000 [15:11:48] RECOVERY - swiftobject151 NTP time on swiftobject151 is OK: NTP OK: Offset 0.006392389536 secs [15:11:53] RECOVERY - swiftobject151 APT on swiftobject151 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [15:11:53] RECOVERY - swiftobject151 SSH on swiftobject151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u2 (protocol 2.0) [15:12:03] RECOVERY - swiftobject151 Puppet on swiftobject151 is OK: OK: Puppet is currently enabled, last run 24 minutes ago with 0 failures [15:12:22] PROBLEM - mw134 HTTPS on mw134 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.726 second response time [15:12:30] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [15:14:32] PROBLEM - mw151 HTTPS on mw151 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.074 second response time [15:14:33] RECOVERY - mail121 HTTPS on mail121 is OK: HTTP OK: HTTP/2 301 - 227 bytes in 0.449 second response time [15:15:06] RECOVERY - mail121 webmail.miraheze.org HTTPS on mail121 is OK: HTTP OK: HTTP/1.1 200 OK - 6169 bytes in 2.621 second response time [15:20:06] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 0%, RTA = 261.15 ms [15:20:23] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/762bdb0bc62f...df9bbcfd606c [15:20:25] [02miraheze/dns] 07paladox 03df9bbcf - Fix [15:22:05] PROBLEM - ping6 on cp51 is WARNING: PING WARNING - Packet loss = 0%, RTA = 186.04 ms [15:26:43] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/e90429b486e5...18d4219cb2c2 [15:26:46] [02miraheze/puppet] 07paladox 0318d4219 - monitoring::services: exempt cp51 from ping [15:27:54] PROBLEM - mw143 Current Load on mw143 is CRITICAL: LOAD CRITICAL - total load average: 16.28, 10.93, 9.01 [15:27:58] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 161.47 ms [15:29:37] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 2.62, 3.19, 3.98 [15:29:54] RECOVERY - mw143 Current Load on mw143 is OK: LOAD OK - total load average: 9.40, 10.02, 8.91 [15:31:31] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/df9bbcfd606c...1415c24f85d2 [15:31:32] [02miraheze/dns] 07paladox 031415c24 - Update cloud17 ip [15:33:28] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 4.15, 3.53, 3.94 [15:34:31] PROBLEM - ping6 on cloud17 is CRITICAL: PING CRITICAL - Packet loss = 100% [15:34:37] PROBLEM - cloud17 SSH on cloud17 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:35:23] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.57, 3.52, 3.89 [15:35:38] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/1415c24f85d2...30f56e7f84d1 [15:35:41] [02miraheze/dns] 07paladox 0330f56e7 - Fix cloud17 ip [15:36:38] PROBLEM - cloud17 Current Load on cloud17 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:36:52] PROBLEM - Host cloud17 is DOWN: PING CRITICAL - Packet loss = 100% [15:37:19] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 4.29, 3.72, 3.91 [15:39:14] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.24, 3.47, 3.80 [15:40:42] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 5.53, 6.98, 7.80 [15:41:58] [02miraheze/dns] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/dns/commit/4bb604f09390 [15:42:00] [02miraheze/dns] 07paladox 034bb604f - Add vms under cloud17 to dns [15:42:02] [02dns] 07paladox created branch 03paladox-patch-1 - 13https://github.com/miraheze/dns [15:42:04] [02dns] 07paladox opened pull request 03#464: Add vms under cloud17 to dns - 13https://github.com/miraheze/dns/pull/464 [15:42:08] [02dns] 07paladox closed pull request 03#464: Add vms under cloud17 to dns - 13https://github.com/miraheze/dns/pull/464 [15:42:10] [02miraheze/dns] 07paladox deleted branch 03paladox-patch-1 [15:42:12] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/30f56e7f84d1...b00c2de549d0 [15:42:15] [02miraheze/dns] 07paladox 03b00c2de - Add vms under cloud17 to dns (#464) [15:42:16] [02dns] 07paladox deleted branch 03paladox-patch-1 - 13https://github.com/miraheze/dns [15:46:41] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.27, 7.61, 7.76 [15:48:28] RECOVERY - Host cloud17 is UP: PING OK - Packet loss = 0%, RTA = 0.19 ms [15:48:39] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.33, 7.42, 7.67 [15:49:12] PROBLEM - mw143 HTTPS on mw143 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [15:49:17] PROBLEM - cp41 HTTPS on cp41 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [15:49:21] PROBLEM - cp24 HTTPS on cp24 is CRITICAL: HTTP CRITICAL: HTTP/2 502 - 3370 bytes in 0.466 second response time [15:49:26] PROBLEM - cp26 HTTPS on cp26 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [15:49:26] PROBLEM - mw142 HTTPS on mw142 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [15:49:44] PROBLEM - cp34 HTTPS on cp34 is CRITICAL: HTTP CRITICAL: HTTP/2 503 - 3363 bytes in 0.224 second response time [15:49:47] PROBLEM - cp25 HTTPS on cp25 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [15:49:51] RECOVERY - cloud17 SSH on cloud17 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u2 (protocol 2.0) [15:50:11] RECOVERY - cloud17 Current Load on cloud17 is OK: LOAD OK - total load average: 0.25, 0.22, 0.09 [15:50:11] RECOVERY - ping6 on cloud17 is OK: PING OK - Packet loss = 0%, RTA = 0.16 ms [15:50:27] PROBLEM - cp35 HTTPS on cp35 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [15:50:28] PROBLEM - cp51 HTTPS on cp51 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [15:50:40] PROBLEM - mw143 Current Load on mw143 is CRITICAL: LOAD CRITICAL - total load average: 13.37, 11.34, 9.54 [15:50:49] PROBLEM - mw134 HTTPS on mw134 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [15:51:11] PROBLEM - cp34 Varnish Backends on cp34 is CRITICAL: 4 backends are down. mw131 mw132 mw133 mw134 [15:51:20] PROBLEM - cp25 Varnish Backends on cp25 is CRITICAL: 4 backends are down. mw131 mw132 mw133 mw134 [15:51:22] PROBLEM - mw142 Current Load on mw142 is CRITICAL: LOAD CRITICAL - total load average: 18.75, 12.80, 9.72 [15:51:42] PROBLEM - cp35 Varnish Backends on cp35 is CRITICAL: 4 backends are down. mw131 mw141 mw142 mw143 [15:51:52] PROBLEM - cp34 HTTPS on cp34 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 8.710 second response time [15:51:52] PROBLEM - cp25 HTTPS on cp25 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 6.110 second response time [15:51:55] PROBLEM - cp51 Varnish Backends on cp51 is CRITICAL: 4 backends are down. mw131 mw141 mw142 mw143 [15:51:57] PROBLEM - cp41 Varnish Backends on cp41 is CRITICAL: 4 backends are down. mw131 mw141 mw142 mw143 [15:51:57] PROBLEM - mw141 Current Load on mw141 is CRITICAL: LOAD CRITICAL - total load average: 19.85, 14.67, 10.60 [15:52:04] PROBLEM - mw132 HTTPS on mw132 is CRITICAL: HTTP CRITICAL: HTTP/2 502 - 170 bytes in 0.377 second response time [15:52:08] PROBLEM - cp24 Varnish Backends on cp24 is CRITICAL: 6 backends are down. mw131 mw132 mw141 mw142 mw133 mw143 [15:52:09] PROBLEM - mw133 HTTPS on mw133 is CRITICAL: HTTP CRITICAL: HTTP/2 502 - 170 bytes in 0.375 second response time [15:52:13] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 1.50, 2.65, 3.25 [15:52:26] PROBLEM - cp35 HTTPS on cp35 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.813 second response time [15:52:45] PROBLEM - mw134 HTTPS on mw134 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.478 second response time [15:53:09] RECOVERY - cp34 Varnish Backends on cp34 is OK: All 19 backends are healthy [15:53:10] PROBLEM - cp41 HTTPS on cp41 is WARNING: HTTP WARNING: HTTP/2 404 - 3457 bytes in 2.523 second response time [15:53:16] RECOVERY - cp25 Varnish Backends on cp25 is OK: All 19 backends are healthy [15:53:20] PROBLEM - mw143 HTTPS on mw143 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.565 second response time [15:53:27] PROBLEM - cp26 HTTPS on cp26 is WARNING: HTTP WARNING: HTTP/2 404 - 3479 bytes in 0.659 second response time [15:53:27] PROBLEM - cp24 HTTPS on cp24 is WARNING: HTTP WARNING: HTTP/2 404 - 3480 bytes in 1.142 second response time [15:53:34] PROBLEM - mw142 HTTPS on mw142 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.500 second response time [15:53:42] RECOVERY - cp35 Varnish Backends on cp35 is OK: All 19 backends are healthy [15:53:51] RECOVERY - cp51 Varnish Backends on cp51 is OK: All 19 backends are healthy [15:53:58] RECOVERY - cp41 Varnish Backends on cp41 is OK: All 19 backends are healthy [15:54:03] PROBLEM - mw132 HTTPS on mw132 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.693 second response time [15:54:05] PROBLEM - mw133 HTTPS on mw133 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.745 second response time [15:54:05] RECOVERY - cp24 Varnish Backends on cp24 is OK: All 19 backends are healthy [15:54:29] PROBLEM - cp51 HTTPS on cp51 is WARNING: HTTP WARNING: HTTP/2 404 - 3478 bytes in 3.627 second response time [15:55:12] PROBLEM - mw142 Current Load on mw142 is WARNING: LOAD WARNING - total load average: 10.34, 11.82, 10.01 [15:55:51] PROBLEM - mw141 Current Load on mw141 is WARNING: LOAD WARNING - total load average: 6.29, 11.38, 10.27 [15:56:01] PROBLEM - cp34 HTTPS on cp34 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [15:56:11] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.98, 3.22, 3.32 [15:56:43] PROBLEM - mw143 Current Load on mw143 is WARNING: LOAD WARNING - total load average: 9.77, 10.98, 10.21 [15:58:02] PROBLEM - cp34 HTTPS on cp34 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 1.599 second response time [15:58:08] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 3.24, 3.25, 3.32 [15:58:41] PROBLEM - mw143 Current Load on mw143 is CRITICAL: LOAD CRITICAL - total load average: 12.96, 11.36, 10.42 [15:59:44] PROBLEM - mw141 Current Load on mw141 is CRITICAL: LOAD CRITICAL - total load average: 14.65, 12.87, 11.05 [16:00:40] PROBLEM - mw143 Current Load on mw143 is WARNING: LOAD WARNING - total load average: 9.15, 10.66, 10.29 [16:00:56] RECOVERY - mw142 Current Load on mw142 is OK: LOAD OK - total load average: 8.58, 10.15, 9.82 [16:01:41] PROBLEM - mw141 Current Load on mw141 is WARNING: LOAD WARNING - total load average: 7.06, 10.39, 10.35 [16:02:40] RECOVERY - mw143 Current Load on mw143 is OK: LOAD OK - total load average: 6.11, 9.07, 9.76 [16:03:37] RECOVERY - mw141 Current Load on mw141 is OK: LOAD OK - total load average: 7.19, 9.50, 10.03 [16:03:55] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/b00c2de549d0...33020cc557bb [16:03:56] [02miraheze/dns] 07paladox 0333020cc - Fix cloud17 ip [16:04:13] PROBLEM - cp51 HTTPS on cp51 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [16:04:17] PROBLEM - cp34 HTTPS on cp34 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [16:06:09] PROBLEM - cp51 HTTPS on cp51 is WARNING: HTTP WARNING: HTTP/2 404 - 3478 bytes in 1.300 second response time [16:06:17] PROBLEM - cp34 HTTPS on cp34 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.640 second response time [16:06:27] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 5.49, 6.25, 6.74 [16:06:34] PROBLEM - ping6 on cloud17 is CRITICAL: PING CRITICAL - Packet loss = 100% [16:06:40] PROBLEM - mw143 Current Load on mw143 is WARNING: LOAD WARNING - total load average: 10.39, 11.24, 10.59 [16:06:45] PROBLEM - cloud17 APT on cloud17 is CRITICAL: connect to address 2602:294:0:b23::100 port 5666: No route to hostconnect to host 2602:294:0:b23::100 port 5666: No route to host [16:06:45] PROBLEM - cloud17 Current Load on cloud17 is CRITICAL: connect to address 2602:294:0:b23::100 port 5666: No route to hostconnect to host 2602:294:0:b23::100 port 5666: No route to host [16:06:50] PROBLEM - mw142 Current Load on mw142 is WARNING: LOAD WARNING - total load average: 9.93, 10.80, 10.19 [16:07:16] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 5.53, 4.10, 2.32 [16:07:38] PROBLEM - mw141 Current Load on mw141 is WARNING: LOAD WARNING - total load average: 9.96, 10.65, 10.47 [16:07:39] PROBLEM - Host cloud17 is DOWN: PING CRITICAL - Packet loss = 100% [16:08:46] PROBLEM - mw132 Disk Space on mw132 is WARNING: DISK WARNING - free space: / 2608MiB (10% inode=69%); [16:08:49] RECOVERY - mw142 Current Load on mw142 is OK: LOAD OK - total load average: 8.10, 9.80, 9.89 [16:09:12] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 1.54, 3.12, 2.17 [16:09:35] RECOVERY - mw141 Current Load on mw141 is OK: LOAD OK - total load average: 7.80, 9.35, 10.00 [16:10:23] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.87, 6.84, 6.89 [16:12:22] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 6.21, 6.60, 6.80 [16:14:40] PROBLEM - mw143 Current Load on mw143 is CRITICAL: LOAD CRITICAL - total load average: 14.48, 10.94, 10.48 [16:15:39] PROBLEM - mw142 HTTPS on mw142 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10000 milliseconds with 0 bytes received [16:15:55] PROBLEM - cp26 Current Load on cp26 is CRITICAL: LOAD CRITICAL - total load average: 8.61, 7.94, 6.42 [16:16:18] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.57, 7.29, 7.04 [16:17:00] RECOVERY - mw143 Current Load on mw143 is OK: LOAD OK - total load average: 6.71, 9.18, 9.89 [16:17:16] RECOVERY - Host cloud17 is UP: PING OK - Packet loss = 0%, RTA = 0.16 ms [16:17:29] RECOVERY - cloud17 APT on cloud17 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [16:17:38] PROBLEM - mw142 HTTPS on mw142 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.684 second response time [16:17:38] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.84, 3.49, 3.28 [16:17:55] PROBLEM - cp26 Current Load on cp26 is WARNING: LOAD WARNING - total load average: 7.09, 7.77, 6.55 [16:18:16] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.03, 7.35, 7.08 [16:18:39] RECOVERY - cloud17 Current Load on cloud17 is OK: LOAD OK - total load average: 0.08, 0.09, 0.08 [16:19:04] RECOVERY - ping6 on cloud17 is OK: PING OK - Packet loss = 0%, RTA = 0.14 ms [16:19:16] !log [salt-user@mwtask141] Renamed hypercanewiki to megacanewiki using renamewiki.py [16:19:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:20:17] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.13, 7.35, 7.13 [16:21:32] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 2.96, 3.36, 3.29 [16:23:32] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/18d4219cb2c2...ab9d225db9d2 [16:23:34] [02miraheze/puppet] 07paladox 03ab9d225 - Fix [16:24:04] PROBLEM - cloud14 Puppet on cloud14 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ferm] [16:24:20] PROBLEM - cloud14 ferm_active on cloud14 is CRITICAL: ERROR ferm input drop default policy not set, ferm might not have been started correctly [16:24:34] PROBLEM - cp35 HTTPS on cp35 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [16:25:10] PROBLEM - mw131 HTTPS on mw131 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [16:25:14] PROBLEM - cp24 HTTPS on cp24 is CRITICAL: HTTP CRITICAL: HTTP/2 502 - 3370 bytes in 0.462 second response time [16:25:32] PROBLEM - cp51 HTTPS on cp51 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [16:25:36] PROBLEM - mw142 HTTPS on mw142 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [16:25:48] RECOVERY - cp26 Current Load on cp26 is OK: LOAD OK - total load average: 4.66, 6.40, 6.45 [16:26:05] RECOVERY - cloud14 Puppet on cloud14 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:26:06] PROBLEM - cp25 HTTPS on cp25 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [16:26:13] PROBLEM - cp34 HTTPS on cp34 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [16:26:15] PROBLEM - mw142 Current Load on mw142 is CRITICAL: LOAD CRITICAL - total load average: 14.70, 12.02, 10.25 [16:26:16] RECOVERY - cloud14 ferm_active on cloud14 is OK: OK ferm input default policy is set [16:26:34] PROBLEM - mw134 HTTPS on mw134 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [16:26:34] PROBLEM - cp35 HTTPS on cp35 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 2.059 second response time [16:26:46] PROBLEM - mw143 Current Load on mw143 is CRITICAL: LOAD CRITICAL - total load average: 14.36, 11.95, 10.45 [16:26:58] PROBLEM - mw141 Current Load on mw141 is CRITICAL: LOAD CRITICAL - total load average: 16.19, 12.36, 10.24 [16:27:05] PROBLEM - mw131 HTTPS on mw131 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.552 second response time [16:27:08] PROBLEM - cp51 Varnish Backends on cp51 is CRITICAL: 3 backends are down. mw131 mw133 mw134 [16:27:12] [02miraheze/dns] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/33020cc557bb...8eee69ecbb10 [16:27:14] PROBLEM - cp24 HTTPS on cp24 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.603 second response time [16:27:14] [02miraheze/dns] 07AgentIsai 038eee69e - Update cloud17 IP [16:27:16] PROBLEM - cp25 Varnish Backends on cp25 is CRITICAL: 1 backends are down. mw134 [16:27:21] PROBLEM - cp24 Varnish Backends on cp24 is CRITICAL: 1 backends are down. mw134 [16:27:22] PROBLEM - ping6 on cloud17 is CRITICAL: PING CRITICAL - Packet loss = 100% [16:27:28] PROBLEM - cp51 HTTPS on cp51 is WARNING: HTTP WARNING: HTTP/2 404 - 3456 bytes in 1.290 second response time [16:27:32] PROBLEM - mw142 HTTPS on mw142 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.542 second response time [16:27:59] PROBLEM - cloud17 SSH on cloud17 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:28:05] PROBLEM - cp25 HTTPS on cp25 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 1.760 second response time [16:28:08] PROBLEM - cp34 HTTPS on cp34 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.585 second response time [16:28:33] PROBLEM - mw134 HTTPS on mw134 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.494 second response time [16:28:36] PROBLEM - Host cloud17 is DOWN: PING CRITICAL - Packet loss = 100% [16:29:04] RECOVERY - cp51 Varnish Backends on cp51 is OK: All 19 backends are healthy [16:29:13] RECOVERY - cp25 Varnish Backends on cp25 is OK: All 19 backends are healthy [16:29:21] RECOVERY - cp24 Varnish Backends on cp24 is OK: All 19 backends are healthy [16:30:05] PROBLEM - mw142 Current Load on mw142 is WARNING: LOAD WARNING - total load average: 8.62, 10.84, 10.24 [16:30:16] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 5.51, 6.33, 6.75 [16:30:22] PROBLEM - mw141 HTTPS on mw141 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [16:30:36] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [16:30:41] PROBLEM - mw143 Current Load on mw143 is WARNING: LOAD WARNING - total load average: 8.93, 11.11, 10.57 [16:30:55] PROBLEM - mw141 Current Load on mw141 is WARNING: LOAD WARNING - total load average: 8.08, 10.72, 10.15 [16:32:01] RECOVERY - mw142 Current Load on mw142 is OK: LOAD OK - total load average: 8.16, 10.01, 10.01 [16:32:19] PROBLEM - mw141 HTTPS on mw141 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.581 second response time [16:32:53] RECOVERY - mw141 Current Load on mw141 is OK: LOAD OK - total load average: 6.92, 9.35, 9.71 [16:35:13] [02miraheze/dns] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/8eee69ecbb10...c0702bf3f294 [16:35:14] [02miraheze/dns] 07AgentIsai 03c0702bf - Update cloud16 IP [16:36:34] RECOVERY - mw143 Current Load on mw143 is OK: LOAD OK - total load average: 7.63, 9.37, 9.96 [16:37:50] PROBLEM - cloud16 SSH on cloud16 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:38:16] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 8.38, 7.55, 7.09 [16:38:29] PROBLEM - ping6 on cloud16 is CRITICAL: PING CRITICAL - Packet loss = 100% [16:38:40] PROBLEM - Host cloud16 is DOWN: PING CRITICAL - Packet loss = 100% [16:39:07] PROBLEM - cp51 HTTPS on cp51 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10000 milliseconds with 0 bytes received [16:40:16] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.57, 7.03, 6.95 [16:40:35] PROBLEM - cp41 HTTPS on cp41 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10001 milliseconds with 0 bytes received [16:41:02] PROBLEM - cp51 HTTPS on cp51 is WARNING: HTTP WARNING: HTTP/2 404 - 3457 bytes in 1.303 second response time [16:41:31] PROBLEM - cp34 Varnish Backends on cp34 is CRITICAL: 1 backends are down. mw133 [16:42:31] PROBLEM - cp41 HTTPS on cp41 is WARNING: HTTP WARNING: HTTP/2 404 - 3480 bytes in 1.480 second response time [16:43:12] PROBLEM - cp26 HTTPS on cp26 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10000 milliseconds with 0 bytes received [16:43:22] PROBLEM - cp24 HTTPS on cp24 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [16:43:26] RECOVERY - cp34 Varnish Backends on cp34 is OK: All 19 backends are healthy [16:44:16] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 6.27, 6.65, 6.80 [16:45:10] PROBLEM - cp26 HTTPS on cp26 is WARNING: HTTP WARNING: HTTP/2 404 - 3480 bytes in 0.827 second response time [16:45:20] PROBLEM - mw143 Current Load on mw143 is WARNING: LOAD WARNING - total load average: 8.50, 10.46, 10.34 [16:45:22] PROBLEM - cp24 HTTPS on cp24 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.728 second response time [16:45:36] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [16:46:29] PROBLEM - mw151 HTTPS on mw151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 7 - Failed to connect to mw151.wikitide.net port 443 after 2592 ms: Couldn't connect to server [16:46:51] PROBLEM - ping on mw151 is CRITICAL: CRITICAL - Host Unreachable (10.0.15.114) [16:47:01] PROBLEM - mw152 HTTPS on mw152 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 7 - Failed to connect to mw152.wikitide.net port 443 after 2064 ms: Couldn't connect to server [16:47:10] RECOVERY - mw143 Current Load on mw143 is OK: LOAD OK - total load average: 6.20, 9.11, 9.87 [16:47:33] [Grafana] !sre FIRING: There has been a rise in the MediaWiki exception rate https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [16:47:36] PROBLEM - prometheus151 NTP time on prometheus151 is CRITICAL: connect to address 10.0.15.116 port 5666: No route to hostconnect to host 10.0.15.116 port 5666: No route to host [16:47:42] PROBLEM - test151 HTTPS on test151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 7 - Failed to connect to test151.wikitide.net port 443 after 2438 ms: Couldn't connect to server [16:47:49] PROBLEM - os151 Current Load on os151 is CRITICAL: connect to address 10.0.15.111 port 5666: No route to hostconnect to host 10.0.15.111 port 5666: No route to host [16:47:50] PROBLEM - swiftobject151 Swift Object Service on swiftobject151 is CRITICAL: connect to address 10.0.15.117 and port 6000: No route to host [16:47:50] PROBLEM - cp26 Varnish Backends on cp26 is CRITICAL: 2 backends are down. mw151 mw152 [16:47:51] PROBLEM - mw152 conntrack_table_size on mw152 is CRITICAL: connect to address 10.0.15.115 port 5666: No route to hostconnect to host 10.0.15.115 port 5666: No route to host [16:47:51] PROBLEM - ping on test151 is CRITICAL: CRITICAL - Host Unreachable (10.0.15.118) [16:47:51] PROBLEM - cp35 Varnish Backends on cp35 is CRITICAL: 2 backends are down. mw151 mw152 [16:47:53] PROBLEM - Host os151 is DOWN: CRITICAL - Host Unreachable (10.0.15.111) [16:47:55] PROBLEM - cp51 Varnish Backends on cp51 is CRITICAL: 2 backends are down. mw151 mw152 [16:47:56] PROBLEM - swiftobject151 APT on swiftobject151 is CRITICAL: connect to address 10.0.15.117 port 5666: No route to hostconnect to host 10.0.15.117 port 5666: No route to host [16:47:56] PROBLEM - mem151 memcached on mem151 is CRITICAL: connect to address 10.0.15.113 and port 11211: No route to host [16:47:56] PROBLEM - mem151 NTP time on mem151 is CRITICAL: connect to address 10.0.15.113 port 5666: No route to hostconnect to host 10.0.15.113 port 5666: No route to host [16:47:56] PROBLEM - matomo151 NTP time on matomo151 is CRITICAL: connect to address 10.0.15.112 port 5666: No route to hostconnect to host 10.0.15.112 port 5666: No route to host [16:47:57] PROBLEM - mw151 Current Load on mw151 is CRITICAL: connect to address 10.0.15.114 port 5666: No route to hostconnect to host 10.0.15.114 port 5666: No route to host [16:47:59] PROBLEM - matomo151 ferm_active on matomo151 is CRITICAL: connect to address 10.0.15.112 port 5666: No route to hostconnect to host 10.0.15.112 port 5666: No route to host [16:48:02] PROBLEM - matomo151 SSH on matomo151 is CRITICAL: connect to address 10.0.15.112 and port 22: No route to host [16:48:04] PROBLEM - swiftobject151 ferm_active on swiftobject151 is CRITICAL: connect to address 10.0.15.117 port 5666: No route to hostconnect to host 10.0.15.117 port 5666: No route to host [16:48:05] PROBLEM - mw151 PowerDNS Recursor on mw151 is CRITICAL: connect to address 10.0.15.114 port 5666: No route to hostconnect to host 10.0.15.114 port 5666: No route to host [16:48:05] PROBLEM - mw151 php-fpm on mw151 is CRITICAL: connect to address 10.0.15.114 port 5666: No route to hostconnect to host 10.0.15.114 port 5666: No route to host [16:48:05] PROBLEM - mem151 PowerDNS Recursor on mem151 is CRITICAL: connect to address 10.0.15.113 port 5666: No route to hostconnect to host 10.0.15.113 port 5666: No route to host [16:48:05] PROBLEM - mem151 ferm_active on mem151 is CRITICAL: connect to address 10.0.15.113 port 5666: No route to hostconnect to host 10.0.15.113 port 5666: No route to host [16:48:05] PROBLEM - matomo151 conntrack_table_size on matomo151 is CRITICAL: connect to address 10.0.15.112 port 5666: No route to hostconnect to host 10.0.15.112 port 5666: No route to host [16:48:06] PROBLEM - cp26 Current Load on cp26 is WARNING: LOAD WARNING - total load average: 7.42, 6.70, 6.15 [16:48:07] PROBLEM - swiftobject151 Current Load on swiftobject151 is CRITICAL: connect to address 10.0.15.117 port 5666: No route to hostconnect to host 10.0.15.117 port 5666: No route to host [16:48:08] PROBLEM - mw152 Puppet on mw152 is CRITICAL: connect to address 10.0.15.115 port 5666: No route to hostconnect to host 10.0.15.115 port 5666: No route to host [16:48:09] PROBLEM - prometheus151 Disk Space on prometheus151 is CRITICAL: connect to address 10.0.15.116 port 5666: No route to hostconnect to host 10.0.15.116 port 5666: No route to host [16:48:09] PROBLEM - Host prometheus151 is DOWN: CRITICAL - Host Unreachable (10.0.15.116) [16:48:12] PROBLEM - db151 Backups SQL mhglobal on db151 is CRITICAL: connect to address 10.0.15.110 port 5666: No route to hostconnect to host 10.0.15.110 port 5666: No route to host [16:48:12] PROBLEM - matomo151 Current Load on matomo151 is CRITICAL: connect to address 10.0.15.112 port 5666: No route to hostconnect to host 10.0.15.112 port 5666: No route to host [16:48:14] PROBLEM - test151 ferm_active on test151 is CRITICAL: connect to address 10.0.15.118 port 5666: No route to hostconnect to host 10.0.15.118 port 5666: No route to host [16:48:16] PROBLEM - ping on mw152 is CRITICAL: CRITICAL - Host Unreachable (10.0.15.115) [16:48:17] PROBLEM - swiftobject151 SSH on swiftobject151 is CRITICAL: connect to address 10.0.15.117 and port 22: No route to host [16:48:17] PROBLEM - ping on swiftobject151 is CRITICAL: CRITICAL - Host Unreachable (10.0.15.117) [16:48:18] PROBLEM - cp24 Varnish Backends on cp24 is CRITICAL: 2 backends are down. mw151 mw152 [16:48:20] PROBLEM - db151 MariaDB on db151 is CRITICAL: Can't connect to server on 'db151.wikitide.net' (115) [16:48:22] PROBLEM - matomo151 Disk Space on matomo151 is CRITICAL: connect to address 10.0.15.112 port 5666: No route to hostconnect to host 10.0.15.112 port 5666: No route to host [16:48:24] PROBLEM - mem151 Current Load on mem151 is CRITICAL: connect to address 10.0.15.113 port 5666: No route to hostconnect to host 10.0.15.113 port 5666: No route to host [16:48:24] PROBLEM - db151 NTP time on db151 is CRITICAL: connect to address 10.0.15.110 port 5666: No route to hostconnect to host 10.0.15.110 port 5666: No route to host [16:48:25] PROBLEM - matomo151 Redis Process on matomo151 is CRITICAL: connect to address 10.0.15.112 port 5666: No route to hostconnect to host 10.0.15.112 port 5666: No route to host [16:48:26] PROBLEM - test151 php-fpm on test151 is CRITICAL: connect to address 10.0.15.118 port 5666: No route to hostconnect to host 10.0.15.118 port 5666: No route to host [16:48:27] PROBLEM - swiftobject151 conntrack_table_size on swiftobject151 is CRITICAL: connect to address 10.0.15.117 port 5666: No route to hostconnect to host 10.0.15.117 port 5666: No route to host [16:48:27] PROBLEM - swiftobject151 Puppet on swiftobject151 is CRITICAL: connect to address 10.0.15.117 port 5666: No route to hostconnect to host 10.0.15.117 port 5666: No route to host [16:48:29] PROBLEM - test151 Redis Process on test151 is CRITICAL: connect to address 10.0.15.118 port 5666: No route to hostconnect to host 10.0.15.118 port 5666: No route to host [16:48:30] PROBLEM - Host test151 is DOWN: CRITICAL - Host Unreachable (10.0.15.118) [16:48:33] PROBLEM - mw151 Disk Space on mw151 is CRITICAL: connect to address 10.0.15.114 port 5666: No route to hostconnect to host 10.0.15.114 port 5666: No route to host [16:48:33] PROBLEM - mw152 php-fpm on mw152 is CRITICAL: connect to address 10.0.15.115 port 5666: No route to hostconnect to host 10.0.15.115 port 5666: No route to host [16:48:35] PROBLEM - db151 APT on db151 is CRITICAL: connect to address 10.0.15.110 port 5666: No route to hostconnect to host 10.0.15.110 port 5666: No route to host [16:48:35] PROBLEM - db151 conntrack_table_size on db151 is CRITICAL: connect to address 10.0.15.110 port 5666: No route to hostconnect to host 10.0.15.110 port 5666: No route to host [16:48:38] PROBLEM - db151 PowerDNS Recursor on db151 is CRITICAL: connect to address 10.0.15.110 port 5666: No route to hostconnect to host 10.0.15.110 port 5666: No route to host [16:48:39] PROBLEM - matomo151 APT on matomo151 is CRITICAL: connect to address 10.0.15.112 port 5666: No route to hostconnect to host 10.0.15.112 port 5666: No route to host [16:48:39] PROBLEM - matomo151 PowerDNS Recursor on matomo151 is CRITICAL: connect to address 10.0.15.112 port 5666: No route to hostconnect to host 10.0.15.112 port 5666: No route to host [16:48:40] PROBLEM - mem151 conntrack_table_size on mem151 is CRITICAL: connect to address 10.0.15.113 port 5666: No route to hostconnect to host 10.0.15.113 port 5666: No route to host [16:48:40] PROBLEM - mw151 MediaWiki Rendering on mw151 is CRITICAL: connect to address 10.0.15.114 and port 443: No route to hostHTTP CRITICAL - Unable to open TCP socket [16:48:41] PROBLEM - ping on db151 is CRITICAL: CRITICAL - Host Unreachable (10.0.15.110) [16:48:42] PROBLEM - mw152 NTP time on mw152 is CRITICAL: connect to address 10.0.15.115 port 5666: No route to hostconnect to host 10.0.15.115 port 5666: No route to host [16:48:42] PROBLEM - mw152 Current Load on mw152 is CRITICAL: connect to address 10.0.15.115 port 5666: No route to hostconnect to host 10.0.15.115 port 5666: No route to host [16:48:43] PROBLEM - ping on mem151 is CRITICAL: CRITICAL - Host Unreachable (10.0.15.113) [16:48:43] PROBLEM - mw151 conntrack_table_size on mw151 is CRITICAL: connect to address 10.0.15.114 port 5666: No route to hostconnect to host 10.0.15.114 port 5666: No route to host [16:48:43] PROBLEM - db151 MariaDB Connections on db151 is UNKNOWN: PHP Fatal error: Uncaught mysqli_sql_exception: Connection timed out in /usr/lib/nagios/plugins/check_mysql_connections.php:66Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(66): mysqli_real_connect(Object(mysqli), 'db151.wikitide....', 'icinga', Object(SensitiveParameterValue), NULL, NULL, NULL, true)#1 {main} thrown in /usr/lib/nagios/plugins/check_mysql_connection [16:48:43] on line 66Fatal error: Uncaught mysqli_sql_exception: Connection timed out in /usr/lib/nagios/plugins/check_mysql_connections.php:66Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(66): mysqli_real_connect(Object(mysqli), 'db151.wikitide....', 'icinga', Object(SensitiveParameterValue), NULL, NULL, NULL, true)#1 {main} thrown in /usr/lib/nagios/plugins/check_mysql_connections.php on line 66 [16:48:45] PROBLEM - mw152 PowerDNS Recursor on mw152 is CRITICAL: connect to address 10.0.15.115 port 5666: No route to hostconnect to host 10.0.15.115 port 5666: No route to host [16:48:45] PROBLEM - Host db151 is DOWN: CRITICAL - Host Unreachable (10.0.15.110) [16:48:46] PROBLEM - ping on matomo151 is CRITICAL: CRITICAL - Host Unreachable (10.0.15.112) [16:48:46] PROBLEM - Host mw151 is DOWN: CRITICAL - Host Unreachable (10.0.15.114) [16:48:48] PROBLEM - cp25 Varnish Backends on cp25 is CRITICAL: 2 backends are down. mw151 mw152 [16:48:50] PROBLEM - cp41 Varnish Backends on cp41 is CRITICAL: 2 backends are down. mw151 mw152 [16:48:58] PROBLEM - swiftobject151 NTP time on swiftobject151 is CRITICAL: connect to address 10.0.15.117 port 5666: No route to hostconnect to host 10.0.15.117 port 5666: No route to host [16:49:02] PROBLEM - matomo151 Puppet on matomo151 is CRITICAL: connect to address 10.0.15.112 port 5666: No route to hostconnect to host 10.0.15.112 port 5666: No route to host [16:49:02] PROBLEM - Host mw152 is DOWN: CRITICAL - Host Unreachable (10.0.15.115) [16:49:05] PROBLEM - Host swiftobject151 is DOWN: CRITICAL - Host Unreachable (10.0.15.117) [16:49:14] PROBLEM - Host matomo151 is DOWN: CRITICAL - Host Unreachable (10.0.15.112) [16:49:21] PROBLEM - cp34 Varnish Backends on cp34 is CRITICAL: 2 backends are down. mw151 mw152 [16:49:26] PROBLEM - mem151 Disk Space on mem151 is CRITICAL: connect to address 10.0.15.113 port 5666: No route to hostconnect to host 10.0.15.113 port 5666: No route to host [16:49:34] PROBLEM - Host mem151 is DOWN: CRITICAL - Host Unreachable (10.0.15.113) [16:49:53] RECOVERY - Host cloud17 is UP: PING OK - Packet loss = 0%, RTA = 0.17 ms [16:50:04] RECOVERY - cp26 Current Load on cp26 is OK: LOAD OK - total load average: 4.24, 5.81, 5.89 [16:50:16] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.51, 6.73, 6.73 [16:50:21] RECOVERY - Host cloud16 is UP: PING OK - Packet loss = 0%, RTA = 0.17 ms [16:50:30] RECOVERY - ping6 on cloud17 is OK: PING OK - Packet loss = 0%, RTA = 0.20 ms [16:50:35] RECOVERY - cloud17 SSH on cloud17 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u2 (protocol 2.0) [16:51:05] RECOVERY - ping6 on cloud16 is OK: PING OK - Packet loss = 0%, RTA = 0.15 ms [16:51:21] RECOVERY - cloud11 IPMI Sensors on cloud11 is OK: IPMI Status: OK [16:52:20] RECOVERY - cloud16 SSH on cloud16 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u2 (protocol 2.0) [16:53:09] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.22, 6.82, 7.87 [16:54:16] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 5.64, 6.68, 6.77 [16:54:54] PROBLEM - mw131 HTTPS on mw131 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [16:55:13] PROBLEM - cp34 HTTPS on cp34 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [16:55:17] PROBLEM - cloud11 IPMI Sensors on cloud11 is CRITICAL: IPMI Status: Critical [Cntlr 2 Bay 8 = Critical] [16:55:21] PROBLEM - cp51 HTTPS on cp51 is CRITICAL: HTTP CRITICAL: HTTP/2 502 - 3390 bytes in 1.251 second response time [16:55:29] PROBLEM - mw132 HTTPS on mw132 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [16:55:30] PROBLEM - cp25 HTTPS on cp25 is CRITICAL: HTTP CRITICAL: HTTP/2 502 - 3370 bytes in 0.459 second response time [16:55:37] PROBLEM - mw141 HTTPS on mw141 is CRITICAL: HTTP CRITICAL: HTTP/2 502 - 170 bytes in 0.373 second response time [16:55:38] PROBLEM - cp26 HTTPS on cp26 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10000 milliseconds with 0 bytes received [16:56:52] PROBLEM - mw131 HTTPS on mw131 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.524 second response time [16:56:54] PROBLEM - mw143 Current Load on mw143 is WARNING: LOAD WARNING - total load average: 10.08, 10.21, 9.85 [16:57:07] PROBLEM - cp34 HTTPS on cp34 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.592 second response time [16:57:07] PROBLEM - cloud18 PowerDNS Recursor on cloud18 is CRITICAL: connect to address 2602:294:0:b12::100 port 5666: Connection refusedconnect to host 2602:294:0:b12::100 port 5666: Connection refused [16:57:18] PROBLEM - cloud18 NTP time on cloud18 is CRITICAL: connect to address 2602:294:0:b12::100 port 5666: Connection refusedconnect to host 2602:294:0:b12::100 port 5666: Connection refused [16:57:19] PROBLEM - cp51 HTTPS on cp51 is WARNING: HTTP WARNING: HTTP/2 404 - 3479 bytes in 1.414 second response time [16:57:26] PROBLEM - cp25 HTTPS on cp25 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.939 second response time [16:57:28] PROBLEM - mw132 HTTPS on mw132 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.580 second response time [16:57:30] PROBLEM - cloud18 Puppet on cloud18 is CRITICAL: connect to address 2602:294:0:b12::100 port 5666: Connection refusedconnect to host 2602:294:0:b12::100 port 5666: Connection refused [16:57:32] PROBLEM - mw141 HTTPS on mw141 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.501 second response time [16:57:34] PROBLEM - cloud18 Current Load on cloud18 is CRITICAL: connect to address 2602:294:0:b12::100 port 5666: Connection refusedconnect to host 2602:294:0:b12::100 port 5666: Connection refused [16:57:34] PROBLEM - cp26 HTTPS on cp26 is WARNING: HTTP WARNING: HTTP/2 404 - 3480 bytes in 0.691 second response time [16:57:52] PROBLEM - cloud18 APT on cloud18 is CRITICAL: connect to address 2602:294:0:b12::100 port 5666: Connection refusedconnect to host 2602:294:0:b12::100 port 5666: Connection refused [16:58:01] PROBLEM - cloud18 Disk Space on cloud18 is CRITICAL: connect to address 2602:294:0:b12::100 port 5666: Connection refusedconnect to host 2602:294:0:b12::100 port 5666: Connection refused [16:58:05] PROBLEM - cloud18 conntrack_table_size on cloud18 is CRITICAL: connect to address 2602:294:0:b12::100 port 5666: Connection refusedconnect to host 2602:294:0:b12::100 port 5666: Connection refused [16:58:27] PROBLEM - cloud18 ferm_active on cloud18 is CRITICAL: connect to address 2602:294:0:b12::100 port 5666: Connection refusedconnect to host 2602:294:0:b12::100 port 5666: Connection refused [16:58:52] RECOVERY - mw143 Current Load on mw143 is OK: LOAD OK - total load average: 7.72, 9.22, 9.52 [19:49:41] [02mediawiki-repos] 07redbluegreenhat opened pull request 03#5: T10615: Install SemanticScribunto - 13https://github.com/miraheze/mediawiki-repos/pull/5 [19:51:01] [02mediawiki-repos] 07Universal-Omega closed pull request 03#5: T10615: Install SemanticScribunto - 13https://github.com/miraheze/mediawiki-repos/pull/5 [19:51:03] [02miraheze/mediawiki-repos] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mediawiki-repos/compare/b6510207f1b1...ce2016f2c770 [19:51:06] [02miraheze/mediawiki-repos] 07redbluegreenhat 03ce2016f - T10615: Install SemanticScribunto (#5) [19:52:06] [02mediawiki-repos] 07redbluegreenhat commented on pull request 03#5: T10615: Install SemanticScribunto - 13https://github.com/miraheze/mediawiki-repos/pull/5#issuecomment-1908815426 [19:52:43] [02mediawiki-repos] 07redbluegreenhat commented on pull request 03#5: T10615: Install SemanticScribunto - 13https://github.com/miraheze/mediawiki-repos/pull/5#issuecomment-1908816275 [19:54:53] RECOVERY - mem151 PowerDNS Recursor on mem151 is OK: DNS OK: 0.123 seconds response time. miraheze.org returns 2607:5300:205:200::1c93,2607:5300:205:200::3121,51.222.12.133,51.222.14.30 [19:54:54] RECOVERY - mw152 php-fpm on mw152 is OK: PROCS OK: 31 processes with command name 'php-fpm8.2' [19:54:54] RECOVERY - db151 NTP time on db151 is OK: NTP OK: Offset 0.002074509859 secs [19:54:54] RECOVERY - os151 Current Load on os151 is OK: LOAD OK - total load average: 0.55, 0.30, 0.11 [19:54:58] RECOVERY - ping on test151 is OK: PING OK - Packet loss = 0%, RTA = 0.71 ms [19:55:02] RECOVERY - swiftobject151 APT on swiftobject151 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [19:55:03] RECOVERY - test151 php-fpm on test151 is OK: PROCS OK: 13 processes with command name 'php-fpm8.2' [19:55:04] PROBLEM - mwtask181 HTTPS on mwtask181 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.042 second response time [19:55:05] RECOVERY - swiftobject151 Current Load on swiftobject151 is OK: LOAD OK - total load average: 0.36, 0.17, 0.06 [19:55:06] RECOVERY - mw151 Disk Space on mw151 is OK: DISK OK - free space: / 39390MiB (73% inode=89%); [19:55:07] PROBLEM - mw151 Puppet on mw151 is WARNING: WARNING: Puppet last ran 3 hours ago [19:55:10] RECOVERY - mw152 PowerDNS Recursor on mw152 is OK: DNS OK: 0.114 seconds response time. miraheze.org returns 2607:5300:205:200::1c93,2607:5300:205:200::3121,51.222.12.133,51.222.14.30 [19:55:12] RECOVERY - matomo151 PowerDNS Recursor on matomo151 is OK: DNS OK: 0.029 seconds response time. miraheze.org returns 2607:5300:205:200::1c93,2607:5300:205:200::3121,51.222.12.133,51.222.14.30 [19:55:12] RECOVERY - matomo151 Puppet on matomo151 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [19:55:12] RECOVERY - ping on matomo151 is OK: PING OK - Packet loss = 0%, RTA = 0.22 ms [19:55:12] RECOVERY - swiftobject151 Puppet on swiftobject151 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [19:55:18] RECOVERY - mw151 php-fpm on mw151 is OK: PROCS OK: 31 processes with command name 'php-fpm8.2' [19:55:18] RECOVERY - cp26 Varnish Backends on cp26 is OK: All 19 backends are healthy [19:55:20] RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 0.30, 0.14, 0.05 [19:55:20] RECOVERY - test151 ferm_active on test151 is OK: OK ferm input default policy is set [19:55:20] RECOVERY - cp51 Varnish Backends on cp51 is OK: All 19 backends are healthy [19:55:22] RECOVERY - mem151 Current Load on mem151 is OK: LOAD OK - total load average: 0.17, 0.15, 0.06 [19:55:22] RECOVERY - db151 APT on db151 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [19:55:23] RECOVERY - ping on mw152 is OK: PING OK - Packet loss = 0%, RTA = 0.24 ms [19:55:24] PROBLEM - mw151 HTTPS on mw151 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.073 second response time [19:55:25] RECOVERY - mw151 PowerDNS Recursor on mw151 is OK: DNS OK: 0.117 seconds response time. miraheze.org returns 2607:5300:205:200::1c93,2607:5300:205:200::3121,51.222.12.133,51.222.14.30 [19:55:27] RECOVERY - matomo151 SSH on matomo151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u2 (protocol 2.0) [19:55:30] RECOVERY - swiftobject151 conntrack_table_size on swiftobject151 is OK: OK: nf_conntrack is 0 % full [19:55:31] RECOVERY - swiftobject151 Swift Object Service on swiftobject151 is OK: TCP OK - 0.000 second response time on 10.0.15.117 port 6000 [19:55:32] RECOVERY - db151 MariaDB on db151 is OK: Uptime: 144 Threads: 1 Questions: 51 Slow queries: 0 Opens: 19 Open tables: 13 Queries per second avg: 0.354 [19:55:32] RECOVERY - mem151 Disk Space on mem151 is OK: DISK OK - free space: / 5886MiB (66% inode=86%); [19:55:33] PROBLEM - mw133 Current Load on mw133 is CRITICAL: LOAD CRITICAL - total load average: 13.67, 11.41, 10.35 [19:55:36] PROBLEM - cp35 Puppet on cp35 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:55:37] PROBLEM - cloud13 Puppet on cloud13 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:55:37] RECOVERY - mem151 conntrack_table_size on mem151 is OK: OK: nf_conntrack is 0 % full [19:55:37] RECOVERY - db151 PowerDNS Recursor on db151 is OK: DNS OK: 0.034 seconds response time. miraheze.org returns 2607:5300:205:200::1c93,2607:5300:205:200::3121,51.222.12.133,51.222.14.30 [19:55:37] PROBLEM - cp25 Puppet on cp25 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:55:37] PROBLEM - cp24 Puppet on cp24 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:55:37] PROBLEM - reports121 Puppet on reports121 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:55:37] PROBLEM - mw142 Puppet on mw142 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:55:38] PROBLEM - mw134 Puppet on mw134 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:55:38] PROBLEM - cp41 Puppet on cp41 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:55:39] PROBLEM - bast121 Puppet on bast121 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:55:39] PROBLEM - cloud12 Puppet on cloud12 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:55:40] PROBLEM - db101 Puppet on db101 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:55:40] PROBLEM - graylog131 Puppet on graylog131 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:55:52] PROBLEM - cloud10 Puppet on cloud10 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:55:52] RECOVERY - ping on mw151 is OK: PING OK - Packet loss = 0%, RTA = 0.24 ms [19:55:53] RECOVERY - mw152 conntrack_table_size on mw152 is OK: OK: nf_conntrack is 0 % full [19:55:53] RECOVERY - ping on db151 is OK: PING OK - Packet loss = 0%, RTA = 0.25 ms [19:55:54] PROBLEM - bast141 Puppet on bast141 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:55:57] PROBLEM - test151 HTTPS on test151 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.072 second response time [19:55:57] RECOVERY - prometheus151 NTP time on prometheus151 is OK: NTP OK: Offset 0.006077170372 secs [19:55:59] PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:56:00] PROBLEM - swiftac111 Puppet on swiftac111 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:56:00] RECOVERY - mem151 NTP time on mem151 is OK: NTP OK: Offset 0.004633426666 secs [19:56:02] PROBLEM - mem141 Puppet on mem141 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:56:04] PROBLEM - mail121 Puppet on mail121 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:56:04] PROBLEM - swiftobject121 Puppet on swiftobject121 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:56:05] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.andreijiroh.uk.eu.org All nameservers failed to answer the query. [19:56:06] PROBLEM - swiftobject111 Puppet on swiftobject111 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:56:07] PROBLEM - test151 MediaWiki Rendering on test151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:56:07] RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 0.31, 0.13, 0.04 [19:56:12] RECOVERY - matomo151 NTP time on matomo151 is OK: NTP OK: Offset 0.004572689533 secs [19:56:12] RECOVERY - mw152 NTP time on mw152 is OK: NTP OK: Offset 0.002489984035 secs [19:56:13] PROBLEM - cloud16 Puppet on cloud16 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:56:14] PROBLEM - mw131 Puppet on mw131 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:56:20] PROBLEM - mw132 Puppet on mw132 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:56:20] PROBLEM - cp51 Puppet on cp51 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:56:22] PROBLEM - mw152 MediaWiki Rendering on mw152 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:56:23] PROBLEM - cloud11 Puppet on cloud11 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:56:23] PROBLEM - cp26 Puppet on cp26 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:56:23] PROBLEM - mw143 Puppet on mw143 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:56:25] RECOVERY - cp35 Varnish Backends on cp35 is OK: All 19 backends are healthy [19:56:25] RECOVERY - cp34 Varnish Backends on cp34 is OK: All 19 backends are healthy [19:56:27] RECOVERY - mw152 Puppet on mw152 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:56:28] PROBLEM - swiftobject122 Puppet on swiftobject122 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:56:30] RECOVERY - cp25 Varnish Backends on cp25 is OK: All 19 backends are healthy [19:56:31] PROBLEM - swiftobject101 Puppet on swiftobject101 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:56:32] PROBLEM - db142 Puppet on db142 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:56:32] RECOVERY - prometheus151 Puppet on prometheus151 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:56:32] PROBLEM - mw132 Current Load on mw132 is CRITICAL: LOAD CRITICAL - total load average: 13.92, 11.31, 10.13 [19:56:36] [02mediawiki-repos] 07Universal-Omega commented on pull request 03#5: T10615: Install SemanticScribunto - 13https://github.com/miraheze/mediawiki-repos/pull/5#issuecomment-1908821431 [19:56:41] PROBLEM - swiftobject113 Puppet on swiftobject113 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:56:43] RECOVERY - swiftobject151 NTP time on swiftobject151 is OK: NTP OK: Offset 0.004867106676 secs [19:56:55] RECOVERY - cp24 Varnish Backends on cp24 is OK: All 19 backends are healthy [19:57:04] RECOVERY - mw151 Puppet on mw151 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:57:05] PROBLEM - mw134 Current Load on mw134 is CRITICAL: LOAD CRITICAL - total load average: 13.02, 10.90, 10.29 [19:57:28] PROBLEM - mw131 Current Load on mw131 is CRITICAL: LOAD CRITICAL - total load average: 14.59, 11.82, 10.45 [19:57:38] RECOVERY - reports121 Puppet on reports121 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [19:57:38] RECOVERY - mw151 MediaWiki Rendering on mw151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 9.331 second response time [19:57:57] [02miraheze/mediawiki-repos] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mediawiki-repos/compare/ce2016f2c770...e575465186bd [19:58:00] [02miraheze/mediawiki-repos] 07Universal-Omega 03e575465 - Fix branch [19:58:11] [Grafana] !sre RESOLVED: MediaWiki Exception Rate https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [19:58:23] RECOVERY - mw152 MediaWiki Rendering on mw152 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 6.071 second response time [19:58:32] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 10.92, 11.36, 10.31 [19:59:05] RECOVERY - mw134 Current Load on mw134 is OK: LOAD OK - total load average: 9.00, 9.95, 10.00 [19:59:37] RECOVERY - cp25 Puppet on cp25 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:59:55] RECOVERY - bast141 Puppet on bast141 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:00:15] RECOVERY - test151 MediaWiki Rendering on test151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 9.846 second response time [20:01:20] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 11.41, 11.72, 10.75 [20:01:29] RECOVERY - mw142 Puppet on mw142 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [20:01:33] RECOVERY - mw133 Puppet on mw133 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:01:34] RECOVERY - db131 Puppet on db131 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:01:42] RECOVERY - db112 Puppet on db112 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [20:02:05] RECOVERY - cloud16 Puppet on cloud16 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:02:11] RECOVERY - cp51 Puppet on cp51 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:02:58] RECOVERY - swiftobject101 Puppet on swiftobject101 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:03:20] PROBLEM - mw131 Current Load on mw131 is CRITICAL: LOAD CRITICAL - total load average: 15.66, 12.71, 11.20 [20:03:41] RECOVERY - cloud15 Puppet on cloud15 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [20:03:45] RECOVERY - ldap141 Puppet on ldap141 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [20:03:47] RECOVERY - graylog131 Puppet on graylog131 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:06:07] RECOVERY - mw143 Puppet on mw143 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [20:06:11] RECOVERY - mw132 Puppet on mw132 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [20:06:18] [02miraheze/puppet] 07AgentIsai pushed 031 commit to 03master [+0/-0/±15] 13https://github.com/miraheze/puppet/compare/ab9d225db9d2...8b6b68677e2e [20:06:21] [02miraheze/puppet] 07AgentIsai 038b6b686 - Fix motds [20:06:32] PROBLEM - mw132 Current Load on mw132 is CRITICAL: LOAD CRITICAL - total load average: 17.31, 12.96, 11.16 [20:06:35] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 11.31, 8.28, 6.89 [20:06:40] RECOVERY - swiftobject111 Puppet on swiftobject111 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:06:52] PROBLEM - mw141 Current Load on mw141 is WARNING: LOAD WARNING - total load average: 10.54, 9.08, 7.73 [20:07:06] PROBLEM - mw134 Current Load on mw134 is CRITICAL: LOAD CRITICAL - total load average: 19.96, 14.71, 11.93 [20:07:17] RECOVERY - mw134 Puppet on mw134 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:07:19] RECOVERY - cp41 Puppet on cp41 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:07:26] RECOVERY - mw141 Puppet on mw141 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:07:43] RECOVERY - os141 Puppet on os141 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [20:07:47] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.36, 6.90, 6.30 [20:07:54] RECOVERY - swiftproxy131 Puppet on swiftproxy131 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [20:08:40] PROBLEM - mw133 HTTPS on mw133 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [20:08:47] PROBLEM - cp25 HTTPS on cp25 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [20:08:51] PROBLEM - mw141 Current Load on mw141 is CRITICAL: LOAD CRITICAL - total load average: 14.99, 10.57, 8.41 [20:09:33] RECOVERY - cloud12 Puppet on cloud12 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [20:09:37] RECOVERY - phab121 Puppet on phab121 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:09:42] RECOVERY - cp24 Puppet on cp24 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:09:47] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.44, 6.68, 6.29 [20:09:50] RECOVERY - cloud10 Puppet on cloud10 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [20:09:59] [02miraheze/puppet] 07AgentIsai pushed 031 commit to 03master [+2/-0/±1] 13https://github.com/miraheze/puppet/compare/8b6b68677e2e...bda0bca715f6 [20:10:01] [02miraheze/puppet] 07AgentIsai 03bda0bca - Add bash defaults [20:10:31] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 5.83, 7.13, 6.78 [20:10:36] PROBLEM - mw133 HTTPS on mw133 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 1.571 second response time [20:10:40] RECOVERY - cloud18 Puppet on cloud18 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:10:42] PROBLEM - cp25 HTTPS on cp25 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.791 second response time [20:10:50] PROBLEM - mw141 Current Load on mw141 is WARNING: LOAD WARNING - total load average: 9.72, 10.57, 8.70 [20:11:11] RECOVERY - bast121 Puppet on bast121 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [20:11:37] RECOVERY - cp35 Puppet on cp35 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:11:58] RECOVERY - mw131 Puppet on mw131 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:12:14] RECOVERY - db142 Puppet on db142 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:12:28] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 5.43, 6.50, 6.59 [20:12:49] RECOVERY - mw141 Current Load on mw141 is OK: LOAD OK - total load average: 9.17, 9.82, 8.63 [20:13:09] RECOVERY - swiftproxy111 Puppet on swiftproxy111 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:13:22] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 143.43 ms [20:14:13] RECOVERY - mail121 Puppet on mail121 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:14:14] PROBLEM - cp35 HTTPS on cp35 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [20:14:56] RECOVERY - swiftobject113 Puppet on swiftobject113 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:15:24] RECOVERY - os131 Puppet on os131 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [20:15:24] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 144.39 ms [20:15:36] RECOVERY - mem141 Puppet on mem141 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:15:44] RECOVERY - cloud17 Puppet on cloud17 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:16:13] PROBLEM - cp35 HTTPS on cp35 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 0.622 second response time [20:16:34] RECOVERY - swiftobject112 Puppet on swiftobject112 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [20:16:43] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.71, 7.07, 6.55 [20:16:46] PROBLEM - mw141 Current Load on mw141 is WARNING: LOAD WARNING - total load average: 9.06, 10.79, 9.37 [20:17:56] RECOVERY - jobchron121 Puppet on jobchron121 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:18:19] RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [20:18:40] PROBLEM - mw141 HTTPS on mw141 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10003 milliseconds with 0 bytes received [20:18:42] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.44, 6.51, 6.41 [20:18:45] PROBLEM - mw141 Current Load on mw141 is CRITICAL: LOAD CRITICAL - total load average: 21.74, 15.24, 11.17 [20:20:04] RECOVERY - db121 Puppet on db121 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [20:20:17] RECOVERY - cloud11 Puppet on cloud11 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [20:20:19] PROBLEM - mw143 Current Load on mw143 is WARNING: LOAD WARNING - total load average: 8.74, 10.66, 9.18 [20:20:32] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 10.37, 11.79, 11.91 [20:20:37] PROBLEM - mw141 HTTPS on mw141 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.619 second response time [20:20:44] PROBLEM - mw141 Current Load on mw141 is WARNING: LOAD WARNING - total load average: 7.28, 11.73, 10.36 [20:21:35] RECOVERY - swiftobject122 Puppet on swiftobject122 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:21:43] RECOVERY - cloud13 Puppet on cloud13 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:21:48] RECOVERY - swiftobject121 Puppet on swiftobject121 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:22:18] RECOVERY - mw143 Current Load on mw143 is OK: LOAD OK - total load average: 6.64, 9.14, 8.79 [20:22:43] RECOVERY - mw141 Current Load on mw141 is OK: LOAD OK - total load average: 6.77, 10.07, 9.92 [20:22:52] RECOVERY - db101 Puppet on db101 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:23:06] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 10.21, 11.25, 11.82 [20:23:17] RECOVERY - cloud14 Puppet on cloud14 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:23:35] RECOVERY - cp26 Puppet on cp26 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:23:46] RECOVERY - swiftac111 Puppet on swiftac111 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [20:24:55] RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp35.miraheze.org - CNAME OK [20:27:37] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 11.42, 8.20, 7.02 [20:28:28] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.89, 6.51, 6.37 [20:28:46] PROBLEM - cp34 HTTPS on cp34 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [20:28:47] PROBLEM - mw142 HTTPS on mw142 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [20:29:36] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.50, 7.15, 6.79 [20:30:32] PROBLEM - mw132 Current Load on mw132 is CRITICAL: LOAD CRITICAL - total load average: 18.14, 12.35, 11.62 [20:30:43] PROBLEM - mw142 HTTPS on mw142 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.723 second response time [20:30:47] PROBLEM - cp34 HTTPS on cp34 is WARNING: HTTP WARNING: HTTP/2 404 - 3458 bytes in 6.754 second response time [20:31:07] PROBLEM - mw134 Current Load on mw134 is CRITICAL: LOAD CRITICAL - total load average: 20.79, 14.01, 12.36 [20:32:28] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.06, 6.85, 6.49 [20:33:34] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.52, 6.66, 6.72 [20:34:28] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 5.13, 6.05, 6.23 [20:34:32] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 8.63, 10.83, 11.25 [20:34:33] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 9.35, 10.92, 11.94 [20:37:07] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 8.34, 11.33, 11.78 [20:37:49] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/bda0bca715f6...27a4c8290740 [20:37:52] [02miraheze/puppet] 07Universal-Omega 0327a4c82 - varnish: add cloud16 and cloud17 to ACLs [20:39:09] [02miraheze/dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/c0702bf3f294...c87feb383c53 [20:39:18] [02miraheze/dns] 07Universal-Omega 03c87feb3 - Remove ns1-private and mail1-private [20:43:20] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 9.38, 10.62, 11.81 [20:44:32] RECOVERY - mw132 Current Load on mw132 is OK: LOAD OK - total load average: 8.06, 9.18, 10.14 [20:51:10] PROBLEM - mw134 Current Load on mw134 is CRITICAL: LOAD CRITICAL - total load average: 12.13, 10.51, 10.86 [20:53:10] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 11.78, 11.13, 11.06 [20:55:10] PROBLEM - mw134 Current Load on mw134 is CRITICAL: LOAD CRITICAL - total load average: 13.03, 11.64, 11.24 [20:55:20] PROBLEM - mw131 Current Load on mw131 is CRITICAL: LOAD CRITICAL - total load average: 14.04, 11.72, 11.54 [20:55:55] PROBLEM - cloud16 NTP time on cloud16 is WARNING: NTP WARNING: Offset -0.1376124322 secs [20:56:08] PROBLEM - test151 MediaWiki Rendering on test151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:56:33] PROBLEM - mw133 Current Load on mw133 is CRITICAL: LOAD CRITICAL - total load average: 14.49, 11.62, 11.03 [20:56:47] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 6.89, 5.91, 4.04 [20:57:20] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 10.65, 11.46, 11.48 [20:57:49] RECOVERY - cloud16 NTP time on cloud16 is OK: NTP OK: Offset -0.0004131495953 secs [20:58:32] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 11.42, 11.43, 10.39 [20:58:33] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 11.44, 11.41, 11.03 [20:58:47] RECOVERY - db121 Current Load on db121 is OK: OK - load average: 5.10, 5.82, 4.25 [20:59:10] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 9.25, 11.38, 11.33 [21:00:18] RECOVERY - test151 MediaWiki Rendering on test151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 9.821 second response time [21:00:47] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/c87feb383c53...a2f7b4bf0265 [21:00:48] [02miraheze/dns] 07paladox 03a2f7b4b - Add vms to cloud16 dns [21:04:32] RECOVERY - mw132 Current Load on mw132 is OK: LOAD OK - total load average: 9.56, 10.08, 10.15 [21:05:10] PROBLEM - mw134 Current Load on mw134 is CRITICAL: LOAD CRITICAL - total load average: 12.56, 11.32, 11.34 [21:05:20] PROBLEM - mw131 Current Load on mw131 is CRITICAL: LOAD CRITICAL - total load average: 12.25, 11.08, 11.26 [21:07:10] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 10.93, 11.16, 11.29 [21:07:20] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 11.56, 11.42, 11.37 [21:07:21] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.87, 6.93, 6.54 [21:08:32] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 8.91, 10.43, 10.37 [21:09:18] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 5.94, 6.43, 6.40 [21:11:10] PROBLEM - mw134 Current Load on mw134 is CRITICAL: LOAD CRITICAL - total load average: 12.49, 11.58, 11.37 [21:11:16] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.20, 7.37, 6.73 [21:12:32] RECOVERY - mw132 Current Load on mw132 is OK: LOAD OK - total load average: 9.83, 9.97, 10.18 [21:13:10] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 8.76, 10.46, 10.98 [21:13:15] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.63, 6.77, 6.59 [21:14:33] RECOVERY - mw133 Current Load on mw133 is OK: LOAD OK - total load average: 8.23, 9.05, 10.05 [21:17:10] PROBLEM - mw134 Current Load on mw134 is CRITICAL: LOAD CRITICAL - total load average: 14.17, 11.46, 11.19 [21:19:10] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 6.43, 9.54, 10.52 [21:19:20] RECOVERY - mw131 Current Load on mw131 is OK: LOAD OK - total load average: 7.26, 9.02, 10.12 [21:21:10] RECOVERY - mw134 Current Load on mw134 is OK: LOAD OK - total load average: 5.94, 8.60, 10.08 [21:27:20] PROBLEM - mw131 Current Load on mw131 is CRITICAL: LOAD CRITICAL - total load average: 15.78, 11.78, 10.69 [21:27:24] PROBLEM - mw133 Current Load on mw133 is CRITICAL: LOAD CRITICAL - total load average: 13.28, 11.72, 10.77 [21:29:20] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 10.41, 10.79, 10.44 [21:31:18] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 10.72, 11.33, 10.82 [21:35:11] RECOVERY - mw133 Current Load on mw133 is OK: LOAD OK - total load average: 6.22, 8.88, 9.93 [21:37:20] RECOVERY - mw131 Current Load on mw131 is OK: LOAD OK - total load average: 8.89, 9.76, 10.06 [21:44:11] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 11.80, 10.05, 9.68 [21:44:28] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.23, 6.91, 6.60 [21:45:00] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.38, 7.47, 6.86 [21:46:11] PROBLEM - mw134 Current Load on mw134 is CRITICAL: LOAD CRITICAL - total load average: 16.96, 11.84, 10.33 [21:46:28] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 6.41, 6.79, 6.59 [21:46:57] PROBLEM - mw133 Current Load on mw133 is CRITICAL: LOAD CRITICAL - total load average: 12.32, 11.52, 10.38 [21:48:11] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 9.94, 10.78, 10.12 [21:48:54] RECOVERY - mw133 Current Load on mw133 is OK: LOAD OK - total load average: 7.13, 10.04, 9.98 [21:50:11] RECOVERY - mw134 Current Load on mw134 is OK: LOAD OK - total load average: 7.37, 9.29, 9.64 [21:52:57] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.43, 6.35, 6.62 [21:56:28] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.10, 6.73, 6.41 [21:57:20] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 11.16, 10.40, 9.96 [21:58:28] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 6.21, 6.43, 6.34 [21:59:37] PROBLEM - mw181 HTTPS on mw181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 7 - Failed to connect to mw181.wikitide.net port 443 after 3061 ms: Couldn't connect to server [21:59:41] PROBLEM - mwtask181 HTTPS on mwtask181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 7 - Failed to connect to mwtask181.wikitide.net port 443 after 2676 ms: Couldn't connect to server [21:59:50] PROBLEM - test151 HTTPS on test151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 7 - Failed to connect to test151.wikitide.net port 443 after 1165 ms: Couldn't connect to server [22:00:20] PROBLEM - mw152 HTTPS on mw152 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 7 - Failed to connect to mw152.wikitide.net port 443 after 1241 ms: Couldn't connect to server [22:00:21] PROBLEM - mw182 HTTPS on mw182 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 7 - Failed to connect to mw182.wikitide.net port 443 after 2183 ms: Couldn't connect to server [22:00:21] PROBLEM - mw151 HTTPS on mw151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 7 - Failed to connect to mw151.wikitide.net port 443 after 2232 ms: Couldn't connect to server [22:00:55] PROBLEM - ping on mw151 is CRITICAL: CRITICAL - Host Unreachable (10.0.15.114) [22:00:56] PROBLEM - db181 SSH on db181 is CRITICAL: connect to address 10.0.18.102 and port 22: No route to host [22:01:01] PROBLEM - ping on db181 is CRITICAL: CRITICAL - Host Unreachable (10.0.18.102) [22:01:02] PROBLEM - ping on db182 is CRITICAL: CRITICAL - Host Unreachable (10.0.18.103) [22:01:02] PROBLEM - ping on bast181 is CRITICAL: CRITICAL - Host Unreachable (10.0.18.101) [22:01:07] PROBLEM - db151 Current Load on db151 is CRITICAL: connect to address 10.0.15.110 port 5666: No route to hostconnect to host 10.0.15.110 port 5666: No route to host [22:01:08] PROBLEM - mw151 php-fpm on mw151 is CRITICAL: connect to address 10.0.15.114 port 5666: No route to hostconnect to host 10.0.15.114 port 5666: No route to host [22:01:09] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: connect to address 10.0.18.106 port 5666: No route to hostconnect to host 10.0.18.106 port 5666: No route to host [22:01:09] PROBLEM - mwtask181 PowerDNS Recursor on mwtask181 is CRITICAL: connect to address 10.0.18.106 port 5666: No route to hostconnect to host 10.0.18.106 port 5666: No route to host [22:01:11] PROBLEM - mw151 Disk Space on mw151 is CRITICAL: connect to address 10.0.15.114 port 5666: No route to hostconnect to host 10.0.15.114 port 5666: No route to host [22:01:12] PROBLEM - mwtask181 SSH on mwtask181 is CRITICAL: connect to address 10.0.18.106 and port 22: No route to host [22:10:29] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.14, 7.04, 6.49 [22:10:36] PROBLEM - bast181 NTP time on bast181 is CRITICAL: NTP CRITICAL: Offset 0.7745099962 secs [22:10:36] PROBLEM - os141 Puppet on os141 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [22:10:41] RECOVERY - bast181 SSH on bast181 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u2 (protocol 2.0) [22:10:42] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 0.68, 0.23, 0.08 [22:10:42] RECOVERY - puppet181 PowerDNS Recursor on puppet181 is OK: DNS OK: 0.126 seconds response time. miraheze.org returns 2607:5300:205:200::1c93,2607:5300:205:200::3121,51.222.12.133,51.222.14.30 [22:10:43] RECOVERY - mw182 php-fpm on mw182 is OK: PROCS OK: 31 processes with command name 'php-fpm8.2' [22:10:45] RECOVERY - bast181 Disk Space on bast181 is OK: DISK OK - free space: / 5837MiB (66% inode=86%); [22:10:45] RECOVERY - bast181 PowerDNS Recursor on bast181 is OK: DNS OK: 0.121 seconds response time. miraheze.org returns 2607:5300:205:200::1c93,2607:5300:205:200::3121,51.222.12.133,51.222.14.30 [22:10:46] PROBLEM - cp51 Varnish Backends on cp51 is CRITICAL: 1 backends are down. mw182 [22:10:47] RECOVERY - mwtask181 JobRunner Service on mwtask181 is OK: PROCS OK: 1 process with args 'redisJobRunnerService' [22:10:47] RECOVERY - Host db181 is UP: PING OK - Packet loss = 0%, RTA = 0.19 ms [22:10:48] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.64, 6.88, 6.59 [22:10:48] RECOVERY - bast181 Puppet on bast181 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [22:10:50] RECOVERY - mwtask181 Disk Space on mwtask181 is OK: DISK OK - free space: / 197454MiB (89% inode=95%); [22:10:52] RECOVERY - ping on db181 is OK: PING OK - Packet loss = 0%, RTA = 0.16 ms [22:10:52] RECOVERY - mwtask181 SSH on mwtask181 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u2 (protocol 2.0) [22:10:54] RECOVERY - mw182 SSH on mw182 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u2 (protocol 2.0) [22:10:57] RECOVERY - Host swiftproxy181 is UP: PING OK - Packet loss = 0%, RTA = 0.20 ms [22:11:00] RECOVERY - mwtask181 php-fpm on mwtask181 is OK: PROCS OK: 13 processes with command name 'php-fpm8.2' [22:11:00] PROBLEM - mw182 HTTPS on mw182 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.156 second response time [22:11:00] RECOVERY - mw182 conntrack_table_size on mw182 is OK: OK: nf_conntrack is 0 % full [22:11:01] RECOVERY - mwtask181 ferm_active on mwtask181 is OK: OK ferm input default policy is set [22:11:02] RECOVERY - matomo151 HTTPS on matomo151 is OK: HTTP OK: HTTP/2 200 - 552 bytes in 0.123 second response time [22:11:02] PROBLEM - cp24 Puppet on cp24 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [22:11:03] RECOVERY - Host db182 is UP: PING OK - Packet loss = 0%, RTA = 0.18 ms [22:11:04] RECOVERY - Host swiftobject181 is UP: PING OK - Packet loss = 0%, RTA = 0.19 ms [22:11:08] RECOVERY - mwtask181 NTP time on mwtask181 is OK: NTP OK: Offset 0.003966599703 secs [22:11:09] RECOVERY - swiftobject181 Disk Space on swiftobject181 is OK: DISK OK - free space: / 1332280MiB (99% inode=99%); [22:11:10] RECOVERY - mw182 NTP time on mw182 is OK: NTP OK: Offset -0.01828172803 secs [22:11:11] RECOVERY - ping on mwtask181 is OK: PING OK - Packet loss = 0%, RTA = 0.15 ms [22:11:13] RECOVERY - mw182 Puppet on mw182 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [22:11:14] RECOVERY - Host mw181 is UP: PING OK - Packet loss = 0%, RTA = 0.18 ms [22:11:16] PROBLEM - mw132 Puppet on mw132 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [22:11:16] RECOVERY - ping on db182 is OK: PING OK - Packet loss = 0%, RTA = 0.17 ms [22:11:17] RECOVERY - mwtask181 PowerDNS Recursor on mwtask181 is OK: DNS OK: 0.110 seconds response time. miraheze.org returns 2607:5300:205:200::1c93,2607:5300:205:200::3121,51.222.12.133,51.222.14.30 [22:11:17] RECOVERY - ping on bast181 is OK: PING OK - Packet loss = 0%, RTA = 0.23 ms [22:11:17] RECOVERY - db182 conntrack_table_size on db182 is OK: OK: nf_conntrack is 0 % full [22:11:19] RECOVERY - mw182 ferm_active on mw182 is OK: OK ferm input default policy is set [22:11:20] RECOVERY - mw182 Disk Space on mw182 is OK: DISK OK - free space: / 39343MiB (73% inode=89%); [22:11:21] PROBLEM - mw143 Puppet on mw143 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [22:11:30] PROBLEM - swiftobject111 Puppet on swiftobject111 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [22:11:32] PROBLEM - cloud15 Puppet on cloud15 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [22:11:33] PROBLEM - swiftproxy131 Puppet on swiftproxy131 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [22:11:35] PROBLEM - mw134 Puppet on mw134 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [22:11:37] PROBLEM - mw141 Puppet on mw141 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [22:11:38] RECOVERY - db181 SSH on db181 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u2 (protocol 2.0) [22:11:39] RECOVERY - test151 MediaWiki Rendering on test151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 9.802 second response time [22:11:40] PROBLEM - cp41 Puppet on cp41 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [22:11:47] PROBLEM - mw181 MediaWiki Rendering on mw181 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:11:54] RECOVERY - cp24 Varnish Backends on cp24 is OK: All 19 backends are healthy [22:12:23] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 6.35, 6.80, 6.47 [22:12:37] RECOVERY - db182 PowerDNS Recursor on db182 is OK: DNS OK: 0.035 seconds response time. miraheze.org returns 2607:5300:205:200::1c93,2607:5300:205:200::3121,51.222.12.133,51.222.14.30 [22:12:39] RECOVERY - bast181 NTP time on bast181 is OK: NTP OK: Offset 0.008202731609 secs [22:12:44] RECOVERY - cp51 Varnish Backends on cp51 is OK: All 19 backends are healthy [22:12:47] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.42, 6.76, 6.59 [22:12:47] RECOVERY - swiftobject181 SSH on swiftobject181 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u2 (protocol 2.0) [22:12:48] RECOVERY - db182 Backups SQL icingaweb2 on db182 is OK: FILE_AGE OK: /var/log/sql-icingaweb2-backup-fortnightly.log is 839554 seconds old and 125 bytes [22:12:52] RECOVERY - mw181 SSH on mw181 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u2 (protocol 2.0) [22:12:53] RECOVERY - db182 Backups SQL roundcubemail on db182 is OK: FILE_AGE OK: /var/log/sql-roundcubemail-backup-fortnightly.log is 839556 seconds old and 219 bytes [22:12:53] RECOVERY - swiftobject181 Puppet on swiftobject181 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [22:12:56] PROBLEM - mwtask181 HTTPS on mwtask181 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.097 second response time [22:12:57] RECOVERY - swiftobject181 APT on swiftobject181 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [22:12:58] RECOVERY - ping on swiftobject181 is OK: PING OK - Packet loss = 0%, RTA = 0.34 ms [22:13:02] RECOVERY - mw181 Puppet on mw181 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [22:13:03] PROBLEM - mw181 HTTPS on mw181 is WARNING: HTTP WARNING: HTTP/2 404 - 285 bytes in 0.053 second response time [22:13:03] RECOVERY - mw181 conntrack_table_size on mw181 is OK: OK: nf_conntrack is 0 % full [22:13:11] RECOVERY - mwtask181 Puppet on mwtask181 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [22:13:33] RECOVERY - mw182 MediaWiki Rendering on mw182 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 5.954 second response time [22:13:47] RECOVERY - mw181 MediaWiki Rendering on mw181 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 5.945 second response time [22:14:45] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 10.44, 10.20, 10.01 [22:15:43] RECOVERY - mwtask181 MediaWiki Rendering on mwtask181 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 5.581 second response time [22:16:41] RECOVERY - mw133 Current Load on mw133 is OK: LOAD OK - total load average: 9.07, 9.96, 9.95 [22:20:22] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 11.30, 10.55, 9.86 [22:20:54] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/27a4c8290740...752376fe490a [22:20:56] [02miraheze/puppet] 07paladox 03752376f - base::puppet: support using bookworm dust [22:22:17] RECOVERY - mw134 Current Load on mw134 is OK: LOAD OK - total load average: 9.20, 9.87, 9.69 [22:22:38] PROBLEM - mw133 Current Load on mw133 is CRITICAL: LOAD CRITICAL - total load average: 13.97, 11.79, 10.70 [22:24:22] PROBLEM - puppet181 Puppet on puppet181 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): Exec[apt_update_puppetlabs],Exec[apt_update] [22:24:22] PROBLEM - mw132 Current Load on mw132 is CRITICAL: LOAD CRITICAL - total load average: 12.24, 10.81, 10.01 [22:24:38] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 11.43, 11.72, 10.81 [22:24:41] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/752376fe490a...e21d33f2f1d3 [22:24:44] [02miraheze/puppet] 07paladox 03e21d33f - Revert "base::puppet: support using bookworm dust" [22:25:14] PROBLEM - cp26 Puppet on cp26 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 3 minutes ago with 2 failures. Failed resources (up to 3 shown): Exec[apt_update_puppetlabs],Exec[apt_update] [22:25:53] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03remove-mail [+1/-0/±0] 13https://github.com/miraheze/puppet/compare/6b37593936f9...7c9dca745be6 [22:25:55] [02miraheze/puppet] 07Universal-Omega 037c9dca7 - Temp [22:25:56] [02puppet] 07Universal-Omega synchronize pull request 03#3669: Remove mail121, and roundcubemail, dovecot, and postfix modules - 13https://github.com/miraheze/puppet/pull/3669 [22:26:04] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 11.75, 10.88, 10.15 [22:26:07] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 10.02, 10.52, 10.04 [22:26:16] RECOVERY - puppet181 Puppet on puppet181 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [22:26:22] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 10.28, 10.59, 10.04 [22:26:38] PROBLEM - mw181 Puppet on mw181 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): Exec[apt_update_puppetlabs],Exec[apt_update] [22:26:38] PROBLEM - mw133 Current Load on mw133 is CRITICAL: LOAD CRITICAL - total load average: 12.47, 11.79, 10.94 [22:26:43] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03remove-mail [+1/-0/±0] 13https://github.com/miraheze/puppet/compare/7c9dca745be6...6723b7bb4ed3 [22:26:43] PROBLEM - mw152 Puppet on mw152 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 3 minutes ago with 2 failures. Failed resources (up to 3 shown): Exec[apt_update_puppetlabs],Exec[apt_update] [22:26:45] [02miraheze/puppet] 07Universal-Omega 036723b7b - temp [22:26:48] [02puppet] 07Universal-Omega synchronize pull request 03#3669: Remove mail121, and roundcubemail, dovecot, and postfix modules - 13https://github.com/miraheze/puppet/pull/3669 [22:27:36] [02miraheze/puppet] 07Universal-Omega pushed 0351 commits to 03remove-mail [+12/-0/±135] 13https://github.com/miraheze/puppet/compare/6723b7bb4ed3...4e0da2cf0cda [22:27:37] [02miraheze/puppet] 07Universal-Omega 034e0da2c - Merge branch 'master' into remove-mail [22:27:39] [02puppet] 07Universal-Omega synchronize pull request 03#3669: Remove mail121, and roundcubemail, dovecot, and postfix modules - 13https://github.com/miraheze/puppet/pull/3669 [22:27:45] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): Exec[apt_update_puppetlabs],Exec[apt_update] [22:27:57] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03remove-mail [+0/-1/±0] 13https://github.com/miraheze/puppet/compare/4e0da2cf0cda...a5fdf04bcaef [22:28:00] [02miraheze/puppet] 07Universal-Omega 03a5fdf04 - - [22:28:03] [02puppet] 07Universal-Omega synchronize pull request 03#3669: Remove mail121, and roundcubemail, dovecot, and postfix modules - 13https://github.com/miraheze/puppet/pull/3669 [22:28:10] [02miraheze/puppet] 07github-actions[bot] pushed 032 commits to 03remove-mail [+1/-0/±1] 13https://github.com/miraheze/puppet/compare/a5fdf04bcaef...4ca58c1d9d64 [22:28:12] [02miraheze/puppet] 07Universal-Omega 03797f3f9 - Merge 4e0da2cf0cdad6c5b29cd07dac85c7960f194095 into e21d33f2f1d3a2534701c762fcdeb7006bd8dc23 [22:28:13] [02miraheze/puppet] 07github-actions 034ca58c1 - CI: lint puppet code to standards [22:28:14] [02puppet] 07github-actions[bot] synchronize pull request 03#3669: Remove mail121, and roundcubemail, dovecot, and postfix modules - 13https://github.com/miraheze/puppet/pull/3669 [22:28:38] [02puppet] 07Universal-Omega synchronize pull request 03#3669: Remove mail121, and roundcubemail, dovecot, and postfix modules - 13https://github.com/miraheze/puppet/pull/3669 [22:28:41] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03remove-mail [+0/-1/±0] 13https://github.com/miraheze/puppet/compare/4ca58c1d9d64...1fcbf730f3ef [22:28:43] [02miraheze/puppet] 07Universal-Omega 031fcbf73 - - [22:28:49] PROBLEM - swiftobject181 Puppet on swiftobject181 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [22:28:56] PROBLEM - mw182 Puppet on mw182 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [22:29:48] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03remove-mail [+0/-1/±0] 13https://github.com/miraheze/puppet/compare/1fcbf730f3ef...8b5f00e8925a [22:29:51] [02miraheze/puppet] 07Universal-Omega 038b5f00e - - [22:29:54] [02puppet] 07Universal-Omega synchronize pull request 03#3669: Remove mail121, and roundcubemail, dovecot, and postfix modules - 13https://github.com/miraheze/puppet/pull/3669 [22:29:56] RECOVERY - mw134 Current Load on mw134 is OK: LOAD OK - total load average: 7.67, 9.44, 9.74 [22:30:21] RECOVERY - mw132 Current Load on mw132 is OK: LOAD OK - total load average: 8.91, 9.83, 9.88 [22:31:49] RECOVERY - mw131 Current Load on mw131 is OK: LOAD OK - total load average: 8.55, 9.97, 10.07 [22:32:38] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 7.77, 10.32, 10.66 [22:32:41] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.88, 7.55, 6.91 [22:33:33] RECOVERY - os151 Puppet on os151 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [22:33:41] RECOVERY - ldap141 Puppet on ldap141 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [22:33:57] RECOVERY - graylog131 Puppet on graylog131 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [22:34:00] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.73, 6.94, 6.57 [22:34:39] RECOVERY - mem151 Puppet on mem151 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [22:34:41] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.82, 6.85, 6.73 [22:34:50] RECOVERY - cloud15 Puppet on cloud15 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [22:35:03] RECOVERY - swiftobject111 Puppet on swiftobject111 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [22:35:17] RECOVERY - mw151 Puppet on mw151 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [22:35:38] RECOVERY - cp41 Puppet on cp41 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [22:35:57] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 5.25, 6.30, 6.38 [22:36:46] RECOVERY - mw132 Puppet on mw132 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [22:36:56] RECOVERY - mw143 Puppet on mw143 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [22:37:09] RECOVERY - mw141 Puppet on mw141 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [22:37:15] RECOVERY - mw134 Puppet on mw134 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [22:38:02] RECOVERY - swiftproxy131 Puppet on swiftproxy131 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [22:38:35] RECOVERY - cp24 Puppet on cp24 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [22:38:38] RECOVERY - mw133 Current Load on mw133 is OK: LOAD OK - total load average: 8.85, 9.43, 10.19 [22:38:41] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.42, 6.38, 6.58 [22:39:43] RECOVERY - os141 Puppet on os141 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [22:43:49] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 10.75, 7.39, 6.64 [22:43:49] PROBLEM - mw131 Current Load on mw131 is CRITICAL: LOAD CRITICAL - total load average: 12.10, 10.54, 10.07 [22:45:32] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 10.99, 10.52, 10.04 [22:45:46] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.59, 7.30, 6.70 [22:46:41] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.61, 7.16, 6.77 [22:47:49] RECOVERY - mw131 Current Load on mw131 is OK: LOAD OK - total load average: 7.82, 9.98, 10.04 [22:48:23] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 10.30, 9.34, 9.16 [22:50:21] RECOVERY - mw132 Current Load on mw132 is OK: LOAD OK - total load average: 9.49, 9.23, 9.14 [22:52:38] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 10.80, 10.43, 10.20 [22:52:51] RECOVERY - cp26 Puppet on cp26 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [22:54:09] RECOVERY - mw181 Puppet on mw181 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [22:54:41] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.49, 7.78, 7.21 [22:55:05] RECOVERY - mw134 Current Load on mw134 is OK: LOAD OK - total load average: 7.84, 9.50, 9.90 [22:55:33] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 5.06, 6.34, 6.60 [22:56:10] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [22:56:41] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.30, 7.50, 7.17 [22:58:31] RECOVERY - mw182 Puppet on mw182 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [22:58:38] PROBLEM - mw133 Current Load on mw133 is CRITICAL: LOAD CRITICAL - total load average: 13.13, 11.09, 10.48 [22:58:48] RECOVERY - swiftobject181 Puppet on swiftobject181 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [22:58:55] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/056df779dc06...58d3a1f781f0 [22:58:56] [02miraheze/ssl] 07MirahezeSSLBot 0358d3a1f - Bot: Add SSL cert for wiki.mhe.my.id [22:59:19] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/58d3a1f781f0...8c1f5405859b [22:59:22] [02miraheze/ssl] 07MirahezeSSLBot 038c1f540 - Bot: Add SSL cert for wiki.gamergeeked.us [23:00:22] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 11.55, 10.62, 9.72 [23:00:39] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 9.82, 10.44, 10.18 [23:01:55] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 9.97, 10.26, 10.18 [23:02:21] RECOVERY - mw132 Current Load on mw132 is OK: LOAD OK - total load average: 9.49, 10.10, 9.64 [23:02:34] RECOVERY - mw131 Current Load on mw131 is OK: LOAD OK - total load average: 9.79, 10.02, 10.05 [23:02:38] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 9.92, 11.80, 11.07 [23:02:41] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.91, 6.39, 6.79 [23:03:13] PROBLEM - cp26 Current Load on cp26 is WARNING: LOAD WARNING - total load average: 7.85, 5.24, 3.56 [23:03:55] RECOVERY - mw134 Current Load on mw134 is OK: LOAD OK - total load average: 8.08, 9.59, 9.96 [23:05:13] RECOVERY - cp26 Current Load on cp26 is OK: LOAD OK - total load average: 6.53, 5.43, 3.82 [23:05:20] [02dns] 07MacFan4000 opened pull request 03#465: add ff8.wiki zone and add www.satepedia.com - 13https://github.com/miraheze/dns/pull/465 [23:08:18] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 11.52, 10.15, 10.00 [23:09:55] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 11.39, 10.64, 10.23 [23:10:16] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.83, 2.89, 1.86 [23:10:34] PROBLEM - mw141 Current Load on mw141 is WARNING: LOAD WARNING - total load average: 8.47, 10.55, 8.28 [23:10:38] RECOVERY - mw133 Current Load on mw133 is OK: LOAD OK - total load average: 8.09, 8.97, 9.99 [23:11:12] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.97, 6.53, 6.30 [23:12:08] PROBLEM - mw131 Current Load on mw131 is CRITICAL: LOAD CRITICAL - total load average: 12.84, 10.53, 10.11 [23:12:14] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.20, 3.00, 2.00 [23:12:33] RECOVERY - mw141 Current Load on mw141 is OK: LOAD OK - total load average: 8.88, 9.52, 8.15 [23:13:09] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 6.18, 6.47, 6.30 [23:13:55] RECOVERY - mw134 Current Load on mw134 is OK: LOAD OK - total load average: 8.11, 9.65, 9.96 [23:14:03] RECOVERY - mw131 Current Load on mw131 is OK: LOAD OK - total load average: 9.12, 9.91, 9.93 [23:16:09] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 2.54, 3.18, 2.34 [23:16:50] [02dns] 07Universal-Omega closed pull request 03#465: add ff8.wiki zone and add www.satepedia.com - 13https://github.com/miraheze/dns/pull/465 [23:16:53] [02miraheze/dns] 07Universal-Omega pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/dns/compare/a2f7b4bf0265...42d0eb50de86 [23:16:54] [02miraheze/dns] 07MacFan4000 0342d0eb5 - add ff8.wiki zone and add www.satepedia.com (#465) [23:17:00] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/aad1af1ab0e0 [23:17:03] [02miraheze/puppet] 07paladox 03aad1af1 - Install jobchron171 [23:17:05] [02puppet] 07paladox created branch 03paladox-patch-12 - 13https://github.com/miraheze/puppet [23:17:07] [02puppet] 07paladox opened pull request 03#3679: Install jobchron171 - 13https://github.com/miraheze/puppet/pull/3679 [23:17:39] [02miraheze/puppet] 07github-actions[bot] pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/aad1af1ab0e0...d742d28bfb42 [23:17:40] [02miraheze/puppet] 07github-actions 03d742d28 - CI: lint puppet code to standards [23:17:43] [02puppet] 07github-actions[bot] synchronize pull request 03#3679: Install jobchron171 - 13https://github.com/miraheze/puppet/pull/3679 [23:17:59] PROBLEM - db171 MariaDB Connections on db171 is UNKNOWN: PHP Fatal error: Uncaught mysqli_sql_exception: Connection timed out in /usr/lib/nagios/plugins/check_mysql_connections.php:66Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(66): mysqli_real_connect(Object(mysqli), 'db171.wikitide....', 'icinga', Object(SensitiveParameterValue), NULL, NULL, NULL, true)#1 {main} thrown in /usr/lib/nagios/plugins/check_mysql_connection [23:17:59] on line 66Fatal error: Uncaught mysqli_sql_exception: Connection timed out in /usr/lib/nagios/plugins/check_mysql_connections.php:66Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(66): mysqli_real_connect(Object(mysqli), 'db171.wikitide....', 'icinga', Object(SensitiveParameterValue), NULL, NULL, NULL, true)#1 {main} thrown in /usr/lib/nagios/plugins/check_mysql_connections.php on line 66 [23:18:03] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+1/-0/±0] 13https://github.com/miraheze/puppet/compare/d742d28bfb42...b5a3f6e7369d [23:18:06] [02miraheze/puppet] 07paladox 03b5a3f6e - Create jobchron171.yaml [23:18:08] [02puppet] 07paladox synchronize pull request 03#3679: Install jobchron171 - 13https://github.com/miraheze/puppet/pull/3679 [23:18:40] PROBLEM - db171 Puppet on db171 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [23:19:04] PROBLEM - db171 Backups SQL on db171 is CRITICAL: FILE_AGE CRITICAL: File not found - /var/log/sql-backup.log [23:19:05] PROBLEM - db171 ferm_active on db171 is UNKNOWN: NRPE: Unable to read output [23:19:13] PROBLEM - db171 MariaDB on db171 is CRITICAL: Can't connect to server on 'db171.wikitide.net' (115) [23:19:17] PROBLEM - db171 Backups SQL mhglobal on db171 is CRITICAL: FILE_AGE CRITICAL: File not found - /var/log/sql-mhglobal-backup-weekly.log [23:20:01] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [23:20:25] [02miraheze/mw-config] 07MacFan4000 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/8cd58587e8b1...f0df7989f939 [23:20:26] [02miraheze/mw-config] 07MacFan4000 03f0df798 - add migration sitenotice [23:20:40] RECOVERY - db171 Puppet on db171 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:21:02] RECOVERY - db171 ferm_active on db171 is OK: OK ferm input default policy is set [23:21:17] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/e21d33f2f1d3...782e206c9bb8 [23:21:18] [02miraheze/puppet] 07paladox 03782e206 - db171: disable ssl [23:21:25] miraheze/mw-config - MacFan4000 the build passed. [23:22:20] !log [macfan@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [23:22:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:23:07] [02puppet] 07paladox closed pull request 03#3679: Install jobchron171 - 13https://github.com/miraheze/puppet/pull/3679 [23:23:09] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+1/-0/±2] 13https://github.com/miraheze/puppet/compare/782e206c9bb8...a15775f8a3dc [23:23:12] [02miraheze/puppet] 07paladox 03a15775f - Install jobchron171 (#3679) [23:23:14] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-12 [23:23:29] !log [macfan@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 68s [23:23:35] [02puppet] 07paladox deleted branch 03paladox-patch-12 - 13https://github.com/miraheze/puppet [23:23:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:24:26] RECOVERY - mw152 Puppet on mw152 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [23:25:58] PROBLEM - cp26 Disk Space on cp26 is WARNING: DISK WARNING - free space: / 10231MiB (10% inode=98%); [23:26:28] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/a15775f8a3dc...f3de74ca6a45 [23:26:31] [02miraheze/puppet] 07paladox 03f3de74c - Fix jobchron171 redis ip [23:27:26] @agentisai I think we gave to much cache to cp26 that it is filling up completely, the max should be less then total available by a bit probably [23:30:43] CosmicAlpha: do we want to go for two servers? [23:31:24] Yes please [23:31:31] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/f3de74ca6a45...fb7e80e2532e [23:31:33] [02miraheze/puppet] 07paladox 03fb7e80e - squid: add cloud16/17 ipv6 ranges [23:32:03] PROBLEM - mw133 Current Load on mw133 is CRITICAL: LOAD CRITICAL - total load average: 13.59, 11.57, 10.04 [23:33:22] PROBLEM - mw134 Current Load on mw134 is CRITICAL: LOAD CRITICAL - total load average: 15.92, 11.88, 10.20 [23:33:22] PROBLEM - mw131 Current Load on mw131 is CRITICAL: LOAD CRITICAL - total load average: 12.19, 10.31, 9.46 [23:36:03] PROBLEM - mw132 Current Load on mw132 is CRITICAL: LOAD CRITICAL - total load average: 13.59, 11.73, 10.08 [23:36:51] PROBLEM - mw141 Current Load on mw141 is WARNING: LOAD WARNING - total load average: 11.35, 10.01, 8.71 [23:38:25] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/e1fecc8a548c [23:38:27] [02miraheze/puppet] 07paladox 03e1fecc8 - install ldap171 [23:38:29] [02puppet] 07paladox created branch 03paladox-patch-12 - 13https://github.com/miraheze/puppet [23:38:30] [02puppet] 07paladox opened pull request 03#3680: install ldap171 - 13https://github.com/miraheze/puppet/pull/3680 [23:38:50] RECOVERY - mw141 Current Load on mw141 is OK: LOAD OK - total load average: 6.33, 8.54, 8.33 [23:39:10] PROBLEM - mw131 Current Load on mw131 is WARNING: LOAD WARNING - total load average: 10.62, 11.34, 10.35 [23:40:03] PROBLEM - mw132 Current Load on mw132 is WARNING: LOAD WARNING - total load average: 11.57, 11.97, 10.59 [23:41:06] PROBLEM - mw134 Current Load on mw134 is WARNING: LOAD WARNING - total load average: 10.06, 11.65, 10.95 [23:41:27] [02puppet] 07paladox closed pull request 03#3680: install ldap171 - 13https://github.com/miraheze/puppet/pull/3680 [23:41:28] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/fb7e80e2532e...1de7b53a1bd9 [23:41:30] [02miraheze/puppet] 07paladox 031de7b53 - install ldap171 (#3680) [23:41:31] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-12 [23:41:34] [02puppet] 07paladox deleted branch 03paladox-patch-12 - 13https://github.com/miraheze/puppet [23:43:03] RECOVERY - mw131 Current Load on mw131 is OK: LOAD OK - total load average: 7.81, 9.97, 10.05 [23:45:47] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 9.51, 11.70, 11.43 [23:48:34] PROBLEM - ldap171 LDAP on ldap171 is CRITICAL: Could not bind to the LDAP server [23:48:35] PROBLEM - ldap171 Puppet on ldap171 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Openldap_overlay[ppolicy on cn=config] [23:51:15] PROBLEM - mw134 Current Load on mw134 is CRITICAL: LOAD CRITICAL - total load average: 12.18, 10.87, 10.76 [23:51:33] PROBLEM - mw133 Current Load on mw133 is CRITICAL: LOAD CRITICAL - total load average: 12.92, 12.19, 11.67 [23:51:39] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/1de7b53a1bd9...c22694e05740 [23:51:40] [02miraheze/puppet] 07paladox 03c22694e - mwdeploy: add mw171/172 [23:52:03] RECOVERY - mw132 Current Load on mw132 is OK: LOAD OK - total load average: 9.82, 10.04, 10.19 [23:52:58] miraheze/puppet - paladox the build has errored. [23:53:28] PROBLEM - mw133 Current Load on mw133 is WARNING: LOAD WARNING - total load average: 11.31, 11.67, 11.53 [23:54:11] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/c22694e05740...727d80286f03 [23:54:13] [02miraheze/puppet] 07paladox 03727d802 - varnish: add mw171/172 but don't pool [23:55:02] PROBLEM - mw171 MediaWiki Rendering on mw171 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:55:03] PROBLEM - mw171 HTTPS on mw171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Connection timed out after 10002 milliseconds [23:55:22] PROBLEM - mw172 HTTPS on mw172 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Connection timed out after 10004 milliseconds [23:55:24] PROBLEM - mw172 MediaWiki Rendering on mw172 is CRITICAL: connect to address 10.0.17.123 and port 443: Connection refusedHTTP CRITICAL - Unable to open TCP socket [23:55:26] PROBLEM - mw133 Current Load on mw133 is CRITICAL: LOAD CRITICAL - total load average: 20.06, 15.11, 12.82 [23:55:32] PROBLEM - mw132 Current Load on mw132 is CRITICAL: LOAD CRITICAL - total load average: 16.40, 13.55, 11.55 [23:55:34] PROBLEM - mw172 Puppet on mw172 is UNKNOWN: NRPE: Unable to read output [23:55:57] PROBLEM - mw171 APT on mw171 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [23:55:58] PROBLEM - mw171 php-fpm on mw171 is CRITICAL: PROCS CRITICAL: 0 processes with command name 'php-fpm8.2' [23:55:59] PROBLEM - mw171 conntrack_table_size on mw171 is UNKNOWN: NRPE: Unable to read output [23:55:59] PROBLEM - mw131 Current Load on mw131 is CRITICAL: LOAD CRITICAL - total load average: 17.20, 13.93, 11.46 [23:56:00] PROBLEM - mw172 php-fpm on mw172 is CRITICAL: PROCS CRITICAL: 0 processes with command name 'php-fpm8.2' [23:56:01] PROBLEM - mw171 Puppet on mw171 is UNKNOWN: UNKNOWN: Failed to check. Reason is: no_summary_file [23:56:04] PROBLEM - mw172 ferm_active on mw172 is UNKNOWN: NRPE: Unable to read output [23:56:12] PROBLEM - mw171 ferm_active on mw171 is UNKNOWN: NRPE: Unable to read output [23:56:13] PROBLEM - mw172 conntrack_table_size on mw172 is UNKNOWN: NRPE: Unable to read output [23:57:26] PROBLEM - cp35 Varnish Backends on cp35 is CRITICAL: 2 backends are down. mw171 mw172 [23:57:52] PROBLEM - cp41 Varnish Backends on cp41 is CRITICAL: 2 backends are down. mw171 mw172 [23:57:53] RECOVERY - mw171 conntrack_table_size on mw171 is OK: OK: nf_conntrack is 0 % full [23:57:56] RECOVERY - mw171 APT on mw171 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [23:58:08] PROBLEM - cp34 Varnish Backends on cp34 is CRITICAL: 2 backends are down. mw171 mw172 [23:58:09] RECOVERY - mw172 conntrack_table_size on mw172 is OK: OK: nf_conntrack is 0 % full [23:58:18] PROBLEM - cp26 Varnish Backends on cp26 is CRITICAL: 2 backends are down. mw171 mw172 [23:58:27] PROBLEM - cp51 Varnish Backends on cp51 is CRITICAL: 2 backends are down. mw171 mw172 [23:58:28] PROBLEM - cp24 Varnish Backends on cp24 is CRITICAL: 2 backends are down. mw171 mw172 [23:58:43] PROBLEM - cp25 Varnish Backends on cp25 is CRITICAL: 2 backends are down. mw171 mw172