[00:03:07] PROBLEM - db112 Current Load on db112 is CRITICAL: CRITICAL - load average: 12.09, 6.81, 3.21 [00:10:57] PROBLEM - matomo121 Current Load on matomo121 is WARNING: LOAD WARNING - total load average: 3.64, 3.39, 2.44 [00:12:57] RECOVERY - matomo121 Current Load on matomo121 is OK: LOAD OK - total load average: 2.96, 3.29, 2.53 [00:13:07] RECOVERY - db112 Current Load on db112 is OK: OK - load average: 2.23, 6.32, 5.52 [00:15:36] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address www.twistwoodtaleswiki.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [00:16:57] PROBLEM - matomo121 Current Load on matomo121 is WARNING: LOAD WARNING - total load average: 3.13, 3.45, 2.79 [00:18:57] RECOVERY - matomo121 Current Load on matomo121 is OK: LOAD OK - total load average: 3.04, 3.32, 2.82 [00:19:33] RECOVERY - acgn.sfdev.eu.org - LetsEncrypt on sslhost is OK: OK - Certificate 'acgn.sfdev.eu.org' will expire on Sun 03 Mar 2024 22:21:38 GMT +0000. [00:20:51] PROBLEM - os131 Current Load on os131 is CRITICAL: LOAD CRITICAL - total load average: 5.50, 3.66, 2.29 [00:22:25] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 6.60, 3.97, 2.98 [00:24:28] PROBLEM - mw132 Current Load on mw132 is CRITICAL: CRITICAL - load average: 17.39, 12.63, 9.23 [00:24:48] PROBLEM - os131 Current Load on os131 is WARNING: LOAD WARNING - total load average: 3.92, 3.93, 2.72 [00:25:47] PROBLEM - matomo121 Current Load on matomo121 is WARNING: LOAD WARNING - total load average: 3.90, 3.54, 3.07 [00:26:17] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 1.59, 3.23, 2.96 [00:26:27] PROBLEM - mw132 Current Load on mw132 is WARNING: WARNING - load average: 8.77, 10.96, 9.04 [00:27:44] RECOVERY - matomo121 Current Load on matomo121 is OK: LOAD OK - total load average: 3.11, 3.22, 3.00 [00:28:27] RECOVERY - mw132 Current Load on mw132 is OK: OK - load average: 6.88, 9.60, 8.78 [00:28:48] RECOVERY - os131 Current Load on os131 is OK: LOAD OK - total load average: 2.37, 3.14, 2.68 [00:42:48] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.86, 6.28, 6.12 [00:44:43] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 4.17, 5.39, 5.80 [00:45:36] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.twistwoodtaleswiki.com' expires in 15 day(s) (Wed 20 Dec 2023 22:08:34 GMT +0000). [00:52:20] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.36, 7.03, 6.29 [00:54:14] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.31, 6.80, 6.30 [01:15:36] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address www.twistwoodtaleswiki.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [01:44:53] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/9231dfbd381e...917061eff912 [01:44:55] [02miraheze/mw-config] 07paladox 03917061e - Remove readTimeout [01:45:52] miraheze/mw-config - paladox the build passed. [01:45:54] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/917061eff912...050dabdd4704 [01:45:56] [02miraheze/mw-config] 07paladox 03050dabd - Revert "Remove readTimeout" [01:46:52] miraheze/mw-config - paladox the build passed. [01:51:36] RECOVERY - test131 Puppet on test131 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [02:20:19] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.38, 7.23, 6.54 [02:22:14] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.60, 6.87, 6.49 [02:26:02] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.71, 6.70, 6.49 [02:45:36] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.twistwoodtaleswiki.com' expires in 15 day(s) (Wed 20 Dec 2023 22:08:34 GMT +0000). [03:00:07] RECOVERY - db121 Backups SQL on db121 is OK: FILE_AGE OK: /var/log/sql-backup.log is 6 seconds old and 0 bytes [03:03:45] PROBLEM - mon141 Backups Grafana on mon141 is WARNING: FILE_AGE WARNING: /var/log/grafana-backup.log is 864203 seconds old and 93 bytes [03:15:36] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address www.twistwoodtaleswiki.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [03:59:23] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [04:44:47] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-15 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/2da532db649c [04:44:49] [02miraheze/puppet] 07paladox 032da532d - mariadb: use ssl-ca not ssl-capath [04:44:52] [02puppet] 07paladox created branch 03paladox-patch-15 - 13https://github.com/miraheze/puppet [04:44:55] [02puppet] 07paladox opened pull request 03#3546: mariadb: use ssl-ca not ssl-capath - 13https://github.com/miraheze/puppet/pull/3546 [04:45:04] [02puppet] 07paladox closed pull request 03#3546: mariadb: use ssl-ca not ssl-capath - 13https://github.com/miraheze/puppet/pull/3546 [04:45:05] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/7c710e657910...662e4cf68153 [04:45:08] [02miraheze/puppet] 07paladox 03662e4cf - mariadb: use ssl-ca not ssl-capath (#3546) [04:45:10] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-15 [04:45:11] [02puppet] 07paladox deleted branch 03paladox-patch-15 - 13https://github.com/miraheze/puppet [04:45:36] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.twistwoodtaleswiki.com' expires in 15 day(s) (Wed 20 Dec 2023 22:08:34 GMT +0000). [04:46:06] [02miraheze/puppet] 07paladox pushed 035 commits to 03revert-2853-Universal-Omega-patch-1 [+0/-0/±6] 13https://github.com/miraheze/puppet/compare/e19b63a96145...e3200c57dd67 [04:46:07] [02miraheze/puppet] 07paladox 036f4f527 - Revert "Completely remove blackbox from MediaWiki (#2853)" [04:46:09] [02miraheze/puppet] 07paladox 03f8fb595 - Update blackbox_exporter.yml.erb [04:46:12] [02miraheze/puppet] 07paladox 038a74303 - Update prometheus.pp [04:46:13] [02miraheze/puppet] ... and 2 more commits. [04:46:14] [02puppet] 07paladox synchronize pull request 03#3544: Revert "Completely remove blackbox from MediaWiki" - 13https://github.com/miraheze/puppet/pull/3544 [04:50:26] PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [04:50:52] PROBLEM - db112 Puppet on db112 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [04:52:07] PROBLEM - puppet141 Puppet on puppet141 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [04:52:21] PROBLEM - swiftobject122 Puppet on swiftobject122 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [04:52:23] PROBLEM - swiftobject121 Puppet on swiftobject121 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [04:52:31] PROBLEM - db121 Puppet on db121 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [04:52:35] PROBLEM - cloud11 Puppet on cloud11 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [04:53:02] PROBLEM - db101 Puppet on db101 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [04:53:03] PROBLEM - cloud13 Puppet on cloud13 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [04:53:59] PROBLEM - mon141 Puppet on mon141 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [04:54:23] PROBLEM - cloud14 Puppet on cloud14 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [04:55:17] PROBLEM - swiftac111 Puppet on swiftac111 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [04:55:27] PROBLEM - prometheus131 Puppet on prometheus131 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [04:56:40] PROBLEM - mem131 Puppet on mem131 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [04:57:35] PROBLEM - test131 Puppet on test131 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [04:58:04] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [04:58:53] PROBLEM - reports121 Puppet on reports121 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [04:59:07] PROBLEM - cp34 Puppet on cp34 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [04:59:24] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query pt.eu.org. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [04:59:58] PROBLEM - cp25 Puppet on cp25 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [05:01:23] PROBLEM - bast141 Puppet on bast141 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [05:02:07] PROBLEM - db131 Puppet on db131 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [05:02:35] PROBLEM - mw133 Puppet on mw133 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [05:02:41] PROBLEM - mw142 Puppet on mw142 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [05:03:22] PROBLEM - swiftobject101 Puppet on swiftobject101 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [05:05:38] PROBLEM - graylog131 Puppet on graylog131 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [05:06:26] PROBLEM - ldap141 Puppet on ldap141 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [05:06:28] PROBLEM - mw143 Puppet on mw143 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [05:07:26] PROBLEM - matomo121 Puppet on matomo121 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [05:07:33] PROBLEM - mw134 Puppet on mw134 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [05:07:47] PROBLEM - mw141 Puppet on mw141 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [05:08:14] PROBLEM - mw132 Puppet on mw132 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [05:08:26] PROBLEM - swiftobject111 Puppet on swiftobject111 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [05:08:38] PROBLEM - os141 Puppet on os141 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [05:09:10] PROBLEM - swiftproxy131 Puppet on swiftproxy131 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [05:09:51] PROBLEM - puppetdb121 Puppet on puppetdb121 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [05:11:50] PROBLEM - phab121 Puppet on phab121 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [05:11:53] PROBLEM - cp24 Puppet on cp24 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [05:11:54] PROBLEM - mwtask141 Puppet on mwtask141 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [05:12:23] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-15 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/cc3e48af724d [05:12:25] [02miraheze/puppet] 07paladox 03cc3e48a - puppetserver: fix logging [05:12:26] [02puppet] 07paladox created branch 03paladox-patch-15 - 13https://github.com/miraheze/puppet [05:12:28] [02puppet] 07paladox opened pull request 03#3547: puppetserver: fix logging - 13https://github.com/miraheze/puppet/pull/3547 [05:12:44] PROBLEM - mw131 Puppet on mw131 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [05:12:52] PROBLEM - cloud12 Puppet on cloud12 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [05:12:58] PROBLEM - cloud10 Puppet on cloud10 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [05:13:00] PROBLEM - cp35 Puppet on cp35 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [05:13:42] PROBLEM - bast121 Puppet on bast121 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [05:13:52] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-15 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/cc3e48af724d...8858c035590a [05:13:54] [02miraheze/puppet] 07paladox 038858c03 - Update puppetserver-request-logging.xml [05:13:56] [02puppet] 07paladox synchronize pull request 03#3547: puppetserver: fix logging - 13https://github.com/miraheze/puppet/pull/3547 [05:13:59] [02puppet] 07paladox closed pull request 03#3547: puppetserver: fix logging - 13https://github.com/miraheze/puppet/pull/3547 [05:14:01] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/puppet/compare/662e4cf68153...a04a5faf8caf [05:14:03] PROBLEM - db142 Puppet on db142 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [05:14:03] [02miraheze/puppet] 07paladox 03a04a5fa - puppetserver: fix logging (#3547) [05:14:05] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-15 [05:14:08] [02puppet] 07paladox deleted branch 03paladox-patch-15 - 13https://github.com/miraheze/puppet [05:14:30] [02miraheze/puppet] 07paladox pushed 035 commits to 03revert-2853-Universal-Omega-patch-1 [+0/-0/±6] 13https://github.com/miraheze/puppet/compare/e3200c57dd67...58a51d77091c [05:14:33] [02miraheze/puppet] 07paladox 03a2899ea - Revert "Completely remove blackbox from MediaWiki (#2853)" [05:14:34] [02miraheze/puppet] 07paladox 039d18014 - Update blackbox_exporter.yml.erb [05:14:36] [02miraheze/puppet] 07paladox 03f80e664 - Update prometheus.pp [05:14:38] [02miraheze/puppet] ... and 2 more commits. [05:14:40] [02puppet] 07paladox synchronize pull request 03#3544: Revert "Completely remove blackbox from MediaWiki" - 13https://github.com/miraheze/puppet/pull/3544 [05:14:58] PROBLEM - swiftproxy111 Puppet on swiftproxy111 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [05:15:07] PROBLEM - mail121 Puppet on mail121 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [05:15:38] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address www.twistwoodtaleswiki.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [05:16:26] PROBLEM - swiftobject113 Puppet on swiftobject113 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [05:17:11] PROBLEM - mem141 Puppet on mem141 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [05:17:19] PROBLEM - os131 Puppet on os131 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [05:19:09] PROBLEM - swiftobject112 Puppet on swiftobject112 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [05:19:11] PROBLEM - jobchron121 Puppet on jobchron121 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [05:22:55] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-15 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/59fb8e0fe528 [05:22:58] [02miraheze/puppet] 07paladox 0359fb8e0 - puppetserver: fix path to JsonEncoder in log config [05:23:01] [02puppet] 07paladox created branch 03paladox-patch-15 - 13https://github.com/miraheze/puppet [05:23:02] [02puppet] 07paladox opened pull request 03#3548: puppetserver: fix path to JsonEncoder in log config - 13https://github.com/miraheze/puppet/pull/3548 [05:34:35] [02puppet] 07paladox synchronize pull request 03#3548: puppetserver: fix path to JsonEncoder in log config - 13https://github.com/miraheze/puppet/pull/3548 [05:34:36] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-15 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/59fb8e0fe528...abf0e904d125 [05:34:37] [02miraheze/puppet] 07paladox 03abf0e90 - Update puppetserver_logback.xml [05:34:49] [02puppet] 07paladox edited pull request 03#3548: puppetserver: fix log part 2 - 13https://github.com/miraheze/puppet/pull/3548 [05:35:47] [02puppet] 07paladox closed pull request 03#3548: puppetserver: fix log part 2 - 13https://github.com/miraheze/puppet/pull/3548 [05:35:48] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/a04a5faf8caf...1205075e180a [05:35:51] [02miraheze/puppet] 07paladox 031205075 - puppetserver: fix log part 2 (#3548) [05:35:52] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-15 [05:35:54] [02puppet] 07paladox deleted branch 03paladox-patch-15 - 13https://github.com/miraheze/puppet [05:37:07] [02miraheze/puppet] 07paladox pushed 035 commits to 03revert-2853-Universal-Omega-patch-1 [+0/-0/±6] 13https://github.com/miraheze/puppet/compare/58a51d77091c...9b8e5c1541d5 [05:37:08] [02miraheze/puppet] 07paladox 03bb26b45 - Revert "Completely remove blackbox from MediaWiki (#2853)" [05:37:09] [02miraheze/puppet] 07paladox 036f0b81b - Update blackbox_exporter.yml.erb [05:37:10] [02miraheze/puppet] 07paladox 032fa7d53 - Update prometheus.pp [05:37:12] [02miraheze/puppet] ... and 2 more commits. [05:37:15] [02puppet] 07paladox synchronize pull request 03#3544: Revert "Completely remove blackbox from MediaWiki" - 13https://github.com/miraheze/puppet/pull/3544 [05:39:14] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/1205075e180a...918be706ef6a [05:39:17] [02miraheze/puppet] 07paladox 03918be70 - Fix [05:39:53] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/918be706ef6a...bf84bb8622a8 [05:39:54] RECOVERY - cp24 Puppet on cp24 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [05:39:55] [02miraheze/puppet] 07paladox 03bf84bb8 - Fix [05:40:08] RECOVERY - puppet141 Puppet on puppet141 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [05:40:32] [02miraheze/puppet] 07paladox pushed 035 commits to 03revert-2853-Universal-Omega-patch-1 [+0/-0/±6] 13https://github.com/miraheze/puppet/compare/9b8e5c1541d5...fa999c40bf8b [05:40:35] [02miraheze/puppet] 07paladox 03b1169f4 - Revert "Completely remove blackbox from MediaWiki (#2853)" [05:40:36] [02miraheze/puppet] 07paladox 039ee9383 - Update blackbox_exporter.yml.erb [05:40:39] [02miraheze/puppet] 07paladox 03ed5b6af - Update prometheus.pp [05:40:42] [02miraheze/puppet] ... and 2 more commits. [05:40:43] [02puppet] 07paladox synchronize pull request 03#3544: Revert "Completely remove blackbox from MediaWiki" - 13https://github.com/miraheze/puppet/pull/3544 [05:41:00] RECOVERY - cp35 Puppet on cp35 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [05:41:30] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±0] 13https://github.com/miraheze/puppet/compare/bf84bb8622a8...758913d201a8 [05:41:33] [02miraheze/puppet] 07paladox 03758913d - Fix [05:41:45] [02miraheze/puppet] 07paladox pushed 035 commits to 03revert-2853-Universal-Omega-patch-1 [+0/-0/±6] 13https://github.com/miraheze/puppet/compare/fa999c40bf8b...f5e12bdb6792 [05:41:48] [02miraheze/puppet] 07paladox 03973ab3e - Revert "Completely remove blackbox from MediaWiki (#2853)" [05:41:51] [02miraheze/puppet] 07paladox 03ea702c5 - Update blackbox_exporter.yml.erb [05:41:53] [02miraheze/puppet] 07paladox 0346541b8 - Update prometheus.pp [05:41:55] [02miraheze/puppet] ... and 2 more commits. [05:41:58] [02puppet] 07paladox synchronize pull request 03#3544: Revert "Completely remove blackbox from MediaWiki" - 13https://github.com/miraheze/puppet/pull/3544 [05:42:58] RECOVERY - swiftproxy111 Puppet on swiftproxy111 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [05:43:14] RECOVERY - mail121 Puppet on mail121 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [05:44:51] RECOVERY - db112 Puppet on db112 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [05:45:19] RECOVERY - os131 Puppet on os131 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [05:46:06] RECOVERY - swiftobject112 Puppet on swiftobject112 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [05:47:18] RECOVERY - jobchron121 Puppet on jobchron121 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [05:48:40] RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [05:50:34] !log upgrade db112 mariadb including os (still on bullseye) [05:50:35] RECOVERY - cloud11 Puppet on cloud11 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [05:50:37] RECOVERY - db121 Puppet on db121 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [05:50:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:51:02] RECOVERY - db101 Puppet on db101 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [05:51:03] RECOVERY - cloud13 Puppet on cloud13 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [05:51:12] RECOVERY - swiftobject122 Puppet on swiftobject122 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [05:51:32] RECOVERY - swiftobject121 Puppet on swiftobject121 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [05:52:26] RECOVERY - cloud14 Puppet on cloud14 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [05:53:17] RECOVERY - swiftac111 Puppet on swiftac111 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [05:53:26] RECOVERY - prometheus131 Puppet on prometheus131 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [05:53:46] !log reboot db112 [05:53:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:53:58] RECOVERY - mon141 Puppet on mon141 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [05:54:22] RECOVERY - db112 APT on db112 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [05:54:50] RECOVERY - mem131 Puppet on mem131 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [05:55:20] PROBLEM - db112 Disk Space on db112 is CRITICAL: connect to address 2a10:6740::6:211 port 5666: Connection refusedconnect to host 2a10:6740::6:211 port 5666: Connection refused [05:55:50] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [05:57:07] RECOVERY - cp34 Puppet on cp34 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [05:57:16] PROBLEM - db112 MariaDB Connections on db112 is UNKNOWN: PHP Fatal error: Uncaught mysqli_sql_exception: Connection refused in /usr/lib/nagios/plugins/check_mysql_connections.php:66Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(66): mysqli_real_connect(Object(mysqli), 'db112.miraheze....', 'icinga', Object(SensitiveParameterValue), NULL, NULL, NULL, true)#1 {main} thrown in /usr/lib/nagios/plugins/check_mysql_connections. [05:57:16] n line 66Fatal error: Uncaught mysqli_sql_exception: Connection refused in /usr/lib/nagios/plugins/check_mysql_connections.php:66Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(66): mysqli_real_connect(Object(mysqli), 'db112.miraheze....', 'icinga', Object(SensitiveParameterValue), NULL, NULL, NULL, true)#1 {main} thrown in /usr/lib/nagios/plugins/check_mysql_connections.php on line 66 [05:57:20] PROBLEM - db112 Disk Space on db112 is WARNING: DISK WARNING - free space: / 12576 MB (9% inode=99%); [05:57:29] PROBLEM - db112 MariaDB on db112 is CRITICAL: Can't connect to MySQL server on 'db112.miraheze.org' (115) [05:57:37] RECOVERY - test131 Puppet on test131 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [05:57:54] PROBLEM - matomo121 HTTPS on matomo121 is CRITICAL: HTTP CRITICAL: HTTP/2 500 - 425 bytes in 0.054 second response time [05:58:18] PROBLEM - phab121 phab.miraheze.wiki HTTPS on phab121 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/1.1 500 Internal Server Error [05:58:53] RECOVERY - reports121 Puppet on reports121 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [05:58:56] PROBLEM - phab121 phabricator.miraheze.org HTTPS on phab121 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 4216 bytes in 0.055 second response time [05:59:22] RECOVERY - bast141 Puppet on bast141 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [05:59:58] RECOVERY - cp25 Puppet on cp25 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:00:07] RECOVERY - db131 Puppet on db131 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [06:00:35] RECOVERY - mw133 Puppet on mw133 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:01:26] RECOVERY - db112 MariaDB on db112 is OK: Uptime: 8 Threads: 35 Questions: 1538 Slow queries: 0 Opens: 155 Open tables: 148 Queries per second avg: 192.250 [06:01:43] RECOVERY - matomo121 HTTPS on matomo121 is OK: HTTP OK: HTTP/2 200 - 552 bytes in 0.214 second response time [06:02:18] RECOVERY - phab121 phab.miraheze.wiki HTTPS on phab121 is OK: HTTP OK: Status line output matched "HTTP/1.1 200" - 17683 bytes in 0.077 second response time [06:02:18] RECOVERY - swiftobject101 Puppet on swiftobject101 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [06:02:41] RECOVERY - mw142 Puppet on mw142 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:02:56] RECOVERY - phab121 phabricator.miraheze.org HTTPS on phab121 is OK: HTTP OK: HTTP/1.1 200 OK - 20259 bytes in 0.159 second response time [06:03:01] RECOVERY - db112 MariaDB Connections on db112 is OK: OK connection usage: 2.8%Current connections: 14 [06:03:04] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/758913d201a8...38ba2559f1d5 [06:03:06] [02miraheze/puppet] 07paladox 0338ba255 - matomo: upgrade to 4.16.0 [06:03:20] [02miraheze/puppet] 07paladox pushed 035 commits to 03revert-2853-Universal-Omega-patch-1 [+0/-0/±6] 13https://github.com/miraheze/puppet/compare/f5e12bdb6792...a453a0b6c686 [06:03:23] [02miraheze/puppet] 07paladox 0312e60da - Revert "Completely remove blackbox from MediaWiki (#2853)" [06:03:24] [02miraheze/puppet] 07paladox 039b3e5d2 - Update blackbox_exporter.yml.erb [06:03:25] [02miraheze/puppet] 07paladox 035bbe76f - Update prometheus.pp [06:03:28] [02miraheze/puppet] ... and 2 more commits. [06:03:31] [02puppet] 07paladox synchronize pull request 03#3544: Revert "Completely remove blackbox from MediaWiki" - 13https://github.com/miraheze/puppet/pull/3544 [06:03:39] RECOVERY - graylog131 Puppet on graylog131 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:04:26] RECOVERY - ldap141 Puppet on ldap141 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:04:51] RECOVERY - mw143 Puppet on mw143 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [06:06:13] RECOVERY - swiftobject111 Puppet on swiftobject111 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:06:23] RECOVERY - mw132 Puppet on mw132 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:07:27] RECOVERY - matomo121 Puppet on matomo121 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [06:07:33] RECOVERY - mw134 Puppet on mw134 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [06:07:53] RECOVERY - mw141 Puppet on mw141 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:08:38] RECOVERY - os141 Puppet on os141 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:09:03] RECOVERY - cloud10 Puppet on cloud10 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [06:09:09] RECOVERY - swiftproxy131 Puppet on swiftproxy131 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:09:50] RECOVERY - phab121 Puppet on phab121 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [06:10:52] RECOVERY - cloud12 Puppet on cloud12 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:10:57] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/38ba2559f1d5...7570101150aa [06:10:58] [02miraheze/puppet] 07paladox 037570101 - mariadb: set innodb_log_file_size to 1G [06:11:09] [02miraheze/puppet] 07paladox pushed 035 commits to 03revert-2853-Universal-Omega-patch-1 [+0/-0/±6] 13https://github.com/miraheze/puppet/compare/a453a0b6c686...e924f53a1737 [06:11:10] [02miraheze/puppet] 07paladox 034a7ac91 - Revert "Completely remove blackbox from MediaWiki (#2853)" [06:11:13] [02miraheze/puppet] 07paladox 034615b47 - Update blackbox_exporter.yml.erb [06:11:14] [02miraheze/puppet] 07paladox 03e1ea2cd - Update prometheus.pp [06:11:17] [02miraheze/puppet] ... and 2 more commits. [06:11:18] [02puppet] 07paladox synchronize pull request 03#3544: Revert "Completely remove blackbox from MediaWiki" - 13https://github.com/miraheze/puppet/pull/3544 [06:11:42] RECOVERY - bast121 Puppet on bast121 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:12:03] RECOVERY - db142 Puppet on db142 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:12:59] RECOVERY - mw131 Puppet on mw131 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:14:40] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/7570101150aa...68a53b1e5fe0 [06:14:42] [02miraheze/puppet] 07paladox 0368a53b1 - db121: set mariadb::config::innodb_buffer_pool_size to 8g [06:15:10] RECOVERY - mem141 Puppet on mem141 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:15:36] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.twistwoodtaleswiki.com' expires in 15 day(s) (Wed 20 Dec 2023 22:08:34 GMT +0000). [06:15:42] RECOVERY - swiftobject113 Puppet on swiftobject113 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:15:54] RECOVERY - mwtask141 Puppet on mwtask141 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:16:16] !log db121: SET GLOBAL innodb_buffer_pool_size=(8 * 1024 * 1024 * 1024); [06:16:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:26:19] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/68a53b1e5fe0...72e5bd47e64d [06:26:21] [02miraheze/puppet] 07paladox 0372e5bd4 - matomo: reduce archivers to 3 [06:35:22] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/72e5bd47e64d...c9e2246c3373 [06:35:24] [02miraheze/puppet] 07paladox 03c9e2246 - mariadb: set table_open_cache to 200k [06:37:50] RECOVERY - puppetdb121 Puppet on puppetdb121 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:45:36] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address www.twistwoodtaleswiki.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [06:59:23] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [07:15:36] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.twistwoodtaleswiki.com' expires in 15 day(s) (Wed 20 Dec 2023 22:08:34 GMT +0000). [07:29:24] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['NS.ANKH.FR.eu.org.', 'NS1.eu.org.', 'NS1.ERIOMEM.NET.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [07:34:58] PROBLEM - ewgf.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'ewgf.wiki' expires in 15 day(s) (Wed 20 Dec 2023 22:59:05 GMT +0000). [07:45:36] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address www.twistwoodtaleswiki.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [08:04:58] PROBLEM - ewgf.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address ewgf.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [08:12:52] PROBLEM - cloud12 Puppet on cloud12 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [08:40:52] RECOVERY - cloud12 Puppet on cloud12 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [08:50:19] PROBLEM - ns1 NTP time on ns1 is CRITICAL: NTP CRITICAL: Offset 0.5001457632 secs [09:15:36] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.twistwoodtaleswiki.com' expires in 15 day(s) (Wed 20 Dec 2023 22:08:34 GMT +0000). [09:41:10] RECOVERY - cloud11 IPMI Sensors on cloud11 is OK: IPMI Status: OK [09:45:10] PROBLEM - cloud11 IPMI Sensors on cloud11 is CRITICAL: IPMI Status: Critical [Cntlr 2 Bay 8 = Critical] [09:45:38] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address www.twistwoodtaleswiki.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [10:15:36] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.twistwoodtaleswiki.com' expires in 15 day(s) (Wed 20 Dec 2023 22:08:34 GMT +0000). [10:26:53] PROBLEM - db142 Backups SQL on db142 is CRITICAL: FILE_AGE CRITICAL: /var/log/sql-backup.log is 1209649 seconds old and 92679 bytes [10:29:24] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [10:45:38] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address www.twistwoodtaleswiki.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [10:59:24] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['NS.ANKH.FR.eu.org.', 'NS1.eu.org.', 'NS1.ERIOMEM.NET.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [11:36:32] PROBLEM - hastursnotebook.org - LetsEncrypt on sslhost is CRITICAL: connect to address hastursnotebook.org and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [12:06:32] PROBLEM - hastursnotebook.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'hastursnotebook.org' expires in 15 day(s) (Wed 20 Dec 2023 16:28:10 GMT +0000). [12:06:42] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/1fdd825fbd1e...a2e578e70be6 [12:06:43] [02miraheze/ssl] 07MirahezeSSLBot 03a2e578e - Bot: Update SSL cert for hastursnotebook.org [12:27:03] PROBLEM - mw133 Current Load on mw133 is WARNING: WARNING - load average: 9.50, 11.77, 8.01 [12:29:03] RECOVERY - mw133 Current Load on mw133 is OK: OK - load average: 5.19, 9.20, 7.51 [12:29:23] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [12:36:32] RECOVERY - hastursnotebook.org - LetsEncrypt on sslhost is OK: OK - Certificate 'hastursnotebook.org' will expire on Mon 04 Mar 2024 11:06:34 GMT +0000. [12:45:39] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.twistwoodtaleswiki.com' expires in 15 day(s) (Wed 20 Dec 2023 22:08:34 GMT +0000). [12:59:24] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['NS.ANKH.FR.eu.org.', 'NS1.eu.org.', 'NS1.ERIOMEM.NET.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [13:15:36] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address www.twistwoodtaleswiki.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [13:34:58] PROBLEM - ewgf.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'ewgf.wiki' expires in 15 day(s) (Wed 20 Dec 2023 22:59:05 GMT +0000). [14:04:58] PROBLEM - ewgf.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address ewgf.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [14:11:25] PROBLEM - cloud10 Puppet on cloud10 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [14:15:36] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.twistwoodtaleswiki.com' expires in 15 day(s) (Wed 20 Dec 2023 22:08:34 GMT +0000). [14:20:25] PROBLEM - wiki.nmc.games - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.nmc.games' expires in 15 day(s) (Thu 21 Dec 2023 14:13:22 GMT +0000). [14:20:42] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/a2e578e70be6...74ed5667fad5 [14:20:44] [02miraheze/ssl] 07MirahezeSSLBot 0374ed566 - Bot: Update SSL cert for wiki.nmc.games [14:36:17] PROBLEM - fallofsanctuary.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'fallofsanctuary.com' expires in 15 day(s) (Thu 21 Dec 2023 14:31:43 GMT +0000). [14:36:31] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/74ed5667fad5...0ea88461e18b [14:36:34] [02miraheze/ssl] 07MirahezeSSLBot 030ea8846 - Bot: Update SSL cert for fallofsanctuary.com [14:39:27] RECOVERY - cloud10 Puppet on cloud10 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [14:45:36] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address www.twistwoodtaleswiki.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [14:49:22] PROBLEM - wiki.nmc.games - LetsEncrypt on sslhost is CRITICAL: connect to address wiki.nmc.games and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [15:06:04] RECOVERY - fallofsanctuary.com - LetsEncrypt on sslhost is OK: OK - Certificate 'fallofsanctuary.com' will expire on Mon 04 Mar 2024 13:36:23 GMT +0000. [15:15:36] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.twistwoodtaleswiki.com' expires in 15 day(s) (Wed 20 Dec 2023 22:08:34 GMT +0000). [15:29:24] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [15:45:36] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address www.twistwoodtaleswiki.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [15:48:27] RECOVERY - wiki.nmc.games - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.nmc.games' will expire on Mon 04 Mar 2024 13:20:34 GMT +0000. [15:59:13] PROBLEM - hr.petrawiki.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'hr.petrawiki.org' expires in 15 day(s) (Thu 21 Dec 2023 15:35:43 GMT +0000). [15:59:24] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['NS.ANKH.FR.eu.org.', 'NS1.eu.org.', 'NS1.ERIOMEM.NET.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [15:59:26] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/0ea88461e18b...37df60eb9d14 [15:59:29] [02miraheze/ssl] 07MirahezeSSLBot 0337df60e - Bot: Update SSL cert for hr.petrawiki.org [16:04:21] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.98, 6.88, 6.06 [16:06:20] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 5.13, 6.42, 6.00 [16:24:58] PROBLEM - files.petrawiki.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'files.petrawiki.org' expires in 15 day(s) (Thu 21 Dec 2023 15:53:39 GMT +0000). [16:25:16] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/37df60eb9d14...36bf856cf234 [16:25:17] [02miraheze/ssl] 07MirahezeSSLBot 0336bf856 - Bot: Update SSL cert for files.petrawiki.org [16:57:13] RECOVERY - hr.petrawiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'hr.petrawiki.org' will expire on Mon 04 Mar 2024 14:59:19 GMT +0000. [16:59:24] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [17:22:58] RECOVERY - files.petrawiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'files.petrawiki.org' will expire on Mon 04 Mar 2024 15:25:04 GMT +0000. [17:29:25] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns.ankh.fr.eu.org.', 'ns1.eu.org.', 'ns1.eriomem.net.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [17:55:11] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-15 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/91a2d7c05eb1 [17:55:13] [02miraheze/puppet] 07paladox 0391a2d7c - Update pool.pp [17:55:14] [02puppet] 07paladox created branch 03paladox-patch-15 - 13https://github.com/miraheze/puppet [17:55:18] [02puppet] 07paladox opened pull request 03#3549: Update pool.pp - 13https://github.com/miraheze/puppet/pull/3549 [18:04:58] PROBLEM - ewgf.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'ewgf.wiki' expires in 15 day(s) (Wed 20 Dec 2023 22:59:05 GMT +0000). [18:05:40] PROBLEM - mw143 Current Load on mw143 is CRITICAL: CRITICAL - load average: 17.02, 14.08, 8.35 [18:09:38] RECOVERY - mw143 Current Load on mw143 is OK: OK - load average: 5.41, 10.06, 8.09 [18:09:50] PROBLEM - puppetdb121 Puppet on puppetdb121 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[puppetdb] [18:17:04] PROBLEM - mw131 Current Load on mw131 is WARNING: WARNING - load average: 11.99, 9.76, 6.94 [18:17:32] PROBLEM - cp35 Varnish Backends on cp35 is CRITICAL: 1 backends are down. mw132 [18:18:44] PROBLEM - mw132 Current Load on mw132 is WARNING: WARNING - load average: 8.81, 11.34, 8.67 [18:19:02] RECOVERY - mw131 Current Load on mw131 is OK: OK - load average: 7.50, 8.78, 6.93 [18:19:30] RECOVERY - cp35 Varnish Backends on cp35 is OK: All 15 backends are healthy [18:22:39] RECOVERY - mw132 Current Load on mw132 is OK: OK - load average: 7.93, 9.56, 8.57 [18:24:20] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/050dabdd4704...405f5299a421 [18:24:23] [02miraheze/mw-config] 07paladox 03405f529 - Lower $wgMaxShellMemory to 512 [18:24:36] !log [paladox@mwtask141] starting deploy of {'pull': 'config', 'config': True} to all [18:24:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:24:50] !log [paladox@mwtask141] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 14s [18:24:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:24:57] !log [paladox@mwtask141] starting deploy of {'config': True} to all [18:25:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:25:03] !log [paladox@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 6s [18:25:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:25:27] miraheze/mw-config - paladox the build passed. [18:26:26] !log [@test131] starting deploy of {'config': True} to all [18:26:27] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [18:26:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:26:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:34:58] PROBLEM - ewgf.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address ewgf.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [18:39:50] PROBLEM - mw141 Current Load on mw141 is CRITICAL: CRITICAL - load average: 28.86, 14.75, 8.17 [18:40:07] PROBLEM - cp25 Varnish Backends on cp25 is CRITICAL: 3 backends are down. mw141 mw142 mw133 [18:40:13] PROBLEM - mw142 Current Load on mw142 is CRITICAL: CRITICAL - load average: 23.16, 14.24, 8.72 [18:40:27] PROBLEM - mw132 Current Load on mw132 is WARNING: WARNING - load average: 10.98, 10.91, 9.32 [18:40:50] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 5.30, 3.44, 2.05 [18:40:58] PROBLEM - cp35 Varnish Backends on cp35 is CRITICAL: 5 backends are down. mw131 mw132 mw142 mw134 mw143 [18:41:13] PROBLEM - cp34 Varnish Backends on cp34 is CRITICAL: 5 backends are down. mw131 mw132 mw142 mw134 mw143 [18:41:19] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 7.22, 5.25, 2.77 [18:41:38] PROBLEM - mw143 Current Load on mw143 is CRITICAL: CRITICAL - load average: 10.76, 12.49, 8.47 [18:41:46] PROBLEM - cp24 Varnish Backends on cp24 is CRITICAL: 2 backends are down. mw141 mw133 [18:42:50] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 1.84, 2.83, 2.00 [18:42:56] RECOVERY - cp35 Varnish Backends on cp35 is OK: All 15 backends are healthy [18:43:13] RECOVERY - cp34 Varnish Backends on cp34 is OK: All 15 backends are healthy [18:43:19] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 1.61, 3.71, 2.50 [18:43:38] RECOVERY - mw143 Current Load on mw143 is OK: OK - load average: 5.89, 10.02, 8.05 [18:43:46] RECOVERY - cp24 Varnish Backends on cp24 is OK: All 15 backends are healthy [18:44:01] PROBLEM - mw142 Current Load on mw142 is WARNING: WARNING - load average: 5.27, 10.41, 8.46 [18:44:05] RECOVERY - cp25 Varnish Backends on cp25 is OK: All 15 backends are healthy [18:44:28] RECOVERY - mw132 Current Load on mw132 is OK: OK - load average: 9.60, 10.00, 9.26 [18:45:36] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.twistwoodtaleswiki.com' expires in 15 day(s) (Wed 20 Dec 2023 22:08:34 GMT +0000). [18:45:49] RECOVERY - mw141 Current Load on mw141 is OK: OK - load average: 5.90, 10.06, 8.45 [18:45:55] RECOVERY - mw142 Current Load on mw142 is OK: OK - load average: 5.48, 8.78, 8.08 [18:46:04] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 7.73, 4.77, 2.72 [18:48:04] PROBLEM - db121 Current Load on db121 is CRITICAL: CRITICAL - load average: 8.10, 5.85, 3.36 [18:50:04] RECOVERY - db121 Current Load on db121 is OK: OK - load average: 2.90, 4.98, 3.37 [19:04:58] PROBLEM - ewgf.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'ewgf.wiki' expires in 15 day(s) (Wed 20 Dec 2023 22:59:05 GMT +0000). [19:07:50] RECOVERY - puppetdb121 Puppet on puppetdb121 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:15:41] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is CRITICAL: Name or service not knownHTTP CRITICAL - Unable to open TCP socket [19:28:01] PROBLEM - wiki.conworlds.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.conworlds.org could not be found [19:31:41] PROBLEM - test131 MediaWiki Rendering on test131 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 692 bytes in 0.393 second response time [19:34:58] PROBLEM - ewgf.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address ewgf.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [19:36:28] PROBLEM - mw132 Current Load on mw132 is WARNING: WARNING - load average: 11.23, 10.67, 9.79 [19:38:27] RECOVERY - mw132 Current Load on mw132 is OK: OK - load average: 9.52, 10.03, 9.65 [19:45:36] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.twistwoodtaleswiki.com' expires in 15 day(s) (Wed 20 Dec 2023 22:08:34 GMT +0000). [19:51:41] RECOVERY - test131 MediaWiki Rendering on test131 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 1.219 second response time [20:15:36] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address www.twistwoodtaleswiki.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [21:15:36] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.twistwoodtaleswiki.com' expires in 15 day(s) (Wed 20 Dec 2023 22:08:34 GMT +0000). [21:26:29] PROBLEM - mw132 Current Load on mw132 is WARNING: WARNING - load average: 10.36, 8.72, 7.32 [21:28:28] RECOVERY - mw132 Current Load on mw132 is OK: OK - load average: 7.53, 8.19, 7.30 [21:30:55] PROBLEM - wiki.seamly.net - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.seamly.net' expires in 12 day(s) (Mon 18 Dec 2023 21:20:59 GMT +0000). [21:31:09] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/36bf856cf234...8f19f32853a9 [21:31:10] [02miraheze/ssl] 07MirahezeSSLBot 038f19f32 - Bot: Update SSL cert for wiki.seamly.net [21:41:56] [02puppet] 07paladox closed pull request 03#3544: Revert "Completely remove blackbox from MediaWiki" - 13https://github.com/miraheze/puppet/pull/3544 [21:41:59] [02miraheze/puppet] 07paladox deleted branch 03revert-2853-Universal-Omega-patch-1 [21:42:00] [02puppet] 07paladox deleted branch 03revert-2853-Universal-Omega-patch-1 - 13https://github.com/miraheze/puppet [21:42:05] [02puppet] 07paladox closed pull request 03#3543: Run blackbox - 13https://github.com/miraheze/puppet/pull/3543 [21:42:06] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-4 [21:42:08] [02puppet] 07paladox deleted branch 03paladox-patch-4 - 13https://github.com/miraheze/puppet [22:00:55] RECOVERY - wiki.seamly.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.seamly.net' will expire on Mon 04 Mar 2024 20:31:01 GMT +0000. [22:15:36] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address www.twistwoodtaleswiki.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [23:21:10] RECOVERY - cloud11 IPMI Sensors on cloud11 is OK: IPMI Status: OK [23:25:10] PROBLEM - cloud11 IPMI Sensors on cloud11 is CRITICAL: IPMI Status: Critical [Cntlr 2 Bay 8 = Critical] [23:57:10]