[00:20:53] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is WARNING: unable to cache publicsuffix.org-tlds.{'urls': ('file:///usr/share/publicsuffix/effective_tld_names.dat', 'https://publicsuffix.org/list/public_suffix_list.dat', 'https://raw.githubusercontent.com/publicsuffix/list/master/public_suffix_list.dat'), 'fallback_to_snapshot': True} in /usr/lib/python3/dist-packages/tldextract/.suffix_cache/publicsuffix.org-tlds/704b6b0e53ba [00:20:54] 92e690044b663f4.tldextract.json. This could refresh the Public Suffix List over HTTP every app startup. Construct your `TLDExtract` with a writable `cache_dir` or set `cache_dir=False` to silence this warning. [Errno 13] Permission denied: '/usr/lib/python3/dist-packages/tldextract/.suffix_cache'NoNameservers: All nameservers failed to answer the query wiki.andreijiroh.uk.eu.org. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SER [00:50:42] RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: unable to cache publicsuffix.org-tlds.{'urls': ('file:///usr/share/publicsuffix/effective_tld_names.dat', 'https://publicsuffix.org/list/public_suffix_list.dat', 'https://raw.githubusercontent.com/publicsuffix/list/master/public_suffix_list.dat'), 'fallback_to_snapshot': True} in /usr/lib/python3/dist-packages/tldextract/.suffix_cache/publicsuffix.org-tlds/704b6b0e53ba2966 [00:50:42] 90044b663f4.tldextract.json. This could refresh the Public Suffix List over HTTP every app startup. Construct your `TLDExtract` with a writable `cache_dir` or set `cache_dir=False` to silence this warning. [Errno 13] Permission denied: '/usr/lib/python3/dist-packages/tldextract/.suffix_cache'SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp25.miraheze.org - CNAME OK [02:12:52] PROBLEM - matomo121 HTTPS on matomo121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:14:46] PROBLEM - matomo121 HTTPS on matomo121 is UNKNOWN: HTTP UNKNOWN: Failed to unchunk message body [02:23:11] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is WARNING: unable to cache publicsuffix.org-tlds.{'urls': ('file:///usr/share/publicsuffix/effective_tld_names.dat', 'https://publicsuffix.org/list/public_suffix_list.dat', 'https://raw.githubusercontent.com/publicsuffix/list/master/public_suffix_list.dat'), 'fallback_to_snapshot': True} in /usr/lib/python3/dist-packages/tldextract/.suffix_cache/publicsuffix.org-tlds/704b6b0e53ba [02:23:11] 92e690044b663f4.tldextract.json. This could refresh the Public Suffix List over HTTP every app startup. Construct your `TLDExtract` with a writable `cache_dir` or set `cache_dir=False` to silence this warning. [Errno 13] Permission denied: '/usr/lib/python3/dist-packages/tldextract/.suffix_cache'NoNameservers: All nameservers failed to answer the query wiki.andreijiroh.uk.eu.org. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SER [02:52:59] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.andreijiroh.uk.eu.org All nameservers failed to answer the query. [02:55:02] PROBLEM - db112 Current Load on db112 is CRITICAL: CRITICAL - load average: 10.49, 9.03, 5.85 [02:57:02] RECOVERY - db112 Current Load on db112 is OK: OK - load average: 3.22, 6.74, 5.39 [03:03:33] PROBLEM - db112 Disk Space on db112 is CRITICAL: DISK CRITICAL - free space: / 7955 MB (5% inode=99%); [03:05:02] PROBLEM - db112 Current Load on db112 is CRITICAL: CRITICAL - load average: 25.87, 11.69, 7.55 [03:19:02] PROBLEM - db112 Current Load on db112 is WARNING: WARNING - load average: 3.34, 7.30, 7.95 [03:22:52] RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: unable to cache publicsuffix.org-tlds.{'urls': ('file:///usr/share/publicsuffix/effective_tld_names.dat', 'https://publicsuffix.org/list/public_suffix_list.dat', 'https://raw.githubusercontent.com/publicsuffix/list/master/public_suffix_list.dat'), 'fallback_to_snapshot': True} in /usr/lib/python3/dist-packages/tldextract/.suffix_cache/publicsuffix.org-tlds/704b6b0e53ba2966 [03:22:52] 90044b663f4.tldextract.json. This could refresh the Public Suffix List over HTTP every app startup. Construct your `TLDExtract` with a writable `cache_dir` or set `cache_dir=False` to silence this warning. [Errno 13] Permission denied: '/usr/lib/python3/dist-packages/tldextract/.suffix_cache'SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp24.miraheze.org - CNAME OK [03:25:02] PROBLEM - db112 Current Load on db112 is CRITICAL: CRITICAL - load average: 24.02, 12.77, 9.53 [03:31:02] PROBLEM - db112 Current Load on db112 is WARNING: WARNING - load average: 3.49, 6.26, 7.60 [03:35:02] PROBLEM - db112 Current Load on db112 is CRITICAL: CRITICAL - load average: 8.67, 8.39, 8.13 [03:37:02] PROBLEM - db112 Current Load on db112 is WARNING: WARNING - load average: 5.59, 7.70, 7.94 [03:37:33] PROBLEM - db112 Disk Space on db112 is WARNING: DISK WARNING - free space: / 13740 MB (10% inode=99%); [03:39:02] PROBLEM - db112 Current Load on db112 is CRITICAL: CRITICAL - load average: 8.25, 7.96, 7.99 [03:51:33] PROBLEM - db112 Disk Space on db112 is CRITICAL: DISK CRITICAL - free space: / 7630 MB (5% inode=99%); [03:54:42] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is WARNING: unable to cache publicsuffix.org-tlds.{'urls': ('file:///usr/share/publicsuffix/effective_tld_names.dat', 'https://publicsuffix.org/list/public_suffix_list.dat', 'https://raw.githubusercontent.com/publicsuffix/list/master/public_suffix_list.dat'), 'fallback_to_snapshot': True} in /usr/lib/python3/dist-packages/tldextract/.suffix_cache/publicsuffix.org-tlds/704b6b0e53ba [03:54:42] 92e690044b663f4.tldextract.json. This could refresh the Public Suffix List over HTTP every app startup. Construct your `TLDExtract` with a writable `cache_dir` or set `cache_dir=False` to silence this warning. [Errno 13] Permission denied: '/usr/lib/python3/dist-packages/tldextract/.suffix_cache'NoNameservers: All nameservers failed to answer the query uk.eu.org. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [04:02:42] PROBLEM - matomo121 HTTPS on matomo121 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 358 bytes in 0.012 second response time [04:04:42] PROBLEM - matomo121 HTTPS on matomo121 is UNKNOWN: HTTP UNKNOWN: Failed to unchunk message body [04:14:07] PROBLEM - www.lgbtqia.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address www.lgbtqia.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [04:16:24] paladox: db112 is full [04:16:56] @orduin also [04:17:05] PROBLEM - phab121 Current Load on phab121 is WARNING: LOAD WARNING - total load average: 1.79, 1.35, 0.87 [04:17:19] PROBLEM - phab121 phabricator.miraheze.org HTTPS on phab121 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.037 second response time [04:17:21] PROBLEM - phab121 phab.miraheze.wiki HTTPS on phab121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:19:07] PROBLEM - phab121 Current Load on phab121 is CRITICAL: LOAD CRITICAL - total load average: 9.08, 3.40, 1.63 [04:20:52] PROBLEM - matomo121 HTTPS on matomo121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:22:52] PROBLEM - matomo121 HTTPS on matomo121 is UNKNOWN: HTTP UNKNOWN: Failed to unchunk message body [04:24:39] As a result non-mediawiki services that use a db are down [04:27:31] RECOVERY - phab121 phabricator.miraheze.org HTTPS on phab121 is OK: HTTP OK: HTTP/1.1 200 OK - 20259 bytes in 8.007 second response time [04:27:33] RECOVERY - db112 Disk Space on db112 is OK: DISK OK - free space: / 15381 MB (11% inode=99%); [04:29:12] RECOVERY - phab121 phab.miraheze.wiki HTTPS on phab121 is OK: HTTP OK: Status line output matched "HTTP/1.1 200" - 17683 bytes in 0.294 second response time [04:35:05] PROBLEM - phab121 Current Load on phab121 is WARNING: LOAD WARNING - total load average: 0.62, 1.40, 1.83 [04:36:28] PROBLEM - db112 Disk Space on db112 is WARNING: DISK WARNING - free space: / 14035 MB (10% inode=99%); [04:36:40] PROBLEM - ns1 NTP time on ns1 is CRITICAL: NTP CRITICAL: Offset 0.5007888377 secs [04:37:05] RECOVERY - phab121 Current Load on phab121 is OK: LOAD OK - total load average: 0.50, 1.13, 1.68 [04:41:05] PROBLEM - phab121 Current Load on phab121 is CRITICAL: LOAD CRITICAL - total load average: 2.26, 1.92, 1.88 [04:43:05] RECOVERY - phab121 Current Load on phab121 is OK: LOAD OK - total load average: 0.36, 1.29, 1.65 [04:48:52] PROBLEM - matomo121 HTTPS on matomo121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:50:46] PROBLEM - matomo121 HTTPS on matomo121 is UNKNOWN: HTTP UNKNOWN: Failed to unchunk message body [04:51:59] RECOVERY - db112 Disk Space on db112 is OK: DISK OK - free space: / 15281 MB (11% inode=99%); [04:59:02] PROBLEM - db112 Current Load on db112 is WARNING: WARNING - load average: 3.96, 6.73, 7.87 [05:03:02] RECOVERY - db112 Current Load on db112 is OK: OK - load average: 2.97, 4.71, 6.78 [05:05:56] PROBLEM - matomo121 Current Load on matomo121 is CRITICAL: LOAD CRITICAL - total load average: 4.33, 3.69, 2.53 [05:07:57] RECOVERY - matomo121 Current Load on matomo121 is OK: LOAD OK - total load average: 2.10, 3.12, 2.47 [05:13:56] PROBLEM - matomo121 Current Load on matomo121 is WARNING: LOAD WARNING - total load average: 3.36, 3.44, 2.82 [05:15:57] RECOVERY - matomo121 Current Load on matomo121 is OK: LOAD OK - total load average: 1.97, 2.88, 2.69 [05:21:37] Seems to have resolved itself without intervention [05:21:51] possibly a log rotation saved some space [05:24:06] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.andreijiroh.uk.eu.org All nameservers failed to answer the query. [05:53:55] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is WARNING: unable to cache publicsuffix.org-tlds.{'urls': ('file:///usr/share/publicsuffix/effective_tld_names.dat', 'https://publicsuffix.org/list/public_suffix_list.dat', 'https://raw.githubusercontent.com/publicsuffix/list/master/public_suffix_list.dat'), 'fallback_to_snapshot': True} in /usr/lib/python3/dist-packages/tldextract/.suffix_cache/publicsuffix.org-tlds/704b6b0e53ba [05:53:55] 92e690044b663f4.tldextract.json. This could refresh the Public Suffix List over HTTP every app startup. Construct your `TLDExtract` with a writable `cache_dir` or set `cache_dir=False` to silence this warning. [Errno 13] Permission denied: '/usr/lib/python3/dist-packages/tldextract/.suffix_cache'NoNameservers: All nameservers failed to answer the query uk.eu.org. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [06:23:43] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.andreijiroh.uk.eu.org All nameservers failed to answer the query. [06:35:31] PROBLEM - ldap141 Puppet on ldap141 is UNKNOWN: NRPE: Unable to read output [08:17:12] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [08:47:09] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: unable to cache publicsuffix.org-tlds.{'urls': ('file:///usr/share/publicsuffix/effective_tld_names.dat', 'https://publicsuffix.org/list/public_suffix_list.dat', 'https://raw.githubusercontent.com/publicsuffix/list/master/public_suffix_list.dat'), 'fallback_to_snapshot': True} in /usr/lib/python3/dist-packages/tldextract/.suffix_cache/publicsuffix.org-tlds/704b6b0e53ba2966d92e [08:47:09] 4b663f4.tldextract.json. This could refresh the Public Suffix List over HTTP every app startup. Construct your `TLDExtract` with a writable `cache_dir` or set `cache_dir=False` to silence this warning. [Errno 13] Permission denied: '/usr/lib/python3/dist-packages/tldextract/.suffix_cache'SSL WARNING - rDNS OK but records conflict. {'NS': ['ns.ankh.fr.eu.org.', 'ns1.eu.org.', 'ns1.eriomem.net.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [08:57:56] PROBLEM - matomo121 Current Load on matomo121 is WARNING: LOAD WARNING - total load average: 3.52, 3.39, 2.74 [08:59:56] RECOVERY - matomo121 Current Load on matomo121 is OK: LOAD OK - total load average: 2.79, 3.37, 2.82 [09:20:09] PROBLEM - wiki.litek.top - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'wiki.litek.top' expires in 7 day(s) (Fri 27 Oct 2023 08:58:10 GMT +0000). [09:22:32] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is WARNING: unable to cache publicsuffix.org-tlds.{'urls': ('file:///usr/share/publicsuffix/effective_tld_names.dat', 'https://publicsuffix.org/list/public_suffix_list.dat', 'https://raw.githubusercontent.com/publicsuffix/list/master/public_suffix_list.dat'), 'fallback_to_snapshot': True} in /usr/lib/python3/dist-packages/tldextract/.suffix_cache/publicsuffix.org-tlds/704b6b0e53ba [09:22:32] 92e690044b663f4.tldextract.json. This could refresh the Public Suffix List over HTTP every app startup. Construct your `TLDExtract` with a writable `cache_dir` or set `cache_dir=False` to silence this warning. [Errno 13] Permission denied: '/usr/lib/python3/dist-packages/tldextract/.suffix_cache'NoNameservers: All nameservers failed to answer the query uk.eu.org. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [09:52:20] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.andreijiroh.uk.eu.org All nameservers failed to answer the query. [10:42:03] RECOVERY - cloud11 IPMI Sensors on cloud11 is OK: IPMI Status: OK [10:46:03] PROBLEM - cloud11 IPMI Sensors on cloud11 is CRITICAL: IPMI Status: Critical [Cntlr 2 Bay 8 = Critical] [10:51:56] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is WARNING: unable to cache publicsuffix.org-tlds.{'urls': ('file:///usr/share/publicsuffix/effective_tld_names.dat', 'https://publicsuffix.org/list/public_suffix_list.dat', 'https://raw.githubusercontent.com/publicsuffix/list/master/public_suffix_list.dat'), 'fallback_to_snapshot': True} in /usr/lib/python3/dist-packages/tldextract/.suffix_cache/publicsuffix.org-tlds/704b6b0e53ba [10:51:56] 92e690044b663f4.tldextract.json. This could refresh the Public Suffix List over HTTP every app startup. Construct your `TLDExtract` with a writable `cache_dir` or set `cache_dir=False` to silence this warning. [Errno 13] Permission denied: '/usr/lib/python3/dist-packages/tldextract/.suffix_cache'NoNameservers: All nameservers failed to answer the query uk.eu.org. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [11:11:53] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/7e6f8c2abc04...af9171593536 [11:11:55] [02miraheze/puppet] 07paladox 03af91715 - opensearch: upgrade to 2.11 [11:17:09] PROBLEM - prometheus131 APT on prometheus131 is CRITICAL: APT CRITICAL: 58 packages available for upgrade (2 critical updates). [11:18:28] PROBLEM - os131 Puppet on os131 is UNKNOWN: NRPE: Unable to read output [11:19:07] RECOVERY - prometheus131 APT on prometheus131 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [11:20:29] RECOVERY - os131 Puppet on os131 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [11:21:47] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.andreijiroh.uk.eu.org All nameservers failed to answer the query. [11:22:54] RECOVERY - os141 APT on os141 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [11:23:20] RECOVERY - os131 APT on os131 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [11:26:19] PROBLEM - prometheus131 Puppet on prometheus131 is UNKNOWN: NRPE: Unable to read output [11:27:57] PROBLEM - ping6 on os131 is CRITICAL: CRITICAL - Host Unreachable (2a10:6740::6:401) [11:28:15] RECOVERY - prometheus131 Puppet on prometheus131 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [11:28:39] PROBLEM - os141 Puppet on os141 is UNKNOWN: NRPE: Unable to read output [11:29:58] RECOVERY - ping6 on os131 is OK: PING OK - Packet loss = 0%, RTA = 0.74 ms [11:30:05] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/af9171593536...031123a3bf52 [11:30:08] [02miraheze/puppet] 07paladox 03031123a - graylog: upgrade mongodb to 5.0.21 [11:31:33] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/031123a3bf52...b39d1c2475c9 [11:31:36] [02miraheze/puppet] 07paladox 03b39d1c2 - graylog: upgrade mongodb to 6.0.11 [11:32:21] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/b39d1c2475c9...0eaafbcbe272 [11:32:24] [02miraheze/puppet] 07paladox 030eaafbc - graylog: upgrade to 5.1.7 [11:35:44] PROBLEM - graylog131 Puppet on graylog131 is UNKNOWN: NRPE: Unable to read output [11:38:19] RECOVERY - os141 Puppet on os141 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [11:39:47] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/0eaafbcbe272...01ce00f0e03b [11:39:50] [02miraheze/puppet] 07paladox 0301ce00f - Revert "graylog: upgrade mongodb to 6.0.11" [11:40:26] PROBLEM - graylog131 APT on graylog131 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [11:41:44] RECOVERY - graylog131 Puppet on graylog131 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [11:42:26] PROBLEM - graylog131 APT on graylog131 is CRITICAL: APT CRITICAL: 58 packages available for upgrade (26 critical updates). [11:46:30] RECOVERY - graylog131 APT on graylog131 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [11:51:36] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is WARNING: unable to cache publicsuffix.org-tlds.{'urls': ('file:///usr/share/publicsuffix/effective_tld_names.dat', 'https://publicsuffix.org/list/public_suffix_list.dat', 'https://raw.githubusercontent.com/publicsuffix/list/master/public_suffix_list.dat'), 'fallback_to_snapshot': True} in /usr/lib/python3/dist-packages/tldextract/.suffix_cache/publicsuffix.org-tlds/704b6b0e53ba [11:51:36] 92e690044b663f4.tldextract.json. This could refresh the Public Suffix List over HTTP every app startup. Construct your `TLDExtract` with a writable `cache_dir` or set `cache_dir=False` to silence this warning. [Errno 13] Permission denied: '/usr/lib/python3/dist-packages/tldextract/.suffix_cache'NoNameservers: All nameservers failed to answer the query wiki.andreijiroh.uk.eu.org. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SER [11:52:35] PROBLEM - graylog131 HTTPS on graylog131 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 295 bytes in 0.014 second response time [11:55:36] PROBLEM - graylog131 APT on graylog131 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [11:57:31] RECOVERY - graylog131 APT on graylog131 is OK: APT OK: 2 packages available for upgrade (0 critical updates). [12:05:43] PROBLEM - graylog131 Puppet on graylog131 is UNKNOWN: NRPE: Unable to read output [12:11:26] PROBLEM - os131 APT on os131 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [12:11:29] PROBLEM - os141 APT on os141 is CRITICAL: APT CRITICAL: 152 packages available for upgrade (13 critical updates). [12:13:28] RECOVERY - os141 APT on os141 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [12:15:37] RECOVERY - os131 APT on os131 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [12:17:44] RECOVERY - graylog131 HTTPS on graylog131 is OK: HTTP OK: HTTP/1.1 200 OK - 2086 bytes in 0.404 second response time [12:17:47] RECOVERY - graylog131 Puppet on graylog131 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [12:21:13] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 5.99, 5.45, 2.52 [12:21:19] PROBLEM - Host os131 is DOWN: CRITICAL - Host Unreachable (2a10:6740::6:401) [12:21:24] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.andreijiroh.uk.eu.org All nameservers failed to answer the query. [12:23:22] RECOVERY - Host os131 is UP: PING OK - Packet loss = 0%, RTA = 2.05 ms [12:23:56] PROBLEM - os141 Puppet on os141 is UNKNOWN: NRPE: Unable to read output [12:24:28] PROBLEM - os131 Puppet on os131 is UNKNOWN: NRPE: Unable to read output [12:26:12] PROBLEM - os131 Current Load on os131 is CRITICAL: LOAD CRITICAL - total load average: 4.30, 2.73, 1.19 [12:28:28] RECOVERY - os131 Puppet on os131 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [12:30:07] PROBLEM - os131 Current Load on os131 is WARNING: LOAD WARNING - total load average: 3.55, 3.62, 1.93 [12:32:04] RECOVERY - os131 Current Load on os131 is OK: LOAD OK - total load average: 2.36, 3.17, 1.97 [12:37:57] RECOVERY - os141 Puppet on os141 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [12:43:13] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.47, 3.42, 3.97 [12:47:33] PROBLEM - db112 Disk Space on db112 is WARNING: DISK WARNING - free space: / 14608 MB (10% inode=99%); [12:49:13] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 1.72, 2.24, 3.26 [12:51:14] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is WARNING: unable to cache publicsuffix.org-tlds.{'urls': ('file:///usr/share/publicsuffix/effective_tld_names.dat', 'https://publicsuffix.org/list/public_suffix_list.dat', 'https://raw.githubusercontent.com/publicsuffix/list/master/public_suffix_list.dat'), 'fallback_to_snapshot': True} in /usr/lib/python3/dist-packages/tldextract/.suffix_cache/publicsuffix.org-tlds/704b6b0e53ba [12:51:14] 92e690044b663f4.tldextract.json. This could refresh the Public Suffix List over HTTP every app startup. Construct your `TLDExtract` with a writable `cache_dir` or set `cache_dir=False` to silence this warning. [Errno 13] Permission denied: '/usr/lib/python3/dist-packages/tldextract/.suffix_cache'NoNameservers: All nameservers failed to answer the query uk.eu.org. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [13:21:03] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.andreijiroh.uk.eu.org All nameservers failed to answer the query. [13:25:59] paladox: seems like /usr/lib/python3/dist-packages/tldextract/.suffix_cache on mon141 is not writable which gives us that warning [13:32:36] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/97715d88f9b3 [13:32:38] [02miraheze/puppet] 07paladox 0397715d8 - check_reverse_dns: don't cache [13:32:41] [02puppet] 07paladox created branch 03paladox-patch-12 - 13https://github.com/miraheze/puppet [13:32:44] [02puppet] 07paladox opened pull request 03#3467: check_reverse_dns: don't cache - 13https://github.com/miraheze/puppet/pull/3467 [13:34:34] [02puppet] 07paladox closed pull request 03#3467: check_reverse_dns: don't cache - 13https://github.com/miraheze/puppet/pull/3467 [13:34:36] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/01ce00f0e03b...691c871997eb [13:34:38] [02miraheze/puppet] 07paladox 03691c871 - check_reverse_dns: don't cache (#3467) [13:34:40] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-12 [13:34:42] [02puppet] 07paladox deleted branch 03paladox-patch-12 - 13https://github.com/miraheze/puppet [13:50:32] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/8fd5885eaebc [13:50:34] [02miraheze/puppet] 07paladox 038fd5885 - monitoring: Add check_curl [13:50:35] [02puppet] 07paladox created branch 03paladox-patch-12 - 13https://github.com/miraheze/puppet [13:50:36] [02puppet] 07paladox opened pull request 03#3468: monitoring: Add check_curl - 13https://github.com/miraheze/puppet/pull/3468 [13:50:39] PROBLEM - permanentfuturelab.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address permanentfuturelab.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [13:50:52] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.andreijiroh.uk.eu.org. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [13:50:55] [02puppet] 07paladox edited pull request 03#3468: monitoring: Add check_curl - 13https://github.com/miraheze/puppet/pull/3468 [13:51:16] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/8fd5885eaebc...a57dd8fa659a [13:51:18] [02miraheze/puppet] 07paladox 03a57dd8f - Update site.pp [13:51:19] [02puppet] 07paladox synchronize pull request 03#3468: monitoring: Add check_curl - 13https://github.com/miraheze/puppet/pull/3468 [13:51:24] [02puppet] 07paladox closed pull request 03#3468: monitoring: Add check_curl - 13https://github.com/miraheze/puppet/pull/3468 [13:51:27] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/puppet/compare/691c871997eb...c5a54fecd487 [13:51:30] [02miraheze/puppet] 07paladox 03c5a54fe - monitoring: Add check_curl (#3468) [13:51:31] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-12 [13:51:34] [02puppet] 07paladox deleted branch 03paladox-patch-12 - 13https://github.com/miraheze/puppet [13:51:35] [02puppet] 07paladox closed pull request 03#3461: openldap: update - 13https://github.com/miraheze/puppet/pull/3461 [13:51:37] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-4 [13:51:39] [02puppet] 07paladox deleted branch 03paladox-patch-4 - 13https://github.com/miraheze/puppet [13:54:10] RECOVERY - mon141 HTTPS on mon141 is OK: HTTP OK: HTTP/2 200 - 336 bytes in 0.052 second response time [13:54:38] RECOVERY - reports121 HTTPS on reports121 is OK: HTTP OK: HTTP/2 200 - 1139 bytes in 0.033 second response time [14:19:48] RECOVERY - permanentfuturelab.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'permanentfuturelab.wiki' will expire on Mon 18 Dec 2023 02:22:48 GMT +0000. [14:20:40] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.andreijiroh.uk.eu.org All nameservers failed to answer the query. [14:24:04] RECOVERY - matomo121 HTTPS on matomo121 is OK: HTTP OK: HTTP/2 200 - 552 bytes in 0.246 second response time [14:52:33] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/438d1fad8080...fca73592067d [14:52:35] [02miraheze/dns] 07paladox 03fca7359 - Add _AA9BB7CAC3A7BBDE50AD91635C760D21 cname to miraheze.org [14:57:54] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/fca73592067d...42fec1d202b3 [14:57:57] [02miraheze/dns] 07paladox 0342fec1d - Fix [15:09:58] [02miraheze/ssl] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ssl/commit/2cddf250c152 [15:09:59] [02miraheze/ssl] 07paladox 032cddf25 - Update wildcard.miraheze.org cert [15:10:00] [02ssl] 07paladox created branch 03paladox-patch-1 - 13https://github.com/miraheze/ssl [15:10:02] [02ssl] 07paladox opened pull request 03#731: Update wildcard.miraheze.org cert - 13https://github.com/miraheze/ssl/pull/731 [15:13:10] [02ssl] 07paladox synchronize pull request 03#731: Update wildcard.miraheze.org cert - 13https://github.com/miraheze/ssl/pull/731 [15:13:13] [02miraheze/ssl] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/2cddf250c152...b7b1d068a335 [15:13:14] [02miraheze/ssl] 07paladox 03b7b1d06 - Update wildcard.miraheze.org-2020-2.crt [15:14:06] [02ssl] 07paladox closed pull request 03#731: Update wildcard.miraheze.org cert - 13https://github.com/miraheze/ssl/pull/731 [15:14:07] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/2d3d776501ac...23e40e4456dc [15:14:10] [02miraheze/ssl] 07paladox 0323e40e4 - Update wildcard.miraheze.org cert (#731) [15:14:12] [02miraheze/ssl] 07paladox deleted branch 03paladox-patch-1 [15:14:14] [02ssl] 07paladox deleted branch 03paladox-patch-1 - 13https://github.com/miraheze/ssl [15:17:40] RECOVERY - nintendo.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Mon 18 Nov 2024 23:59:59 GMT +0000. [15:18:05] RECOVERY - wc.miraheze.org on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Mon 18 Nov 2024 23:59:59 GMT +0000. [15:18:24] RECOVERY - sega.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Mon 18 Nov 2024 23:59:59 GMT +0000. [15:18:58] RECOVERY - smashbros.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Mon 18 Nov 2024 23:59:59 GMT +0000. [15:19:10] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/c5a54fecd487...e877b77afd46 [15:19:12] [02miraheze/puppet] 07paladox 03e877b77 - ssl::wildcard: fix reloading nginx [15:20:06] RECOVERY - wiki.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Mon 18 Nov 2024 23:59:59 GMT +0000. [15:20:21] RECOVERY - recaptime.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Mon 18 Nov 2024 23:59:59 GMT +0000. [15:20:24] PROBLEM - ldap141 Puppet on ldap141 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:20:40] RECOVERY - donkeykong.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Mon 18 Nov 2024 23:59:59 GMT +0000. [15:20:41] RECOVERY - lgbta.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Mon 18 Nov 2024 23:59:59 GMT +0000. [15:21:08] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/e877b77afd46...b7cc72cd435d [15:21:10] RECOVERY - crashspyro.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Mon 18 Nov 2024 23:59:59 GMT +0000. [15:21:11] [02miraheze/puppet] 07paladox 03b7cc72c - Revert "ssl::wildcard: fix reloading nginx" [15:22:09] PROBLEM - cloud13 Puppet on cloud13 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:22:10] PROBLEM - cloud12 Puppet on cloud12 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:22:13] RECOVERY - croc.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Mon 18 Nov 2024 23:59:59 GMT +0000. [15:22:26] PROBLEM - jobchron121 Puppet on jobchron121 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:22:37] PROBLEM - mem141 Puppet on mem141 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:22:38] PROBLEM - db112 Puppet on db112 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:22:43] PROBLEM - db142 Puppet on db142 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:22:44] PROBLEM - bast121 Puppet on bast121 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:22:44] PROBLEM - bast141 Puppet on bast141 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:22:56] PROBLEM - cloud14 Puppet on cloud14 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:22:56] PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:22:57] PROBLEM - db131 Puppet on db131 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:22:59] PROBLEM - db101 Puppet on db101 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:23:00] PROBLEM - swiftobject121 Puppet on swiftobject121 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:23:05] PROBLEM - db121 Puppet on db121 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:23:08] PROBLEM - cloud10 Puppet on cloud10 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:23:32] PROBLEM - cloud11 Puppet on cloud11 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:23:37] PROBLEM - swiftac111 Puppet on swiftac111 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:24:05] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/42fec1d202b3...82bcffe38f62 [15:24:07] [02miraheze/dns] 07paladox 0382bcffe - Remove _AA9BB7CAC3A7BBDE50AD91635C760D21 cname [15:24:08] RECOVERY - fanpedia.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Mon 18 Nov 2024 23:59:59 GMT +0000. [15:24:20] PROBLEM - mem131 Puppet on mem131 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:24:25] PROBLEM - os141 Puppet on os141 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:24:29] PROBLEM - puppetdb121 Puppet on puppetdb121 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:24:36] RECOVERY - db112 Puppet on db112 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:24:42] RECOVERY - db142 Puppet on db142 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:24:52] PROBLEM - ns1 SSH on ns1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:24:57] PROBLEM - swiftobject111 Puppet on swiftobject111 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:24:57] RECOVERY - db131 Puppet on db131 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:24:57] PROBLEM - swiftobject113 Puppet on swiftobject113 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:24:59] RECOVERY - db101 Puppet on db101 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:25:05] RECOVERY - db121 Puppet on db121 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:25:08] PROBLEM - swiftobject112 Puppet on swiftobject112 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:25:09] PROBLEM - swiftobject101 Puppet on swiftobject101 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:25:12] RECOVERY - tintinpedia.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Mon 18 Nov 2024 23:59:59 GMT +0000. [15:25:14] RECOVERY - thelangyalist.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Mon 18 Nov 2024 23:59:59 GMT +0000. [15:25:29] PROBLEM - ns1 NTP time on ns1 is WARNING: NTP WARNING: Offset -0.3742852211 secs [15:25:44] PROBLEM - swiftobject122 Puppet on swiftobject122 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:26:19] RECOVERY - mem131 Puppet on mem131 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:26:22] RECOVERY - piwik.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Mon 18 Nov 2024 23:59:59 GMT +0000. [15:26:52] PROBLEM - ns1 Current Load on ns1 is CRITICAL: connect to address 2607:f1c0:1800:1f2::1 port 5666: Connection refusedconnect to host 2607:f1c0:1800:1f2::1 port 5666: Connection refused [15:27:04] RECOVERY - cnt.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Mon 18 Nov 2024 23:59:59 GMT +0000. [15:27:06] RECOVERY - dragonquest2.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Mon 18 Nov 2024 23:59:59 GMT +0000. [15:27:11] RECOVERY - wario.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Mon 18 Nov 2024 23:59:59 GMT +0000. [15:27:16] RECOVERY - template.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Mon 18 Nov 2024 23:59:59 GMT +0000. [15:28:06] RECOVERY - taswin.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Mon 18 Nov 2024 23:59:59 GMT +0000. [15:28:46] RECOVERY - ns1 SSH on ns1 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u1 (protocol 2.0) [15:28:47] RECOVERY - ns1 Current Load on ns1 is OK: LOAD OK - total load average: 0.60, 0.48, 0.20 [15:28:48] RECOVERY - metroid.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Mon 18 Nov 2024 23:59:59 GMT +0000. [15:29:59] RECOVERY - rare.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Mon 18 Nov 2024 23:59:59 GMT +0000. [15:30:09] RECOVERY - cloud13 Puppet on cloud13 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [15:30:10] RECOVERY - cloud12 Puppet on cloud12 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [15:30:21] PROBLEM - ldap141 Puppet on ldap141 is UNKNOWN: NRPE: Unable to read output [15:30:26] RECOVERY - jobchron121 Puppet on jobchron121 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [15:30:31] RECOVERY - mem141 Puppet on mem141 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [15:30:44] RECOVERY - bast141 Puppet on bast141 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:30:44] RECOVERY - bast121 Puppet on bast121 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:30:56] RECOVERY - cloud14 Puppet on cloud14 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:30:56] RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [15:31:00] RECOVERY - swiftobject101 Puppet on swiftobject101 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [15:31:07] RECOVERY - swiftobject112 Puppet on swiftobject112 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [15:31:08] RECOVERY - cloud10 Puppet on cloud10 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:31:32] RECOVERY - cloud11 Puppet on cloud11 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:31:37] RECOVERY - swiftac111 Puppet on swiftac111 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [15:31:44] RECOVERY - swiftobject122 Puppet on swiftobject122 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [15:32:16] RECOVERY - puppetdb121 Puppet on puppetdb121 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:32:19] PROBLEM - ns1 NTP time on ns1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:32:20] RECOVERY - ajhalili2006.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Mon 18 Nov 2024 23:59:59 GMT +0000. [15:32:26] RECOVERY - os141 Puppet on os141 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:32:51] RECOVERY - swiftobject121 Puppet on swiftobject121 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:32:57] RECOVERY - swiftobject111 Puppet on swiftobject111 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:32:58] RECOVERY - swiftobject113 Puppet on swiftobject113 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:33:08] PROBLEM - ns1 SSH on ns1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:33:23] RECOVERY - schizoidnightmares.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Mon 18 Nov 2024 23:59:59 GMT +0000. [15:33:25] RECOVERY - www.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Mon 18 Nov 2024 23:59:59 GMT +0000. [15:34:21] RECOVERY - yoshi.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Mon 18 Nov 2024 23:59:59 GMT +0000. [15:35:21] RECOVERY - closinglogosgroup.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Mon 18 Nov 2024 23:59:59 GMT +0000. [15:35:36] PROBLEM - ns1 Current Load on ns1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:35:53] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:36:09] RECOVERY - yokaiwatch.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Mon 18 Nov 2024 23:59:59 GMT +0000. [15:36:10] RECOVERY - commonswiki.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Mon 18 Nov 2024 23:59:59 GMT +0000. [15:36:12] PROBLEM - ns1 Disk Space on ns1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:36:15] PROBLEM - ns1 conntrack_table_size on ns1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:36:17] PROBLEM - ns1 APT on ns1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:36:18] RECOVERY - pokemon.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Mon 18 Nov 2024 23:59:59 GMT +0000. [15:36:21] RECOVERY - projectsekai.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Mon 18 Nov 2024 23:59:59 GMT +0000. [15:36:32] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:37:00] PROBLEM - ns1 ferm_active on ns1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:37:17] RECOVERY - sims.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Mon 18 Nov 2024 23:59:59 GMT +0000. [15:38:14] RECOVERY - uncyclopediakr.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Mon 18 Nov 2024 23:59:59 GMT +0000. [15:39:59] RECOVERY - mario.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Mon 18 Nov 2024 23:59:59 GMT +0000. [15:40:09] RECOVERY - allthetropes.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Mon 18 Nov 2024 23:59:59 GMT +0000. [15:40:54] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.43, 3.18, 2.57 [15:41:24] PROBLEM - puppet141 Puppet on puppet141 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:42:16] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 1.84, 2.75, 2.47 [15:44:34] RECOVERY - finalfantasy.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Mon 18 Nov 2024 23:59:59 GMT +0000. [15:46:21] RECOVERY - squareenix.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Mon 18 Nov 2024 23:59:59 GMT +0000. [15:46:47] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/b7cc72cd435d...ab3947cac617 [15:46:50] [02miraheze/puppet] 07paladox 03ab3947c - openldap: Replace libldap-2.4-2 with libldap-common [15:47:52] RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp25.miraheze.org - CNAME OK [15:48:29] RECOVERY - ldap141 Puppet on ldap141 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [16:00:15] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [16:09:04] RECOVERY - puppet141 Puppet on puppet141 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [16:19:35] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.andreijiroh.uk.eu.org All nameservers failed to answer the query. [16:29:27] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns.ankh.fr.eu.org.', 'ns1.eu.org.', 'ns1.eriomem.net.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [17:35:16] [02mw-config] 07redbluegreenhat synchronize pull request 03#5269: T10924: Add some ApprovedRevs settings - 13https://github.com/miraheze/mw-config/pull/5269 [17:36:14] miraheze/mw-config - redbluegreenhat the build passed. [17:39:38] [02mw-config] 07redbluegreenhat commented on pull request 03#5269: T10924: Add some ApprovedRevs settings - 13https://github.com/miraheze/mw-config/pull/5269#issuecomment-1771437618 [17:45:24] [02mw-config] 07redbluegreenhat synchronize pull request 03#5269: T10924: Add some ApprovedRevs settings - 13https://github.com/miraheze/mw-config/pull/5269 [17:46:27] miraheze/mw-config - redbluegreenhat the build passed. [17:48:48] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query uk.eu.org. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [17:57:04] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [18:03:20] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.64, 6.37, 5.57 [18:04:54] [02miraheze/phabricator-extensions] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/phabricator-extensions/compare/74f097b98a80...1e95581044b7 [18:04:57] [02miraheze/phabricator-extensions] 07paladox 031e95581 - Fix path to phorge [18:05:16] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 4.68, 5.85, 5.48 [18:05:53] miraheze/phabricator-extensions - paladox the build passed. [18:08:25] [02miraheze/phabricator-extensions] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/phabricator-extensions/compare/1e95581044b7...5b7ec1620afc [18:08:27] [02miraheze/phabricator-extensions] 07paladox 035b7ec16 - Fix PHP 8.1 auth view strlen(null) error [18:09:14] miraheze/phabricator-extensions - paladox the build passed. [18:18:32] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.andreijiroh.uk.eu.org All nameservers failed to answer the query. [18:22:43] !log [void@phab121] rollback actions by DontLetYourselfBeDisrespected [18:22:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:23:25] !log [void@phab121] bin/remove destroy @7chakrasyogaschool -- spam [18:23:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:26:15] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns.ankh.fr.eu.org.', 'ns1.eu.org.', 'ns1.eriomem.net.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [18:52:45] PROBLEM - swiftobject112 NTP time on swiftobject112 is WARNING: NTP WARNING: Offset 0.1079114974 secs [18:59:22] PROBLEM - mem141 NTP time on mem141 is WARNING: NTP WARNING: Offset 0.1086644828 secs [19:12:57] RECOVERY - swiftobject112 NTP time on swiftobject112 is OK: NTP OK: Offset 0.08900865912 secs [19:17:58] RECOVERY - mem141 NTP time on mem141 is OK: NTP OK: Offset 0.09399971366 secs [19:47:44] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query uk.eu.org. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [20:17:29] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.andreijiroh.uk.eu.org All nameservers failed to answer the query. [20:47:14] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query uk.eu.org. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [20:57:21] RECOVERY - cloud11 IPMI Sensors on cloud11 is OK: IPMI Status: OK [21:01:17] PROBLEM - cloud11 IPMI Sensors on cloud11 is CRITICAL: IPMI Status: Critical [Cntlr 2 Bay 8 = Critical] [21:16:58] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.andreijiroh.uk.eu.org All nameservers failed to answer the query. [21:46:47] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.andreijiroh.uk.eu.org. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [21:50:40] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [22:16:31] RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp25.miraheze.org - CNAME OK [22:19:53] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns.ankh.fr.eu.org.', 'ns1.eu.org.', 'ns1.eriomem.net.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [23:04:26] PROBLEM - db112 Current Load on db112 is CRITICAL: CRITICAL - load average: 13.51, 8.25, 3.88 [23:08:26] PROBLEM - db112 Current Load on db112 is WARNING: WARNING - load average: 3.66, 7.94, 4.94 [23:10:26] RECOVERY - db112 Current Load on db112 is OK: OK - load average: 0.67, 5.41, 4.37 [23:40:15] PROBLEM - wiki.anempireofdreams.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.anempireofdreams.com All nameservers failed to answer the query.