[00:01:29] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.08, 6.33, 5.77 [00:03:29] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 5.94, 6.00, 5.71 [00:21:06] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 10.75, 9.08, 6.78 [00:21:29] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 9.79, 9.56, 7.29 [00:25:06] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.81, 7.31, 6.55 [00:25:29] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 5.37, 7.61, 7.04 [00:26:47] PROBLEM - wiki.artmechanicum.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.artmechanicum.com All nameservers failed to answer the query. [00:27:06] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.67, 7.73, 6.79 [00:29:06] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.50, 7.31, 6.75 [00:31:06] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.09, 6.54, 6.54 [00:38:29] PROBLEM - matomo121 Current Load on matomo121 is WARNING: WARNING - load average: 3.53, 3.20, 2.80 [00:40:28] RECOVERY - matomo121 Current Load on matomo121 is OK: OK - load average: 3.02, 3.12, 2.82 [00:44:26] PROBLEM - matomo121 Current Load on matomo121 is WARNING: WARNING - load average: 3.61, 3.46, 3.03 [00:46:25] RECOVERY - matomo121 Current Load on matomo121 is OK: OK - load average: 2.60, 3.10, 2.96 [00:53:29] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 4.53, 5.88, 6.53 [00:53:30] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.89, 3.67, 3.36 [00:56:20] PROBLEM - wiki.artmechanicum.com - reverse DNS on sslhost is WARNING: Timeout: The DNS operation timed out after 5.406946420669556 seconds [00:57:24] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.60, 3.14, 3.22 [00:57:29] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.99, 6.89, 6.74 [00:59:29] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 5.22, 6.16, 6.49 [01:05:29] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.31, 6.93, 6.73 [01:07:12] PROBLEM - graylog131 Current Load on graylog131 is WARNING: WARNING - load average: 3.95, 3.00, 2.43 [01:09:06] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.31, 6.86, 6.12 [01:09:09] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.06, 3.43, 3.21 [01:09:10] RECOVERY - graylog131 Current Load on graylog131 is OK: OK - load average: 1.51, 2.43, 2.29 [01:09:29] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 5.73, 6.57, 6.67 [01:11:06] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.85, 6.64, 6.11 [01:13:04] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.65, 3.62, 3.35 [01:13:06] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.04, 6.05, 5.96 [01:15:01] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.55, 3.86, 3.47 [01:20:52] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.49, 3.92, 3.63 [01:22:50] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.43, 4.04, 3.70 [01:24:47] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.09, 3.59, 3.57 [01:25:50] PROBLEM - wiki.artmechanicum.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.artmechanicum.com All nameservers failed to answer the query. [01:28:41] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.39, 3.01, 3.34 [01:36:25] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.59, 6.65, 6.19 [01:38:24] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 4.99, 6.10, 6.04 [01:44:16] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.65, 3.44, 3.30 [01:50:09] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.14, 2.99, 3.18 [02:29:48] PROBLEM - sparabel.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'sparabel.wiki' expires in 14 day(s) (Sat 07 Oct 2023 20:33:09 GMT +0000). [02:40:02] RECOVERY - ns1 NTP time on ns1 is OK: NTP OK: Offset 0.004664719105 secs [02:40:09] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.31, 3.52, 3.02 [02:42:09] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.60, 3.09, 2.92 [02:54:24] PROBLEM - wiki.artmechanicum.com - reverse DNS on sslhost is WARNING: Timeout: The DNS operation timed out after 5.405654191970825 seconds [02:59:40] PROBLEM - sparabel.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address sparabel.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [03:23:52] RECOVERY - wiki.artmechanicum.com - reverse DNS on sslhost is OK: SSL OK - wiki.artmechanicum.com reverse DNS resolves to cp25.miraheze.org - CNAME OK [04:21:37] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 12.96, 8.02, 6.42 [04:23:35] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 5.59, 6.89, 6.20 [04:25:34] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 5.21, 6.45, 6.13 [04:35:38] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.76, 3.20, 2.88 [04:39:33] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.05, 3.54, 3.09 [04:41:30] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.83, 3.62, 3.17 [04:51:16] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 3.03, 3.31, 3.28 [05:06:10] PROBLEM - wiki.zulaclub.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.zulaclub.net All nameservers failed to answer the query. [05:10:45] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.89, 3.23, 2.93 [05:11:22] PROBLEM - wikipwaadia.works - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wikipwaadia.works' expires in 15 day(s) (Mon 09 Oct 2023 04:40:35 GMT +0000). [05:12:01] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/affd3cfdf545...df2054273a73 [05:12:03] [02miraheze/ssl] 07MirahezeSSLBot 03df20542 - Bot: Update SSL cert for wikipwaadia.works [05:12:42] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.56, 2.84, 2.82 [05:18:06] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.93, 6.98, 5.98 [05:20:00] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.83, 6.93, 6.09 [05:21:53] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.59, 6.40, 5.99 [05:32:02] PROBLEM - ns1 NTP time on ns1 is WARNING: NTP WARNING: Offset 0.230558902 secs [05:35:53] RECOVERY - wiki.zulaclub.net - reverse DNS on sslhost is OK: SSL OK - wiki.zulaclub.net reverse DNS resolves to cp24.miraheze.org - CNAME OK [05:40:14] RECOVERY - wikipwaadia.works - LetsEncrypt on sslhost is OK: OK - Certificate 'wikipwaadia.works' will expire on Fri 22 Dec 2023 04:11:53 GMT +0000. [06:08:55] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.99, 3.43, 2.82 [06:30:25] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.22, 3.90, 3.83 [06:32:22] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.55, 4.15, 3.93 [07:22:09] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 2.53, 2.93, 3.83 [07:28:09] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.23, 2.56, 3.38 [10:13:43] PROBLEM - betaheze.org - reverse DNS on sslhost is WARNING: Timeout: The DNS operation timed out after 5.41023325920105 seconds [10:43:38] PROBLEM - betaheze.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - betaheze.org reverse DNS resolves to 2607-f1c0-100f-f000-0000-0000-0000-024d.elastic-ssl.ui-r.com [11:32:03] PROBLEM - wiki.bloxotel.tk - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.bloxotel.tk' expires in 15 day(s) (Mon 09 Oct 2023 11:28:57 GMT +0000). [11:32:54] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/df2054273a73...ecb1067c68c0 [11:32:55] [02miraheze/ssl] 07MirahezeSSLBot 03ecb1067 - Bot: Update SSL cert for wiki.bloxotel.tk [12:00:36] RECOVERY - wiki.bloxotel.tk - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.bloxotel.tk' will expire on Fri 22 Dec 2023 10:32:47 GMT +0000. [12:08:09] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.80, 3.01, 2.52 [12:10:09] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.94, 3.66, 2.81 [12:12:09] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 3.19, 3.40, 2.82 [12:16:09] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.71, 3.76, 3.12 [12:18:09] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.09, 3.96, 3.27 [12:20:09] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.90, 3.92, 3.34 [12:34:20] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [12:36:09] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.27, 3.91, 3.64 [12:38:09] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.74, 3.74, 3.60 [12:42:09] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.31, 3.97, 3.73 [12:44:09] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.89, 3.99, 3.77 [12:46:09] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.30, 4.02, 3.80 [12:56:09] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.64, 3.85, 3.86 [13:00:09] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 5.04, 4.05, 3.91 [13:04:09] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.74, 3.95, 3.91 [13:04:19] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['NS.ANKH.FR.eu.org.', 'NS1.eu.org.', 'NS1.ERIOMEM.NET.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [13:06:09] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 5.06, 4.32, 4.05 [13:08:09] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.20, 3.96, 3.96 [13:14:09] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.31, 2.72, 3.37 [14:33:24] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-11 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/080a3bd0a8d2 [14:33:25] [02miraheze/puppet] 07paladox 03080a3bd - varnish: Fix regex for thumb [14:33:28] [02puppet] 07paladox created branch 03paladox-patch-11 - 13https://github.com/miraheze/puppet [14:33:30] [02puppet] 07paladox opened pull request 03#3431: varnish: Fix regex for thumb - 13https://github.com/miraheze/puppet/pull/3431 [14:34:30] [02puppet] 07paladox closed pull request 03#3431: varnish: Fix regex for thumb - 13https://github.com/miraheze/puppet/pull/3431 [14:34:31] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/c6b39d800b3d...80ab24a2525f [14:34:33] [02miraheze/puppet] 07paladox 0380ab24a - varnish: Fix regex for thumb (#3431) [14:34:34] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-11 [14:34:36] [02puppet] 07paladox deleted branch 03paladox-patch-11 - 13https://github.com/miraheze/puppet [14:39:10] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/80ab24a2525f...23842ac26309 [14:39:13] [02miraheze/puppet] 07paladox 0323842ac - varnish: Fix part 2 [14:39:24] PROBLEM - mw131 Current Load on mw131 is CRITICAL: CRITICAL - load average: 20.72, 12.06, 7.89 [14:39:39] PROBLEM - mw134 Current Load on mw134 is CRITICAL: CRITICAL - load average: 19.89, 11.45, 7.07 [14:40:26] PROBLEM - mw132 Current Load on mw132 is CRITICAL: CRITICAL - load average: 13.61, 12.29, 8.33 [14:41:22] PROBLEM - mw131 Current Load on mw131 is WARNING: WARNING - load average: 7.35, 10.48, 7.87 [14:41:39] RECOVERY - mw134 Current Load on mw134 is OK: OK - load average: 5.57, 9.05, 6.74 [14:42:26] RECOVERY - mw132 Current Load on mw132 is OK: OK - load average: 6.50, 10.12, 8.02 [14:43:21] RECOVERY - mw131 Current Load on mw131 is OK: OK - load average: 4.53, 8.49, 7.45 [15:52:02] PROBLEM - ns1 NTP time on ns1 is CRITICAL: NTP CRITICAL: Offset 0.5011648834 secs [15:59:46] PROBLEM - sparabel.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'sparabel.wiki' expires in 14 day(s) (Sat 07 Oct 2023 20:33:09 GMT +0000). [16:09:10] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/23842ac26309...87d8f65e5c82 [16:09:12] [02miraheze/puppet] 07paladox 0387d8f65 - matomo: Set enable_internet_features to 0 [16:12:34] PROBLEM - matomo121 Current Load on matomo121 is CRITICAL: CRITICAL - load average: 5.89, 4.27, 2.71 [16:14:33] PROBLEM - matomo121 Current Load on matomo121 is WARNING: WARNING - load average: 3.18, 3.76, 2.71 [16:16:32] RECOVERY - matomo121 Current Load on matomo121 is OK: OK - load average: 2.69, 3.40, 2.71 [16:29:40] PROBLEM - sparabel.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address sparabel.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [17:03:18] [02mw-config] 07redbluegreenhat synchronize pull request 03#5320: Multiple tasks: Configure LinkTargetParentClass for some wikis - 13https://github.com/miraheze/mw-config/pull/5320 [17:03:43] [02mw-config] 07redbluegreenhat commented on pull request 03#5320: Multiple tasks: Configure LinkTargetParentClass for some wikis - 13https://github.com/miraheze/mw-config/pull/5320#issuecomment-1732365548 [17:04:24] miraheze/mw-config - redbluegreenhat the build passed. [17:11:02] [02mw-config] 07robkam commented on pull request 03#5320: Multiple tasks: Configure LinkTargetParentClass for some wikis - 13https://github.com/miraheze/mw-config/pull/5320#issuecomment-1732367104 [17:12:09] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.72, 3.58, 3.04 [17:50:09] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.68, 3.08, 3.32 [17:54:09] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.90, 3.47, 3.42 [18:00:09] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.03, 3.72, 3.55 [18:02:09] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.27, 3.51, 3.49 [18:06:09] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.63, 3.80, 3.58 [19:14:10] RECOVERY - cloud11 IPMI Sensors on cloud11 is OK: IPMI Status: OK [19:18:10] PROBLEM - cloud11 IPMI Sensors on cloud11 is CRITICAL: IPMI Status: Critical [Cntlr 2 Bay 8 = Critical] [19:20:38] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.69, 6.85, 6.34 [19:22:09] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.22, 3.44, 3.94 [19:22:31] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.37, 7.56, 6.67 [19:24:25] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.45, 7.10, 6.61 [19:28:13] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.84, 6.31, 6.37 [19:29:03] !log [void@phab121] bin/remove destroy @University -- spam account [19:29:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:34:06] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.06, 6.90, 6.23 [19:38:03] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 6.70, 6.59, 6.25 [19:38:09] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.07, 3.37, 3.53 [19:42:09] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.67, 3.68, 3.63 [19:54:29] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-11 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/5e4e17820696 [19:54:31] [02miraheze/puppet] 07paladox 035e4e178 - varnish: Use nginx to connect to backends securely [19:54:32] [02puppet] 07paladox created branch 03paladox-patch-11 - 13https://github.com/miraheze/puppet [19:54:35] [02puppet] 07paladox opened pull request 03#3432: varnish: Use nginx to connect to backends securely - 13https://github.com/miraheze/puppet/pull/3432 [19:54:52] [02puppet] 07paladox synchronize pull request 03#3432: varnish: Use nginx to connect to backends securely - 13https://github.com/miraheze/puppet/pull/3432 [19:54:54] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-11 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/5e4e17820696...b1eaa8c0e13e [19:54:57] [02miraheze/puppet] 07paladox 03b1eaa8c - Update mediawiki.conf [19:55:07] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-11 [+0/-1/±0] 13https://github.com/miraheze/puppet/compare/b1eaa8c0e13e...4916019b798c [19:55:08] [02miraheze/puppet] 07paladox 034916019 - Delete modules/varnish/templates/stunnel.conf [19:55:09] [02puppet] 07paladox synchronize pull request 03#3432: varnish: Use nginx to connect to backends securely - 13https://github.com/miraheze/puppet/pull/3432 [19:55:16] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-11 [+0/-1/±0] 13https://github.com/miraheze/puppet/compare/4916019b798c...02e7bdfb0ac7 [19:55:18] [02miraheze/puppet] 07paladox 0302e7bdf - Delete modules/varnish/templates/initscripts/stunnel4.systemd.erb [19:55:20] [02puppet] 07paladox synchronize pull request 03#3432: varnish: Use nginx to connect to backends securely - 13https://github.com/miraheze/puppet/pull/3432 [19:55:38] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-11 [+0/-1/±0] 13https://github.com/miraheze/puppet/compare/02e7bdfb0ac7...d207b3999e24 [19:55:41] [02miraheze/puppet] 07paladox 03d207b39 - Delete modules/varnish/manifests/stunnel4.pp [19:55:43] [02puppet] 07paladox synchronize pull request 03#3432: varnish: Use nginx to connect to backends securely - 13https://github.com/miraheze/puppet/pull/3432 [19:56:01] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-11 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/d207b3999e24...5f98f1ce804d [19:56:04] [02miraheze/puppet] 07paladox 035f98f1c - Update nginx.pp [19:56:07] [02puppet] 07paladox synchronize pull request 03#3432: varnish: Use nginx to connect to backends securely - 13https://github.com/miraheze/puppet/pull/3432 [19:56:16] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-11 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/5f98f1ce804d...628e803add54 [19:56:17] [02miraheze/puppet] 07paladox 03628e803 - Update init.pp [19:56:20] [02puppet] 07paladox synchronize pull request 03#3432: varnish: Use nginx to connect to backends securely - 13https://github.com/miraheze/puppet/pull/3432 [19:59:12] [02puppet] 07paladox synchronize pull request 03#3432: varnish: Use nginx to connect to backends securely - 13https://github.com/miraheze/puppet/pull/3432 [19:59:13] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-11 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/628e803add54...5abadda8ac88 [19:59:15] [02miraheze/puppet] 07paladox 035abadda - Update mediawiki.conf [20:00:22] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-11 [+0/-1/±0] 13https://github.com/miraheze/puppet/compare/5abadda8ac88...115211e42f87 [20:00:23] [02miraheze/puppet] 07paladox 03115211e - Delete modules/varnish/files/stunnel/stunnel.default [20:00:24] [02puppet] 07paladox synchronize pull request 03#3432: varnish: Use nginx to connect to backends securely - 13https://github.com/miraheze/puppet/pull/3432 [20:00:31] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-11 [+0/-1/±0] 13https://github.com/miraheze/puppet/compare/115211e42f87...2d767fc3f42b [20:00:34] [02miraheze/puppet] 07paladox 032d767fc - Delete modules/varnish/files/stunnel/stunnel4.logrotate.conf [20:00:37] [02puppet] 07paladox synchronize pull request 03#3432: varnish: Use nginx to connect to backends securely - 13https://github.com/miraheze/puppet/pull/3432 [20:02:00] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/d4d08f362e5c...cc768be4692d [20:02:01] [02miraheze/dns] 07paladox 03cc768be - Depool gb [20:02:09] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.05, 3.58, 3.56 [20:03:47] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.11, 6.50, 6.08 [20:04:09] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.71, 3.56, 3.56 [20:06:03] PROBLEM - ns1 SSH on ns1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:06:59] PROBLEM - ns1 NTP time on ns1 is WARNING: NTP WARNING: Offset 0.2490668595 secs [20:07:44] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 10.60, 8.08, 6.72 [20:08:00] RECOVERY - ns1 SSH on ns1 is OK: SSH OK - OpenSSH_9.2p1 Debian-2 (protocol 2.0) [20:09:43] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.59, 7.30, 6.59 [20:10:09] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.54, 3.11, 3.36 [20:11:41] PROBLEM - cp34 Puppet on cp34 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 16 minutes ago with 0 failures [20:11:42] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 4.85, 6.36, 6.32 [20:12:11] PROBLEM - cp25 Puppet on cp25 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 14 minutes ago with 0 failures [20:12:35] PROBLEM - cp35 Puppet on cp35 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 32 minutes ago with 0 failures [20:14:11] RECOVERY - cp25 Puppet on cp25 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [20:15:38] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 11.24, 8.31, 7.05 [20:17:37] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 5.53, 7.09, 6.75 [20:19:36] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 5.02, 6.22, 6.46 [20:19:41] RECOVERY - cp34 Puppet on cp34 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [20:20:35] RECOVERY - cp35 Puppet on cp35 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [20:20:48] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/cc768be4692d...f08a53506a51 [20:20:49] [02miraheze/dns] 07paladox 03f08a535 - Revert "Depool gb" [20:22:27] PROBLEM - cp24 Puppet on cp24 is UNKNOWN: NRPE: Unable to read output [20:23:47] PROBLEM - cp25 Puppet on cp25 is UNKNOWN: NRPE: Unable to read output [20:26:02] PROBLEM - ns1 SSH on ns1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:26:17] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [20:26:24] RECOVERY - cp24 Puppet on cp24 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [20:28:14] PROBLEM - ns1 NTP time on ns1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [20:28:24] PROBLEM - ns1 Current Load on ns1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [20:28:32] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [20:28:34] PROBLEM - ns1 conntrack_table_size on ns1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [20:28:36] PROBLEM - ns1 ferm_active on ns1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [20:29:43] RECOVERY - cp25 Puppet on cp25 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:29:52] PROBLEM - ns1 Auth DNS on ns1 is CRITICAL: CRITICAL - Plugin timed out while executing system call [20:30:00] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 16.24, 11.79, 8.43 [20:30:10] RECOVERY - cloud11 IPMI Sensors on cloud11 is OK: IPMI Status: OK [20:30:19] [02puppet] 07paladox synchronize pull request 03#3432: varnish: Use nginx to connect to backends securely - 13https://github.com/miraheze/puppet/pull/3432 [20:30:22] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-11 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/2d767fc3f42b...e2ae1b04844f [20:30:24] [02miraheze/puppet] 07paladox 03e2ae1b0 - Update init.pp [20:30:28] PROBLEM - ping6 on ns1 is CRITICAL: PING CRITICAL - Packet loss = 100% [20:31:18] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 9.46, 9.07, 7.59 [20:31:35] PROBLEM - ns1 Disk Space on ns1 is CRITICAL: connect to address 2607:f1c0:1800:1f2::1 port 5666: Connection refusedconnect to host 2607:f1c0:1800:1f2::1 port 5666: Connection refused [20:32:07] RECOVERY - ns1 SSH on ns1 is OK: SSH OK - OpenSSH_9.2p1 Debian-2 (protocol 2.0) [20:32:27] RECOVERY - ping6 on ns1 is OK: PING OK - Packet loss = 0%, RTA = 103.97 ms [20:34:01] PROBLEM - cloud11 IPMI Sensors on cloud11 is CRITICAL: IPMI Status: Critical [Cntlr 2 Bay 8 = Critical] [20:46:46] RECOVERY - ns1 APT on ns1 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [20:47:16] RECOVERY - ns1 Current Load on ns1 is OK: LOAD OK - total load average: 0.57, 0.91, 0.53 [20:47:22] PROBLEM - ns1 GDNSD Datacenters on ns1 is UNKNOWN: NRPE: Unable to read output [20:47:24] PROBLEM - ns1 ferm_active on ns1 is UNKNOWN: NRPE: Unable to read output [20:47:25] PROBLEM - ns1 Puppet on ns1 is UNKNOWN: NRPE: Unable to read output [20:47:30] RECOVERY - ns1 conntrack_table_size on ns1 is OK: OK: nf_conntrack is 0 % full [20:47:48] RECOVERY - ns1 Disk Space on ns1 is OK: DISK OK - free space: / 3289MiB (45% inode=83%); [20:47:58] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 6.88, 5.83, 5.14 [20:48:52] RECOVERY - ns1 Auth DNS on ns1 is OK: DNS OK: 0.248 seconds response time. miraheze.org returns 2001:41d0:801:2000::3a18,2001:41d0:801:2000::5d68,51.195.201.140,51.89.139.24 [20:49:19] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [20:49:19] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [20:49:21] RECOVERY - ns1 ferm_active on ns1 is OK: OK ferm input default policy is set [20:49:54] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 5.00, 5.46, 5.09 [20:51:24] [02puppet] 07paladox closed pull request 03#3432: varnish: Use nginx to connect to backends securely - 13https://github.com/miraheze/puppet/pull/3432 [20:51:25] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-5/±3] 13https://github.com/miraheze/puppet/compare/87d8f65e5c82...00c560f366fe [20:51:27] [02miraheze/puppet] 07paladox 0300c560f - varnish: Use nginx to connect to backends securely (#3432) [20:51:29] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-11 [20:51:31] [02puppet] 07paladox deleted branch 03paladox-patch-11 - 13https://github.com/miraheze/puppet [20:54:38] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.77, 7.77, 7.99 [20:56:11] PROBLEM - mw131 Current Load on mw131 is WARNING: WARNING - load average: 11.09, 10.39, 8.62 [20:56:34] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.92, 7.91, 8.00 [20:58:32] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 5.24, 6.96, 7.64 [21:00:06] RECOVERY - mw131 Current Load on mw131 is OK: OK - load average: 5.75, 8.82, 8.45 [21:04:32] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.63, 8.16, 7.97 [21:08:16] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.28, 6.68, 7.74 [21:08:32] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.42, 7.50, 7.78 [21:10:12] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.18, 7.31, 7.85 [21:10:32] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 11.16, 8.41, 8.05 [21:12:12] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.82, 7.19, 7.74 [21:14:53] PROBLEM - matomo121 Puppet on matomo121 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 9 minutes ago with 0 failures [21:16:12] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.48, 7.99, 7.89 [21:16:32] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 4.98, 7.42, 7.89 [21:18:12] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.32, 7.55, 7.73 [21:20:23] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-11 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/b1250a7f75eb [21:20:24] [02miraheze/puppet] 07paladox 03b1250a7 - varnish: Don't set headers in the reverse-proxy that are already set [21:20:27] [02puppet] 07paladox created branch 03paladox-patch-11 - 13https://github.com/miraheze/puppet [21:20:28] [02puppet] 07paladox opened pull request 03#3433: varnish: Don't set headers in the reverse-proxy that are already set - 13https://github.com/miraheze/puppet/pull/3433 [21:21:58] [02puppet] 07paladox synchronize pull request 03#3433: varnish: Don't set headers in the reverse-proxy that are already set - 13https://github.com/miraheze/puppet/pull/3433 [21:22:00] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-11 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/b1250a7f75eb...7868540e71ab [21:22:02] [02miraheze/puppet] 07paladox 037868540 - Update mediawiki.conf [21:23:44] [02puppet] 07paladox closed pull request 03#3433: varnish: Don't set headers in the reverse-proxy that are already set - 13https://github.com/miraheze/puppet/pull/3433 [21:23:46] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/00c560f366fe...9d52832fa0b8 [21:23:47] [02miraheze/puppet] 07paladox 039d52832 - varnish: Don't set headers in the reverse-proxy that are already set (#3433) [21:23:50] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-11 [21:23:52] [02puppet] 07paladox deleted branch 03paladox-patch-11 - 13https://github.com/miraheze/puppet [21:30:03] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.07, 7.44, 7.31 [21:30:27] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-11 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/81af40710806 [21:30:28] [02miraheze/puppet] 07paladox 0381af407 - matomo: Set assume_secure_protocol to 1 [21:30:31] [02puppet] 07paladox created branch 03paladox-patch-11 - 13https://github.com/miraheze/puppet [21:30:33] [02puppet] 07paladox opened pull request 03#3434: matomo: Set assume_secure_protocol to 1 - 13https://github.com/miraheze/puppet/pull/3434 [21:30:37] [02puppet] 07paladox closed pull request 03#3434: matomo: Set assume_secure_protocol to 1 - 13https://github.com/miraheze/puppet/pull/3434 [21:30:41] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/9d52832fa0b8...6396ef8d81e6 [21:30:43] [02miraheze/puppet] 07paladox 036396ef8 - matomo: Set assume_secure_protocol to 1 (#3434) [21:30:45] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-11 [21:30:46] [02puppet] 07paladox deleted branch 03paladox-patch-11 - 13https://github.com/miraheze/puppet [21:31:59] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.14, 6.96, 7.16 [21:32:40] RECOVERY - matomo121 Puppet on matomo121 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [21:35:57] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.43, 6.10, 6.76 [21:36:16] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 4.90, 5.97, 6.63 [21:57:53] RECOVERY - cloud11 IPMI Sensors on cloud11 is OK: IPMI Status: OK [22:01:55] PROBLEM - cloud11 IPMI Sensors on cloud11 is CRITICAL: IPMI Status: Critical [Cntlr 2 Bay 8 = Critical] [23:04:22] PROBLEM - db112 Current Load on db112 is CRITICAL: CRITICAL - load average: 14.18, 8.34, 3.85 [23:08:22] RECOVERY - db112 Current Load on db112 is OK: OK - load average: 1.68, 6.31, 4.21 [23:18:13] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.51, 2.96, 2.41 [23:26:13] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 3.21, 3.18, 2.77