[00:07:14] PROBLEM - matomo121 Current Load on matomo121 is WARNING: WARNING - load average: 3.50, 2.68, 1.75 [00:09:14] RECOVERY - matomo121 Current Load on matomo121 is OK: OK - load average: 3.24, 2.91, 1.95 [00:10:08] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mar.in.ua. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [00:14:37] PROBLEM - cp22 Current Load on cp22 is CRITICAL: LOAD CRITICAL - total load average: 11.57, 6.45, 3.19 [00:15:44] PROBLEM - cp23 Current Load on cp23 is CRITICAL: LOAD CRITICAL - total load average: 7.42, 5.26, 2.65 [00:21:41] RECOVERY - cp23 Current Load on cp23 is OK: LOAD OK - total load average: 0.59, 2.88, 2.52 [00:30:31] PROBLEM - cp22 Current Load on cp22 is WARNING: LOAD WARNING - total load average: 0.68, 1.73, 3.68 [00:32:30] RECOVERY - cp22 Current Load on cp22 is OK: LOAD OK - total load average: 0.91, 1.43, 3.32 [00:35:14] PROBLEM - matomo121 Current Load on matomo121 is CRITICAL: CRITICAL - load average: 5.30, 3.80, 3.10 [00:37:14] PROBLEM - matomo121 Current Load on matomo121 is WARNING: WARNING - load average: 3.82, 3.78, 3.18 [00:39:57] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [00:42:40] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [00:49:14] RECOVERY - matomo121 Current Load on matomo121 is OK: OK - load average: 2.43, 2.99, 3.16 [01:04:00] !log [paladox@mwtask141] finished deploy of {'world': True} to all - SUCCESS in 295s [01:04:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:11:45] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mar.in.ua. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [01:12:27] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns.ankh.fr.eu.org.', 'ns1.eu.org.', 'ns1.eriomem.net.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [01:38:28] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.68, 3.54, 2.85 [01:48:06] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.75, 3.82, 3.42 [01:53:53] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.95, 3.26, 3.30 [02:07:16] PROBLEM - cp33 NTP time on cp33 is CRITICAL: NTP CRITICAL: Offset 0.5288272798 secs [02:09:02] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [02:41:38] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.90, 3.35, 3.07 [02:47:09] PROBLEM - rosettacode.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for rosettacode.org could not be found [02:47:38] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.87, 3.28, 3.14 [03:11:02] PROBLEM - rosettacode.org - LetsEncrypt on sslhost is CRITICAL: connect to address rosettacode.org and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [03:56:28] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.89, 3.50, 3.18 [03:58:24] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.41, 3.02, 3.04 [04:20:40] PROBLEM - wiki.zulaclub.net - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query zulaclub.net. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [04:50:32] RECOVERY - wiki.zulaclub.net - reverse DNS on sslhost is OK: SSL OK - wiki.zulaclub.net reverse DNS resolves to cp23.miraheze.org - CNAME OK [05:15:44] PROBLEM - cp22 NTP time on cp22 is CRITICAL: NTP CRITICAL: Offset -0.5001566112 secs [05:17:19] PROBLEM - cp32 NTP time on cp32 is WARNING: NTP WARNING: Offset -0.1005328596 secs [05:37:53] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [06:07:23] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['NS.ANKH.FR.eu.org.', 'NS1.eu.org.', 'NS1.ERIOMEM.NET.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [06:12:02] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.21, 3.70, 2.97 [06:13:58] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.78, 3.72, 3.06 [06:15:54] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.02, 3.89, 3.21 [06:17:49] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.43, 3.72, 3.22 [06:21:41] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.94, 3.35, 3.18 [06:25:38] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 5.37, 4.09, 3.50 [06:28:26] PROBLEM - cp23 Current Load on cp23 is CRITICAL: LOAD CRITICAL - total load average: 15.65, 7.14, 3.01 [06:29:38] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.37, 3.95, 3.60 [06:32:26] PROBLEM - cp23 Current Load on cp23 is WARNING: LOAD WARNING - total load average: 0.69, 3.63, 2.52 [06:34:26] RECOVERY - cp23 Current Load on cp23 is OK: LOAD OK - total load average: 0.59, 2.59, 2.27 [06:35:38] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.43, 4.14, 3.78 [06:37:38] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.41, 3.86, 3.72 [06:39:38] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.15, 3.88, 3.74 [06:43:38] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.71, 3.93, 3.81 [06:47:38] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.17, 3.98, 3.85 [06:49:38] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.92, 3.89, 3.83 [06:51:38] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 5.06, 4.30, 3.99 [07:03:38] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 2.53, 3.73, 3.93 [07:11:38] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 1.80, 2.50, 3.22 [08:11:04] PROBLEM - kodiak.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'kodiak.wiki' expires in 15 day(s) (Fri 25 Aug 2023 07:41:37 GMT +0000). [08:11:51] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/d69e784e0667...75e351e5cc51 [08:11:52] [02miraheze/ssl] 07MirahezeSSLBot 0375e351e - Bot: Update SSL cert for kodiak.wiki [09:09:10] RECOVERY - kodiak.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'kodiak.wiki' will expire on Tue 07 Nov 2023 07:11:43 GMT +0000. [09:33:49] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [10:03:19] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['NS.ANKH.FR.eu.org.', 'NS1.eu.org.', 'NS1.ERIOMEM.NET.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [10:55:38] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.79, 2.91, 2.31 [10:59:38] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.76, 3.07, 2.54 [11:11:38] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.71, 3.89, 3.22 [11:15:38] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.42, 3.29, 3.14 [11:21:38] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.91, 3.54, 3.29 [11:23:38] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.28, 3.66, 3.35 [11:25:38] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.21, 3.42, 3.30 [11:29:38] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.94, 3.38, 3.32 [11:36:39] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query en.religiononfire.mar.in.ua. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [11:40:22] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.88, 3.44, 3.31 [11:42:18] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.84, 3.20, 3.24 [11:59:26] RECOVERY - cp23 NTP time on cp23 is OK: NTP OK: Offset -0.09926477075 secs [12:04:30] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.66, 3.16, 3.08 [12:05:17] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - en.religiononfire.mar.in.ua All nameservers failed to answer the query. [12:06:26] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.74, 3.64, 3.26 [12:34:39] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query en.religiononfire.mar.in.ua. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [12:44:41] PROBLEM - mw141 MediaWiki Rendering on mw141 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 5903 bytes in 0.009 second response time [12:46:40] PROBLEM - mw141 Current Load on mw141 is CRITICAL: CRITICAL - load average: 20.30, 17.20, 9.43 [12:46:40] RECOVERY - mw141 MediaWiki Rendering on mw141 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.278 second response time [12:50:40] PROBLEM - mw141 Current Load on mw141 is WARNING: WARNING - load average: 4.44, 10.44, 8.48 [12:52:40] RECOVERY - mw141 Current Load on mw141 is OK: OK - load average: 4.09, 8.19, 7.88 [12:53:42] RECOVERY - cp33 NTP time on cp33 is OK: NTP OK: Offset 0.009611964226 secs [13:04:39] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [13:14:37] PROBLEM - wiki.mcsoft.org - reverse DNS on sslhost is WARNING: Timeout: The DNS operation timed out after 5.404881715774536 seconds [13:22:56] PROBLEM - tl.awiki.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query awiki.org. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [13:27:38] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 2.57, 3.20, 3.89 [13:39:38] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.79, 3.57, 3.64 [13:41:38] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.83, 3.71, 3.69 [13:43:27] RECOVERY - wiki.mcsoft.org - reverse DNS on sslhost is OK: SSL OK - wiki.mcsoft.org reverse DNS resolves to cp22.miraheze.org - CNAME OK [13:45:38] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.57, 4.06, 3.83 [13:47:38] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.03, 3.67, 3.71 [13:52:39] RECOVERY - tl.awiki.org - reverse DNS on sslhost is OK: SSL OK - tl.awiki.org reverse DNS resolves to cp22.miraheze.org - CNAME OK [13:53:38] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 1.62, 2.54, 3.22 [16:07:05] PROBLEM - matomo121 Current Load on matomo121 is WARNING: WARNING - load average: 3.73, 2.88, 1.92 [16:09:05] PROBLEM - matomo121 Current Load on matomo121 is CRITICAL: CRITICAL - load average: 4.23, 3.38, 2.22 [16:13:05] PROBLEM - matomo121 Current Load on matomo121 is WARNING: WARNING - load average: 2.95, 3.54, 2.59 [16:17:05] PROBLEM - matomo121 Current Load on matomo121 is CRITICAL: CRITICAL - load average: 4.32, 3.75, 2.88 [16:19:05] PROBLEM - matomo121 Current Load on matomo121 is WARNING: WARNING - load average: 2.53, 3.44, 2.89 [16:23:05] PROBLEM - matomo121 Current Load on matomo121 is CRITICAL: CRITICAL - load average: 4.88, 4.02, 3.23 [16:27:06] PROBLEM - matomo121 Current Load on matomo121 is WARNING: WARNING - load average: 2.31, 3.47, 3.23 [16:31:06] PROBLEM - matomo121 Current Load on matomo121 is CRITICAL: CRITICAL - load average: 4.71, 3.87, 3.44 [16:33:05] PROBLEM - matomo121 Current Load on matomo121 is WARNING: WARNING - load average: 3.05, 3.52, 3.36 [16:45:06] RECOVERY - matomo121 Current Load on matomo121 is OK: OK - load average: 2.73, 3.24, 3.40 [17:19:38] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.92, 3.26, 2.75 [17:21:38] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.79, 3.09, 2.75 [17:29:38] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.38, 3.52, 3.12 [17:31:38] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.45, 3.05, 2.99 [17:36:39] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query en.religiononfire.mar.in.ua. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [17:53:15] PROBLEM - wiki.altilunium.my.id - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': None, 'CNAME': None} [17:59:30] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [18:00:01] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-10 [+1/-0/±0] 13https://github.com/miraheze/puppet/commit/7bc959c5ef0b [18:00:04] [02miraheze/puppet] 07paladox 037bc959c - ssl: add a all_certs module [18:00:05] [02puppet] 07paladox created branch 03paladox-patch-10 - 13https://github.com/miraheze/puppet [18:00:07] [02puppet] 07paladox opened pull request 03#3371: ssl: add a all_certs module - 13https://github.com/miraheze/puppet/pull/3371 [18:01:15] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-10 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/7bc959c5ef0b...b3431be7a599 [18:01:16] [02miraheze/puppet] 07paladox 03b3431be - Update nginx.pp [18:01:18] [02puppet] 07paladox synchronize pull request 03#3371: ssl: add a all_certs module - 13https://github.com/miraheze/puppet/pull/3371 [18:05:17] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [18:05:42] PROBLEM - cp33 NTP time on cp33 is WARNING: NTP WARNING: Offset 0.2380824089 secs [18:06:13] PROBLEM - cp33 Puppet on cp33 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 12 minutes ago with 0 failures [18:06:36] PROBLEM - cp32 Puppet on cp32 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 30 minutes ago with 0 failures [18:07:31] PROBLEM - cp22 Puppet on cp22 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 24 minutes ago with 0 failures [18:08:17] PROBLEM - cp23 Puppet on cp23 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [18:09:38] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.64, 3.49, 2.80 [18:11:38] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.43, 3.47, 2.89 [18:15:38] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.22, 3.67, 3.09 [18:17:38] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.12, 3.50, 3.11 [18:19:37] PROBLEM - jobchron121 Puppet on jobchron121 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 3 minutes ago with 0 failures [18:19:46] PROBLEM - db121 Puppet on db121 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 30 minutes ago with 0 failures [18:19:46] PROBLEM - db131 Puppet on db131 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 20 minutes ago with 0 failures [18:19:51] PROBLEM - bast141 Puppet on bast141 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 21 minutes ago with 0 failures [18:19:54] PROBLEM - db142 Puppet on db142 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 9 minutes ago with 0 failures [18:20:00] PROBLEM - mw141 Puppet on mw141 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 13 minutes ago with 0 failures [18:20:04] PROBLEM - ns2 Puppet on ns2 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 2 minutes ago with 0 failures [18:20:07] PROBLEM - db101 Puppet on db101 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 29 minutes ago with 0 failures [18:20:09] PROBLEM - swiftproxy111 Puppet on swiftproxy111 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 8 minutes ago with 0 failures [18:20:11] PROBLEM - cloud14 Puppet on cloud14 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 29 minutes ago with 0 failures [18:20:11] PROBLEM - puppetdb121 Puppet on puppetdb121 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 12 minutes ago with 0 failures [18:20:21] PROBLEM - bast121 Puppet on bast121 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 10 minutes ago with 0 failures [18:20:21] PROBLEM - mw134 Puppet on mw134 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 13 minutes ago with 0 failures [18:20:23] PROBLEM - cloud11 Puppet on cloud11 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 30 minutes ago with 0 failures [18:20:31] PROBLEM - swiftobject113 Puppet on swiftobject113 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 6 minutes ago with 0 failures [18:20:31] PROBLEM - ns1 Puppet on ns1 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 24 minutes ago with 0 failures [18:20:38] PROBLEM - mem131 Puppet on mem131 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 26 minutes ago with 0 failures [18:20:38] PROBLEM - mw132 Puppet on mw132 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 14 minutes ago with 0 failures [18:20:39] PROBLEM - mw131 Puppet on mw131 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 9 minutes ago with 0 failures [18:20:41] PROBLEM - swiftproxy131 Puppet on swiftproxy131 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 14 minutes ago with 0 failures [18:20:41] PROBLEM - db112 Puppet on db112 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 19 minutes ago with 0 failures [18:20:41] PROBLEM - swiftac111 Puppet on swiftac111 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 27 minutes ago with 0 failures [18:20:43] PROBLEM - mw142 Puppet on mw142 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 19 minutes ago with 0 failures [18:20:49] PROBLEM - cloud10 Puppet on cloud10 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 12 minutes ago with 0 failures [18:20:50] PROBLEM - graylog131 Puppet on graylog131 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 18 minutes ago with 0 failures [18:21:01] PROBLEM - puppet141 Puppet on puppet141 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 12 minutes ago with 0 failures [18:21:03] PROBLEM - mon141 Puppet on mon141 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 28 minutes ago with 0 failures [18:21:04] PROBLEM - swiftobject111 Puppet on swiftobject111 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 16 minutes ago with 0 failures [18:21:04] PROBLEM - mwtask141 Puppet on mwtask141 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 10 minutes ago with 0 failures [18:21:09] PROBLEM - cloud12 Puppet on cloud12 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 12 minutes ago with 0 failures [18:21:09] PROBLEM - mail121 Puppet on mail121 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 8 minutes ago with 0 failures [18:21:09] PROBLEM - reports121 Puppet on reports121 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 24 minutes ago with 0 failures [18:21:11] PROBLEM - swiftobject101 Puppet on swiftobject101 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 20 minutes ago with 0 failures [18:21:12] PROBLEM - swiftobject112 Puppet on swiftobject112 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 5 minutes ago with 0 failures [18:21:17] PROBLEM - prometheus131 Puppet on prometheus131 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 28 minutes ago with 0 failures [18:21:18] PROBLEM - mem141 Puppet on mem141 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 7 minutes ago with 0 failures [18:21:21] PROBLEM - ldap141 Puppet on ldap141 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 18 minutes ago with 0 failures [18:21:22] PROBLEM - mw143 Puppet on mw143 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 16 minutes ago with 0 failures [18:21:22] PROBLEM - os141 Puppet on os141 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 13 minutes ago with 0 failures [18:21:23] PROBLEM - os131 Puppet on os131 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 7 minutes ago with 0 failures [18:21:25] PROBLEM - swiftobject122 Puppet on swiftobject122 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 32 minutes ago with 0 failures [18:21:26] PROBLEM - matomo121 Puppet on matomo121 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 15 minutes ago with 0 failures [18:21:26] PROBLEM - phab121 Puppet on phab121 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 12 minutes ago with 0 failures [18:21:30] PROBLEM - cloud13 Puppet on cloud13 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 31 minutes ago with 0 failures [18:21:32] PROBLEM - test131 Puppet on test131 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 24 minutes ago with 0 failures [18:21:34] PROBLEM - mw133 Puppet on mw133 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 21 minutes ago with 0 failures [18:21:38] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.21, 3.79, 3.31 [18:21:47] PROBLEM - swiftobject121 Puppet on swiftobject121 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 32 minutes ago with 0 failures [18:23:38] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.66, 3.64, 3.31 [18:27:38] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.82, 3.31, 3.25 [18:29:30] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['NS.ANKH.FR.eu.org.', 'NS1.eu.org.', 'NS1.ERIOMEM.NET.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [18:32:17] RECOVERY - cp23 Puppet on cp23 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:33:38] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.52, 3.51, 3.36 [18:41:27] PROBLEM - wiki.altilunium.my.id - LetsEncrypt on sslhost is CRITICAL: connect to address wiki.altilunium.my.id and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [18:41:38] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 3.09, 3.34, 3.35 [18:44:33] PROBLEM - ndgkb.nena.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'ndgkb.nena.org' expires in 15 day(s) (Fri 25 Aug 2023 18:32:28 GMT +0000). [18:45:38] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.64, 3.50, 3.41 [18:46:33] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/75e351e5cc51...3cc753373cd5 [18:46:36] [02miraheze/ssl] 07MirahezeSSLBot 033cc7533 - Bot: Update SSL cert for ndgkb.nena.org [18:47:38] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 3.19, 3.36, 3.37 [18:49:32] [02puppet] 07paladox synchronize pull request 03#3371: ssl: add a all_certs module - 13https://github.com/miraheze/puppet/pull/3371 [18:49:35] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-10 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/b3431be7a599...31acdf24aa66 [18:49:38] [02miraheze/puppet] 07paladox 0331acdf2 - Update all_certs.pp [18:51:31] [02puppet] 07paladox synchronize pull request 03#3371: ssl: add a all_certs module - 13https://github.com/miraheze/puppet/pull/3371 [18:51:34] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-10 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/31acdf24aa66...874b59c31ad2 [18:51:36] [02miraheze/puppet] 07paladox 03874b59c - Update all_certs.pp [18:55:00] PROBLEM - kb.nena.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'kb.nena.org' expires in 15 day(s) (Fri 25 Aug 2023 18:37:16 GMT +0000). [18:55:38] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.19, 3.59, 3.44 [18:57:38] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.94, 3.64, 3.48 [19:02:56] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/3cc753373cd5...8017f0710063 [19:02:57] [02miraheze/ssl] 07MirahezeSSLBot 038017f07 - Bot: Update SSL cert for kb.nena.org [19:05:38] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.41, 3.05, 3.29 [19:15:38] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.57, 3.32, 3.28 [19:17:38] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 3.25, 3.34, 3.29 [19:23:26] [02miraheze/ssl] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-1/±0] 13https://github.com/miraheze/ssl/commit/70fbbde949c1 [19:23:28] [02miraheze/ssl] 07paladox 0370fbbde - Remove unused wiki.thesimswiki.com cert [19:23:30] [02ssl] 07paladox created branch 03paladox-patch-1 - 13https://github.com/miraheze/ssl [19:23:33] [02ssl] 07paladox opened pull request 03#714: Remove unused wiki.thesimswiki.com cert - 13https://github.com/miraheze/ssl/pull/714 [19:23:35] [02ssl] 07paladox closed pull request 03#714: Remove unused wiki.thesimswiki.com cert - 13https://github.com/miraheze/ssl/pull/714 [19:23:38] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.08, 3.69, 3.46 [19:23:38] [02miraheze/ssl] 07paladox pushed 031 commit to 03master [+0/-1/±0] 13https://github.com/miraheze/ssl/compare/8017f0710063...cf1c75500778 [19:23:39] [02miraheze/ssl] 07paladox 03cf1c755 - Remove unused wiki.thesimswiki.com cert (#714) [19:23:42] [02miraheze/ssl] 07paladox deleted branch 03paladox-patch-1 [19:23:44] [02ssl] 07paladox deleted branch 03paladox-patch-1 - 13https://github.com/miraheze/ssl [19:25:38] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.69, 3.66, 3.47 [19:29:38] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 3.07, 3.33, 3.37 [19:53:03] RECOVERY - kb.nena.org - LetsEncrypt on sslhost is OK: OK - Certificate 'kb.nena.org' will expire on Tue 07 Nov 2023 18:02:48 GMT +0000. [19:57:53] [02miraheze/phabricator-extensions] 07paladox pushed 031 commit to 03paladox-patch-1 [+1/-0/±0] 13https://github.com/miraheze/phabricator-extensions/commit/e75fb9fa0d5c [19:57:55] [02miraheze/phabricator-extensions] 07paladox 03e75fb9f - Add Wikimedia workflow tools (Rollback) [19:57:57] [02phabricator-extensions] 07paladox created branch 03paladox-patch-1 - 13https://github.com/miraheze/phabricator-extensions [19:57:58] [02phabricator-extensions] 07paladox opened pull request 03#37: Add Wikimedia workflow tools (Rollback) - 13https://github.com/miraheze/phabricator-extensions/pull/37 [19:59:34] [02miraheze/phabricator-extensions] 07paladox pushed 031 commit to 03paladox-patch-1 [+1/-0/±0] 13https://github.com/miraheze/phabricator-extensions/compare/e75fb9fa0d5c...2e554ffeec64 [19:59:37] [02miraheze/phabricator-extensions] 07paladox 032e554ff - Create MFRollbackWorkflow.php [19:59:39] [02phabricator-extensions] 07paladox synchronize pull request 03#37: Add Wikimedia workflow tools (Rollback) - 13https://github.com/miraheze/phabricator-extensions/pull/37 [20:01:06] [02miraheze/phabricator-extensions] 07paladox pushed 031 commit to 03paladox-patch-1 [+1/-0/±0] 13https://github.com/miraheze/phabricator-extensions/compare/2e554ffeec64...5a924b8532e9 [20:01:09] [02miraheze/phabricator-extensions] 07paladox 035a924b8 - Create rollback.php [20:01:11] [02phabricator-extensions] 07paladox synchronize pull request 03#37: Add Wikimedia workflow tools (Rollback) - 13https://github.com/miraheze/phabricator-extensions/pull/37 [20:02:00] [02miraheze/phabricator-extensions] 07paladox pushed 031 commit to 03paladox-patch-1 [+1/-0/±0] 13https://github.com/miraheze/phabricator-extensions/compare/5a924b8532e9...f479ba17e06d [20:02:01] [02miraheze/phabricator-extensions] 07paladox 03f479ba1 - Create init-script.php [20:02:02] [02phabricator-extensions] 07paladox synchronize pull request 03#37: Add Wikimedia workflow tools (Rollback) - 13https://github.com/miraheze/phabricator-extensions/pull/37 [20:02:11] [02phabricator-extensions] 07paladox edited pull request 03#37: Add Wikimedia workflow tools (Rollback) - 13https://github.com/miraheze/phabricator-extensions/pull/37 [20:02:29] [02phabricator-extensions] 07paladox synchronize pull request 03#37: Add Wikimedia workflow tools (Rollback) - 13https://github.com/miraheze/phabricator-extensions/pull/37 [20:02:32] [02miraheze/phabricator-extensions] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/phabricator-extensions/compare/f479ba17e06d...c1d44ab77217 [20:02:34] [02miraheze/phabricator-extensions] 07paladox 03c1d44ab - Update RollbackTransactionsWorkflow.php [20:02:38] [02phabricator-extensions] 07paladox synchronize pull request 03#37: Add Wikimedia workflow tools (Rollback) - 13https://github.com/miraheze/phabricator-extensions/pull/37 [20:02:40] [02miraheze/phabricator-extensions] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/phabricator-extensions/compare/c1d44ab77217...50f4b0697b85 [20:02:43] [02miraheze/phabricator-extensions] 07paladox 0350f4b06 - Update MFRollbackWorkflow.php [20:02:52] [02phabricator-extensions] 07paladox synchronize pull request 03#37: Add Wikimedia workflow tools (Rollback) - 13https://github.com/miraheze/phabricator-extensions/pull/37 [20:02:54] [02miraheze/phabricator-extensions] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/phabricator-extensions/compare/50f4b0697b85...12b395f98a1b [20:02:57] [02miraheze/phabricator-extensions] 07paladox 0312b395f - Update init-script.php [20:04:04] paladox: ircrcbot needs a restart [20:04:14] ok [20:04:34] done [20:13:29] RECOVERY - ndgkb.nena.org - LetsEncrypt on sslhost is OK: OK - Certificate 'ndgkb.nena.org' will expire on Tue 07 Nov 2023 17:46:25 GMT +0000. [20:13:50] PROBLEM - phab121 phab.miraheze.wiki HTTPS on phab121 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/1.1 500 Internal Server Error [20:13:53] PROBLEM - phab121 phabricator.miraheze.org HTTPS on phab121 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 519 bytes in 0.047 second response time [20:23:13] phab broken seems related to phab-exts change [20:23:45] or at least is something with phab-exts [20:25:50] RECOVERY - phab121 phab.miraheze.wiki HTTPS on phab121 is OK: HTTP OK: Status line output matched "HTTP/1.1 200" - 17684 bytes in 0.103 second response time [20:25:53] RECOVERY - phab121 phabricator.miraheze.org HTTPS on phab121 is OK: HTTP OK: HTTP/1.1 200 OK - 20261 bytes in 0.145 second response time [20:32:26] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.59, 3.39, 2.87 [20:39:51] [02miraheze/phabricator-extensions] 07paladox pushed 031 commit to 03paladox-patch-1 [+1/-0/±0] 13https://github.com/miraheze/phabricator-extensions/compare/12b395f98a1b...0c7b36bbe36e [20:39:53] [02miraheze/phabricator-extensions] 07paladox 030c7b36b - Re-arrange [20:39:55] [02phabricator-extensions] 07paladox synchronize pull request 03#37: Add Wikimedia workflow tools (Rollback) - 13https://github.com/miraheze/phabricator-extensions/pull/37 [20:40:09] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 3.13, 3.39, 3.10 [20:40:28] [02miraheze/phabricator-extensions] 07paladox pushed 031 commit to 03paladox-patch-1 [+16/-11/±0] 13https://github.com/miraheze/phabricator-extensions/compare/0c7b36bbe36e...b957565a11cd [20:40:30] [02miraheze/phabricator-extensions] 07paladox 03b957565 - Re-arrange [20:40:32] [02phabricator-extensions] 07paladox synchronize pull request 03#37: Add Wikimedia workflow tools (Rollback) - 13https://github.com/miraheze/phabricator-extensions/pull/37 [20:40:47] miraheze/phabricator-extensions - paladox the build has errored. [20:40:50] [02miraheze/phabricator-extensions] 07paladox pushed 031 commit to 03paladox-patch-1 [+15/-11/±0] 13https://github.com/miraheze/phabricator-extensions/compare/b957565a11cd...2ae82720f6e6 [20:40:52] [02miraheze/phabricator-extensions] 07paladox 032ae8272 - Re-arrange [20:40:54] [02phabricator-extensions] 07paladox synchronize pull request 03#37: Add Wikimedia workflow tools (Rollback) - 13https://github.com/miraheze/phabricator-extensions/pull/37 [20:41:18] miraheze/phabricator-extensions - paladox the build has errored. [20:41:43] miraheze/phabricator-extensions - paladox the build has errored. [20:51:45] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.60, 3.52, 3.30 [21:01:38] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 3.12, 3.40, 3.35 [21:04:39] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-11 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/329e587c2ef6 [21:04:42] [02miraheze/puppet] 07paladox 03329e587 - phabricator: Install extension in /srv/phab/libext [21:04:44] [02puppet] 07paladox created branch 03paladox-patch-11 - 13https://github.com/miraheze/puppet [21:04:47] [02puppet] 07paladox opened pull request 03#3372: phabricator: Install extension in /srv/phab/libext - 13https://github.com/miraheze/puppet/pull/3372 [21:08:43] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-11 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/329e587c2ef6...a0f7c04b96b4 [21:08:44] [02miraheze/puppet] 07paladox 03a0f7c04 - Update config.yaml [21:08:45] [02puppet] 07paladox synchronize pull request 03#3372: phabricator: Install extension in /srv/phab/libext - 13https://github.com/miraheze/puppet/pull/3372 [21:11:27] RECOVERY - phab121 Puppet on phab121 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [21:15:26] PROBLEM - phab121 Puppet on phab121 is UNKNOWN: NRPE: Unable to read output [21:18:19] [02miraheze/phabricator-extensions] 07paladox pushed 031 commit to 03paladox-patch-1 [+16/-11/±0] 13https://github.com/miraheze/phabricator-extensions/compare/2ae82720f6e6...4d765349fb59 [21:18:21] [02miraheze/phabricator-extensions] 07paladox 034d76534 - Re-arrange [21:18:23] [02phabricator-extensions] 07paladox synchronize pull request 03#37: Add Wikimedia workflow tools (Rollback) - 13https://github.com/miraheze/phabricator-extensions/pull/37 [21:18:58] [02miraheze/phabricator-extensions] 07paladox pushed 031 commit to 03paladox-patch-1 [+14/-11/±0] 13https://github.com/miraheze/phabricator-extensions/compare/4d765349fb59...6217bd43b6cb [21:19:00] [02miraheze/phabricator-extensions] 07paladox 036217bd4 - Re-arrange [21:19:03] [02phabricator-extensions] 07paladox synchronize pull request 03#37: Add Wikimedia workflow tools (Rollback) - 13https://github.com/miraheze/phabricator-extensions/pull/37 [21:19:18] miraheze/phabricator-extensions - paladox the build has errored. [21:19:53] [02miraheze/phabricator-extensions] 07paladox pushed 031 commit to 03paladox-patch-1 [+14/-11/±0] 13https://github.com/miraheze/phabricator-extensions/compare/6217bd43b6cb...0d7a97616f06 [21:19:55] miraheze/phabricator-extensions - paladox the build has errored. [21:19:55] [02miraheze/phabricator-extensions] 07paladox 030d7a976 - Re-arrange [21:19:58] [02phabricator-extensions] 07paladox synchronize pull request 03#37: Add Wikimedia workflow tools (Rollback) - 13https://github.com/miraheze/phabricator-extensions/pull/37 [21:20:50] miraheze/phabricator-extensions - paladox the build has errored. [21:21:13] PROBLEM - wiki.altilunium.my.id - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.altilunium.my.id All nameservers failed to answer the query. [21:24:57] [02miraheze/phabricator-extensions] 07paladox pushed 031 commit to 03paladox-patch-1 [+14/-11/±1] 13https://github.com/miraheze/phabricator-extensions/compare/0d7a97616f06...8693952d00f7 [21:25:00] [02miraheze/phabricator-extensions] 07paladox 038693952 - Re-arrange [21:25:01] [02phabricator-extensions] 07paladox synchronize pull request 03#37: Add Wikimedia workflow tools (Rollback) - 13https://github.com/miraheze/phabricator-extensions/pull/37 [21:25:53] miraheze/phabricator-extensions - paladox the build has errored. [21:26:00] PROBLEM - kb.nena.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'kb.nena.org' expires in 15 day(s) (Fri 25 Aug 2023 18:37:16 GMT +0000). [21:31:45] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/cf1c75500778...a754d97388ba [21:31:48] [02miraheze/ssl] 07MirahezeSSLBot 03a754d97 - Bot: Update SSL cert for kb.nena.org [21:39:28] RECOVERY - phab121 Puppet on phab121 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [21:40:36] [02miraheze/phabricator-extensions] 07paladox pushed 031 commit to 03paladox-patch-1 [+14/-11/±2] 13https://github.com/miraheze/phabricator-extensions/compare/8693952d00f7...3bb38ba32d41 [21:40:37] [02miraheze/phabricator-extensions] 07paladox 033bb38ba - Re-arrange [21:40:39] [02phabricator-extensions] 07paladox synchronize pull request 03#37: Add Wikimedia workflow tools (Rollback) - 13https://github.com/miraheze/phabricator-extensions/pull/37 [21:41:13] [02miraheze/phabricator-extensions] 07paladox pushed 031 commit to 03paladox-patch-1 [+14/-11/±2] 13https://github.com/miraheze/phabricator-extensions/compare/3bb38ba32d41...212b0d4725ec [21:41:15] [02miraheze/phabricator-extensions] 07paladox 03212b0d4 - Re-arrange [21:41:17] [02phabricator-extensions] 07paladox synchronize pull request 03#37: Add Wikimedia workflow tools (Rollback) - 13https://github.com/miraheze/phabricator-extensions/pull/37 [21:41:26] miraheze/phabricator-extensions - paladox the build has errored. [21:42:06] miraheze/phabricator-extensions - paladox the build has errored. [21:43:26] PROBLEM - cp23 NTP time on cp23 is WARNING: NTP WARNING: Offset 0.1011487246 secs [21:45:26] [02miraheze/phabricator-extensions] 07paladox pushed 031 commit to 03paladox-patch-1 [+14/-11/±2] 13https://github.com/miraheze/phabricator-extensions/compare/212b0d4725ec...5401769ce40d [21:45:29] [02miraheze/phabricator-extensions] 07paladox 035401769 - Re-arrange [21:45:31] [02phabricator-extensions] 07paladox synchronize pull request 03#37: Add Wikimedia workflow tools (Rollback) - 13https://github.com/miraheze/phabricator-extensions/pull/37 [21:46:14] miraheze/phabricator-extensions - paladox the build has errored. [21:46:21] PROBLEM - ndgkb.nena.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'ndgkb.nena.org' expires in 15 day(s) (Fri 25 Aug 2023 18:32:28 GMT +0000). [21:47:44] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/a754d97388ba...77d0f3a43a82 [21:47:45] [02miraheze/ssl] 07MirahezeSSLBot 0377d0f3a - Bot: Update SSL cert for ndgkb.nena.org [21:49:22] [02miraheze/phabricator-extensions] 07paladox pushed 031 commit to 03paladox-patch-1 [+14/-11/±2] 13https://github.com/miraheze/phabricator-extensions/compare/5401769ce40d...0f013f9361a4 [21:49:25] [02miraheze/phabricator-extensions] 07paladox 030f013f9 - Re-arrange [21:49:28] [02phabricator-extensions] 07paladox synchronize pull request 03#37: Add Wikimedia workflow tools (Rollback) - 13https://github.com/miraheze/phabricator-extensions/pull/37 [21:49:43] [02miraheze/phabricator-extensions] 07github-actions[bot] pushed 031 commit to 03paladox-patch-1 [+0/-0/±3] 13https://github.com/miraheze/phabricator-extensions/compare/0f013f9361a4...608dcdbd1f6d [21:49:44] [02miraheze/phabricator-extensions] 07github-actions 03608dcdb - CI: lint code to MediaWiki standards [21:49:46] [02phabricator-extensions] 07github-actions[bot] synchronize pull request 03#37: Add Wikimedia workflow tools (Rollback) - 13https://github.com/miraheze/phabricator-extensions/pull/37 [21:50:15] miraheze/phabricator-extensions - paladox the build passed. [21:51:11] PROBLEM - wiki.altilunium.my.id - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.altilunium.my.id could not be found [21:52:31] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.71, 3.58, 3.16 [21:52:50] [02phabricator-extensions] 07paladox closed pull request 03#37: Add Wikimedia workflow tools (Rollback) - 13https://github.com/miraheze/phabricator-extensions/pull/37 [21:52:51] [02miraheze/phabricator-extensions] 07paladox pushed 031 commit to 03master [+16/-9/±1] 13https://github.com/miraheze/phabricator-extensions/compare/cb2860ae0bb0...eb71d83e2066 [21:52:52] [02miraheze/phabricator-extensions] 07paladox 03eb71d83 - Add Wikimedia workflow tools (Rollback) (#37) [21:52:57] [02puppet] 07paladox closed pull request 03#3372: phabricator: Install extension in /srv/phab/libext - 13https://github.com/miraheze/puppet/pull/3372 [21:53:00] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/puppet/compare/c3b55346d12b...ea11bca2b860 [21:53:03] [02miraheze/puppet] 07paladox 03ea11bca - phabricator: Install extension in /srv/phab/libext (#3372) [21:53:06] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-11 [21:53:08] [02puppet] 07paladox deleted branch 03paladox-patch-11 - 13https://github.com/miraheze/puppet [21:53:40] miraheze/phabricator-extensions - paladox the build passed. [21:54:25] [02miraheze/puppet] 07paladox pushed 034 commits to 03paladox-patch-10 [+1/-0/±3] 13https://github.com/miraheze/puppet/compare/874b59c31ad2...5e32e7639eb4 [21:54:28] [02miraheze/puppet] 07paladox 030205148 - ssl: add a all_certs module [21:54:31] [02miraheze/puppet] 07paladox 035a67ac0 - Update nginx.pp [21:54:32] [02miraheze/puppet] 07paladox 03be59c19 - Update all_certs.pp [21:54:34] [02miraheze/puppet] 07paladox 035e32e76 - Update all_certs.pp [21:54:37] [02puppet] 07paladox synchronize pull request 03#3371: ssl: add a all_certs module - 13https://github.com/miraheze/puppet/pull/3371 [21:58:18] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 3.01, 3.36, 3.21 [21:59:43] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-10 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/5e32e7639eb4...caf7dde8f81e [21:59:44] [02miraheze/puppet] 07paladox 03caf7dde - Update ssl.pp [21:59:45] [02puppet] 07paladox synchronize pull request 03#3371: ssl: add a all_certs module - 13https://github.com/miraheze/puppet/pull/3371 [22:00:31] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-10 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/caf7dde8f81e...60d0b64a8582 [22:00:34] [02miraheze/puppet] 07paladox 0360d0b64 - Update cert.pp [22:00:35] [02puppet] 07paladox synchronize pull request 03#3371: ssl: add a all_certs module - 13https://github.com/miraheze/puppet/pull/3371 [22:00:53] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-10 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/60d0b64a8582...b3185051a0d5 [22:00:55] [02miraheze/puppet] 07paladox 03b318505 - Update wildcard.pp [22:00:58] [02puppet] 07paladox synchronize pull request 03#3371: ssl: add a all_certs module - 13https://github.com/miraheze/puppet/pull/3371 [22:01:08] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-10 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/b3185051a0d5...f773c3fac1f1 [22:01:10] [02miraheze/puppet] 07paladox 03f773c3f - Update certs.pp [22:01:13] [02puppet] 07paladox synchronize pull request 03#3371: ssl: add a all_certs module - 13https://github.com/miraheze/puppet/pull/3371 [22:05:05] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.61, 3.48, 3.29 [22:07:01] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 3.29, 3.35, 3.27 [22:10:54] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.26, 3.55, 3.39 [22:12:50] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 1.71, 2.99, 3.21 [22:15:59] RECOVERY - ndgkb.nena.org - LetsEncrypt on sslhost is OK: OK - Certificate 'ndgkb.nena.org' will expire on Tue 07 Nov 2023 20:47:37 GMT +0000. [22:19:03] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-10 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/f773c3fac1f1...e40bab93fd87 [22:19:05] [02miraheze/puppet] 07paladox 03e40bab9 - Update nginx.pp [22:19:07] [02puppet] 07paladox synchronize pull request 03#3371: ssl: add a all_certs module - 13https://github.com/miraheze/puppet/pull/3371 [22:21:31] RECOVERY - cp22 Puppet on cp22 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [22:26:36] RECOVERY - cp32 Puppet on cp32 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [22:26:43] RECOVERY - cp33 Puppet on cp33 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [22:29:04] RECOVERY - mwtask141 Puppet on mwtask141 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [22:33:56] [02puppet] 07paladox closed pull request 03#3371: ssl: add a all_certs module - 13https://github.com/miraheze/puppet/pull/3371 [22:33:57] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+1/-0/±6] 13https://github.com/miraheze/puppet/compare/ea11bca2b860...dbc12c40d41a [22:33:58] [02miraheze/puppet] 07paladox 03dbc12c4 - ssl: add a all_certs module (#3371) [22:34:01] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-10 [22:34:03] [02puppet] 07paladox deleted branch 03paladox-patch-10 - 13https://github.com/miraheze/puppet [22:34:21] RECOVERY - bast121 Puppet on bast121 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [22:34:23] RECOVERY - cloud11 Puppet on cloud11 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [22:34:31] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [22:34:36] RECOVERY - swiftac111 Puppet on swiftac111 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [22:34:38] RECOVERY - mem131 Puppet on mem131 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [22:34:38] RECOVERY - mw132 Puppet on mw132 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [22:34:39] RECOVERY - mw131 Puppet on mw131 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [22:34:41] RECOVERY - swiftproxy131 Puppet on swiftproxy131 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [22:34:41] RECOVERY - db112 Puppet on db112 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [22:34:42] RECOVERY - mw142 Puppet on mw142 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [22:34:49] RECOVERY - cloud10 Puppet on cloud10 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [22:34:50] RECOVERY - graylog131 Puppet on graylog131 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [22:35:01] RECOVERY - puppet141 Puppet on puppet141 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [22:35:03] RECOVERY - mon141 Puppet on mon141 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [22:35:04] RECOVERY - swiftobject111 Puppet on swiftobject111 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [22:35:09] RECOVERY - cloud12 Puppet on cloud12 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [22:35:09] RECOVERY - mail121 Puppet on mail121 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [22:35:10] RECOVERY - reports121 Puppet on reports121 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [22:35:11] RECOVERY - swiftobject101 Puppet on swiftobject101 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [22:35:14] RECOVERY - swiftobject112 Puppet on swiftobject112 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [22:35:17] RECOVERY - prometheus131 Puppet on prometheus131 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [22:35:18] RECOVERY - mem141 Puppet on mem141 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [22:35:20] RECOVERY - ldap141 Puppet on ldap141 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [22:35:21] RECOVERY - mw143 Puppet on mw143 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [22:35:22] RECOVERY - os141 Puppet on os141 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [22:35:23] RECOVERY - os131 Puppet on os131 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [22:35:25] RECOVERY - swiftobject122 Puppet on swiftobject122 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [22:35:26] RECOVERY - matomo121 Puppet on matomo121 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [22:35:30] RECOVERY - cloud13 Puppet on cloud13 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [22:35:32] RECOVERY - test131 Puppet on test131 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [22:35:34] RECOVERY - mw133 Puppet on mw133 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [22:35:37] RECOVERY - jobchron121 Puppet on jobchron121 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [22:35:46] RECOVERY - db121 Puppet on db121 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [22:35:46] RECOVERY - db131 Puppet on db131 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [22:35:47] RECOVERY - swiftobject121 Puppet on swiftobject121 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [22:35:50] RECOVERY - bast141 Puppet on bast141 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [22:35:54] RECOVERY - db142 Puppet on db142 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [22:36:00] RECOVERY - mw141 Puppet on mw141 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [22:36:04] RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [22:36:07] RECOVERY - db101 Puppet on db101 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [22:36:09] RECOVERY - swiftproxy111 Puppet on swiftproxy111 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [22:36:11] RECOVERY - cloud14 Puppet on cloud14 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [22:36:11] RECOVERY - puppetdb121 Puppet on puppetdb121 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [22:36:22] RECOVERY - mw134 Puppet on mw134 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [22:36:29] RECOVERY - swiftobject113 Puppet on swiftobject113 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [22:53:04] RECOVERY - kb.nena.org - LetsEncrypt on sslhost is OK: OK - Certificate 'kb.nena.org' will expire on Tue 07 Nov 2023 20:31:38 GMT +0000. [22:56:02] [02miraheze/CreateWiki] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/commit/50be321394f5 [22:56:04] [02miraheze/CreateWiki] 07paladox 0350be321 - Set a 24H cache expiry [22:56:07] [02CreateWiki] 07paladox created branch 03paladox-patch-2 - 13https://github.com/miraheze/CreateWiki [22:56:09] [02CreateWiki] 07paladox opened pull request 03#431: Set a 24H cache expiry - 13https://github.com/miraheze/CreateWiki/pull/431 [23:04:51] [02CreateWiki] 07paladox synchronize pull request 03#431: Set a 24H cache expiry - 13https://github.com/miraheze/CreateWiki/pull/431 [23:04:54] [02miraheze/CreateWiki] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/50be321394f5...e051682ee00d [23:04:55] [02miraheze/CreateWiki] 07paladox 03e051682 - Update CreateWikiJson.php [23:08:13] [02CreateWiki] 07paladox synchronize pull request 03#431: Set a 24H cache expiry - 13https://github.com/miraheze/CreateWiki/pull/431 [23:08:15] [02miraheze/CreateWiki] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/e051682ee00d...79c798f2a298 [23:08:18] [02miraheze/CreateWiki] 07paladox 0379c798f - Update CreateWikiJson.php [23:08:22] miraheze/CreateWiki - paladox the build has errored. [23:09:50] [02CreateWiki] 07paladox synchronize pull request 03#431: Set a 24H cache expiry - 13https://github.com/miraheze/CreateWiki/pull/431 [23:09:52] [02miraheze/CreateWiki] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/79c798f2a298...2c42fd5464c0 [23:09:54] [02miraheze/CreateWiki] 07paladox 032c42fd5 - Update CreateWikiJson.php [23:10:11] [02miraheze/MatomoAnalytics] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/MatomoAnalytics/commit/88b368943dd1 [23:10:13] [02miraheze/MatomoAnalytics] 07paladox 0388b3689 - Set a 24H cache expiry [23:10:15] [02MatomoAnalytics] 07paladox created branch 03paladox-patch-1 - 13https://github.com/miraheze/MatomoAnalytics [23:10:16] [02MatomoAnalytics] 07paladox opened pull request 03#106: Set a 24H cache expiry - 13https://github.com/miraheze/MatomoAnalytics/pull/106 [23:16:39] miraheze/MatomoAnalytics - paladox the build passed. [23:16:47] miraheze/CreateWiki - paladox the build has errored. [23:17:15] PROBLEM - ndgkb.nena.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'ndgkb.nena.org' expires in 15 day(s) (Fri 25 Aug 2023 18:32:28 GMT +0000). [23:24:46] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/77d0f3a43a82...e358a6000c26 [23:24:47] [02miraheze/ssl] 07MirahezeSSLBot 03e358a60 - Bot: Update SSL cert for ndgkb.nena.org [23:51:24] PROBLEM - with.cpt-ra.bid - reverse DNS on sslhost is WARNING: Timeout: The DNS operation timed out after 5.407057762145996 seconds [23:57:34] miraheze/CreateWiki - paladox the build has errored.