[00:00:25] PROBLEM - cp1 HTTP 4xx/5xx ERROR Rate on cp1 is UNKNOWN: UNKNOWN - NGINX Error Rate is UNKNOWN [00:02:25] PROBLEM - cp1 HTTP 4xx/5xx ERROR Rate on cp1 is CRITICAL: CRITICAL - NGINX Error Rate is 100% [00:08:00] [02WikiTideOrg/ssl] 07WikiTideSSL pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/ssl/compare/92c03967bf16...671452452480 [00:08:03] [02WikiTideOrg/ssl] 07WikiTideSSLBot 036714524 - Bot: Update SSL cert for polandballwiki.com [00:17:38] [02WikiTideOrg/ssl] 07WikiTideSSL pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/ssl/compare/671452452480...1f7a5f612b11 [00:17:41] [02WikiTideOrg/ssl] 07WikiTideSSLBot 031f7a5f6 - Bot: Update SSL cert for valorantjp.wiki [00:21:37] PROBLEM - puppet1 Puppet on puppet1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_ssl] [00:23:08] [02WikiTideOrg/ssl] 07WikiTideSSL pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/ssl/compare/1f7a5f612b11...1c617f591800 [00:23:11] [02WikiTideOrg/ssl] 07WikiTideSSLBot 031c617f5 - Bot: Update SSL cert for wiki.colleimadcat.com [00:24:40] [02WikiTideOrg/ssl] 07WikiTideSSL pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/ssl/compare/1c617f591800...76599e161f20 [00:24:42] [02WikiTideOrg/ssl] 07WikiTideSSLBot 0376599e1 - Bot: Update SSL cert for greatamerica.wiki [00:29:30] [02WikiTideOrg/ssl] 07WikiTideSSL pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/ssl/compare/76599e161f20...d646d20d1b5d [00:29:33] [02WikiTideOrg/ssl] 07WikiTideSSLBot 03d646d20 - Bot: Update SSL cert for meta.sagan4.org [00:35:30] [02WikiTideOrg/ssl] 07WikiTideSSL pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/ssl/compare/d646d20d1b5d...910df21f0255 [00:35:32] [02WikiTideOrg/ssl] 07WikiTideSSLBot 03910df21 - Bot: Update SSL cert for rosettacode.org [00:40:42] [02WikiTideOrg/ssl] 07WikiTideSSL pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/ssl/compare/910df21f0255...53ae94933833 [00:40:45] [02WikiTideOrg/ssl] 07WikiTideSSLBot 0353ae949 - Bot: Update SSL cert for www.lgbtqia.wiki [00:46:56] [02WikiTideOrg/ssl] 07WikiTideSSL pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/ssl/compare/53ae94933833...d06ba3fb487d [00:46:59] [02WikiTideOrg/ssl] 07WikiTideSSLBot 03d06ba3f - Bot: Update SSL cert for distrowiki.net [00:49:37] RECOVERY - puppet1 Puppet on puppet1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:52:04] [02WikiTideOrg/ssl] 07WikiTideSSL pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/ssl/compare/d06ba3fb487d...946cc0c71025 [00:52:05] [02WikiTideOrg/ssl] 07WikiTideSSLBot 03946cc0c - Bot: Update SSL cert for beta.sagan4.org [00:59:24] [02WikiTideOrg/ssl] 07WikiTideSSL pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/ssl/compare/946cc0c71025...9da563660bc7 [00:59:26] [02WikiTideOrg/ssl] 07WikiTideSSLBot 039da5636 - Bot: Update SSL cert for thegreatwar.uk [01:06:59] [02WikiTideOrg/ssl] 07WikiTideSSL pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/ssl/compare/9da563660bc7...546db8b9e495 [01:07:00] [02WikiTideOrg/ssl] 07WikiTideSSLBot 03546db8b - Bot: Update SSL cert for mwcosmos.com [03:01:22] PROBLEM - mon1 Backups Grafana on mon1 is CRITICAL: FILE_AGE CRITICAL: /var/log/grafana-backup.log is 1209681 seconds old and 347 bytes [06:15:10] PROBLEM - newcascadia.net - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.404 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [06:45:09] PROBLEM - newcascadia.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - newcascadia.net All nameservers failed to answer the query. [14:45:10] PROBLEM - newcascadia.net - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [15:15:09] PROBLEM - newcascadia.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - newcascadia.net All nameservers failed to answer the query. [17:01:11] PROBLEM - mw2 MediaWiki Rendering on mw2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:02:51] PROBLEM - swiftproxy1 SSH on swiftproxy1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:03:07] PROBLEM - mw2 MediaWiki Rendering on mw2 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 8191 bytes in 0.191 second response time [17:03:20] PROBLEM - swiftproxy1 PowerDNS Recursor on swiftproxy1 is CRITICAL: CRITICAL - Plugin timed out while executing system call [17:04:56] PROBLEM - swiftproxy1 Puppet on swiftproxy1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:05:44] PROBLEM - swiftproxy1 NTP time on swiftproxy1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:05:47] PROBLEM - swiftproxy1 Current Load on swiftproxy1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:05:49] PROBLEM - swiftproxy1 ferm_active on swiftproxy1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:05:49] PROBLEM - swiftproxy1 conntrack_table_size on swiftproxy1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:08:52] PROBLEM - mw1 MediaWiki Rendering on mw1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:09:11] PROBLEM - mw2 MediaWiki Rendering on mw2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:10:53] PROBLEM - mw1 MediaWiki Rendering on mw1 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 8191 bytes in 6.438 second response time [17:11:07] PROBLEM - mw2 MediaWiki Rendering on mw2 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 8191 bytes in 0.144 second response time [17:11:21] RECOVERY - swiftproxy1 conntrack_table_size on swiftproxy1 is OK: OK: nf_conntrack is 0 % full [17:11:21] RECOVERY - swiftproxy1 ferm_active on swiftproxy1 is OK: OK ferm input default policy is set [17:11:23] RECOVERY - swiftproxy1 PowerDNS Recursor on swiftproxy1 is OK: DNS OK: 5.892 seconds response time. wikitide.org returns 2001:41d0:801:2000::4089,51.75.170.66 [17:11:26] RECOVERY - swiftproxy1 NTP time on swiftproxy1 is OK: NTP OK: Offset -0.003265798092 secs [17:12:14] PROBLEM - cp6 Varnish Backends on cp6 is CRITICAL: 3 backends are down. mw1 mw2 mediawiki [17:12:15] PROBLEM - cp5 Varnish Backends on cp5 is CRITICAL: 3 backends are down. mw1 mw2 mediawiki [17:13:12] PROBLEM - mw2 MediaWiki Rendering on mw2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:14:29] PROBLEM - cp6 HTTPS on cp6 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [17:14:35] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw1 [17:14:38] PROBLEM - mw1 HTTPS on mw1 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [17:14:49] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 1 backends are down. mw1 [17:14:52] PROBLEM - mw1 MediaWiki Rendering on mw1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:16:11] PROBLEM - cp5 HTTP 4xx/5xx ERROR Rate on cp5 is CRITICAL: CRITICAL - NGINX Error Rate is 62% [17:16:16] RECOVERY - cp5 Varnish Backends on cp5 is OK: All 7 backends are healthy [17:16:27] RECOVERY - cp6 HTTPS on cp6 is OK: HTTP OK: HTTP/2 200 - 2995 bytes in 1.360 second response time [17:16:41] RECOVERY - mw1 HTTPS on mw1 is OK: HTTP OK: HTTP/2 200 - 362 bytes in 8.190 second response time [17:17:09] PROBLEM - cp5 HTTPS on cp5 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10002 milliseconds with 0 bytes received [17:18:07] RECOVERY - cp6 Varnish Backends on cp6 is OK: All 7 backends are healthy [17:18:11] PROBLEM - cp5 HTTP 4xx/5xx ERROR Rate on cp5 is WARNING: WARNING - NGINX Error Rate is 46% [17:18:16] PROBLEM - swiftproxy1 NTP time on swiftproxy1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:18:30] PROBLEM - swiftproxy1 PowerDNS Recursor on swiftproxy1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:18:51] PROBLEM - mw1 MediaWiki Rendering on mw1 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 8191 bytes in 0.158 second response time [17:19:13] RECOVERY - cp5 HTTPS on cp5 is OK: HTTP OK: HTTP/2 200 - 2996 bytes in 5.791 second response time [17:19:14] PROBLEM - swiftproxy1 ferm_active on swiftproxy1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:19:15] PROBLEM - swiftproxy1 conntrack_table_size on swiftproxy1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [17:19:18] PROBLEM - mw2 MediaWiki Rendering on mw2 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 8191 bytes in 0.173 second response time [17:20:12] RECOVERY - cp5 HTTP 4xx/5xx ERROR Rate on cp5 is OK: OK - NGINX Error Rate is 2% [17:20:35] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 7 backends are healthy [17:20:49] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 7 backends are healthy [17:24:11] RECOVERY - swiftproxy1 conntrack_table_size on swiftproxy1 is OK: OK: nf_conntrack is 0 % full [17:24:19] RECOVERY - swiftproxy1 NTP time on swiftproxy1 is OK: NTP OK: Offset -0.003544926643 secs [17:25:58] RECOVERY - swiftproxy1 PowerDNS Recursor on swiftproxy1 is OK: DNS OK: 2.562 seconds response time. wikitide.org returns 2001:41d0:801:2000::4089,51.75.170.66 [17:27:00] RECOVERY - swiftproxy1 Puppet on swiftproxy1 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [17:27:39] RECOVERY - swiftproxy1 SSH on swiftproxy1 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u2 (protocol 2.0) [17:27:49] RECOVERY - swiftproxy1 ferm_active on swiftproxy1 is OK: OK ferm input default policy is set [17:52:25] PROBLEM - swiftproxy1 Current Load on swiftproxy1 is WARNING: LOAD WARNING - total load average: 0.18, 0.36, 7.55 [17:54:19] RECOVERY - swiftproxy1 Current Load on swiftproxy1 is OK: LOAD OK - total load average: 0.19, 0.31, 6.69 [23:43:07] [02WikiTideOrg/mw-config] 07Universal-Omega pushed 031 commit to 03main [+0/-0/±1] 13https://github.com/WikiTideOrg/mw-config/compare/89eacb0da2d2...6e27033591a8 [23:43:08] [02WikiTideOrg/mw-config] 07Universal-Omega 036e27033 - Simplify extension loading [23:44:07] WikiTideOrg/mw-config - Universal-Omega the build passed. [23:48:21] !log [@jobrunner1] starting deploy of {'config': True, 'versions': ['']} to [mw1, mw2, jobrunner1] [23:48:25] !log [@jobrunner1] finished deploy of {'config': True, 'versions': ['']} to [mw1, mw2, jobrunner1] - SUCCESS in 3s [23:48:28] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [23:48:35] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log