[00:00:25] PROBLEM - cp1 HTTP 4xx/5xx ERROR Rate on cp1 is UNKNOWN: UNKNOWN - NGINX Error Rate is UNKNOWN [00:02:25] PROBLEM - cp1 HTTP 4xx/5xx ERROR Rate on cp1 is CRITICAL: CRITICAL - NGINX Error Rate is 100% [01:11:08] [02WikiTideOrg/WikiTideMagic] 07Universal-Omega pushed 031 commit to 03managewiki [+0/-0/±1] 13https://github.com/WikiTideOrg/WikiTideMagic/commit/c7a06054dc3e [01:11:11] [02WikiTideOrg/WikiTideMagic] 07Universal-Omega 03c7a0605 - Use Miraheze ManageWiki [01:11:13] [02WikiTideMagic] 07Universal-Omega created branch 03managewiki - 13https://github.com/WikiTideOrg/WikiTideMagic [01:11:15] [02WikiTideMagic] 07Universal-Omega opened pull request 03#7: Use Miraheze ManageWiki - 13https://github.com/WikiTideOrg/WikiTideMagic/pull/7 [01:12:08] [02WikiTideOrg/WikiTideMagic] 07Universal-Omega pushed 031 commit to 03managewiki [+0/-0/±1] 13https://github.com/WikiTideOrg/WikiTideMagic/compare/c7a06054dc3e...6c243f53b4e1 [01:12:11] [02WikiTideOrg/WikiTideMagic] 07Universal-Omega 036c243f5 - Update repo [01:12:12] [02WikiTideMagic] 07Universal-Omega synchronize pull request 03#7: Use Miraheze ManageWiki - 13https://github.com/WikiTideOrg/WikiTideMagic/pull/7 [01:14:32] [02WikiTideOrg/WikiTideMagic] 07Universal-Omega pushed 031 commit to 03managewiki [+0/-0/±1] 13https://github.com/WikiTideOrg/WikiTideMagic/compare/6c243f53b4e1...67104403dd56 [01:14:34] [02WikiTideOrg/WikiTideMagic] 07Universal-Omega 036710440 - Update namespace [01:14:36] [02WikiTideMagic] 07Universal-Omega synchronize pull request 03#7: Use Miraheze ManageWiki - 13https://github.com/WikiTideOrg/WikiTideMagic/pull/7 [01:17:17] [02WikiTideOrg/mw-config] 07Universal-Omega pushed 031 commit to 03main [+0/-0/±1] 13https://github.com/WikiTideOrg/mw-config/compare/d2af935b7ac1...89eacb0da2d2 [01:17:19] [02WikiTideOrg/mw-config] 07Universal-Omega 0389eacb0 - Update linkpage for YouTube [01:18:11] WikiTideOrg/mw-config - Universal-Omega the build passed. [01:18:22] !log [@jobrunner1] starting deploy of {'config': True, 'versions': ['']} to [mw1, mw2, jobrunner1] [01:18:25] !log [@jobrunner1] finished deploy of {'config': True, 'versions': ['']} to [mw1, mw2, jobrunner1] - SUCCESS in 2s [01:18:29] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [01:18:35] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [01:22:38] WikiTideOrg/WikiTideMagic - Universal-Omega the build passed. [01:22:50] [02WikiTideMagic] 07Universal-Omega closed pull request 03#7: Use Miraheze ManageWiki - 13https://github.com/WikiTideOrg/WikiTideMagic/pull/7 [01:22:53] [02WikiTideOrg/WikiTideMagic] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±3] 13https://github.com/WikiTideOrg/WikiTideMagic/compare/62420edfd3ff...89b3619d5db0 [01:22:55] [02WikiTideOrg/WikiTideMagic] 07Universal-Omega 0389b3619 - Use Miraheze ManageWiki [01:22:58] [02WikiTideOrg/WikiTideMagic] 07Universal-Omega deleted branch 03managewiki [01:23:00] [02WikiTideMagic] 07Universal-Omega deleted branch 03managewiki - 13https://github.com/WikiTideOrg/WikiTideMagic [01:30:59] WikiTideOrg/WikiTideMagic - Universal-Omega the build passed. [01:31:30] !log [@test1] starting deploy of {'config': True, 'versions': ['']} to [test1] [01:31:31] !log [@test1] finished deploy of {'config': True, 'versions': ['']} to [test1] - SUCCESS in 0s [01:31:37] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [01:31:46] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [01:31:52] !log [@test1] starting deploy of {'folders': '1.42/extensions/WikiTideMagic', 'versions': ['']} to [test1] [01:31:58] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [01:48:48] !log [@jobrunner1] starting deploy of {'folders': '1.40/extensions/WikiTideMagic', 'versions': ['']} to [mw1, mw2, jobrunner1] [01:48:51] !log [@jobrunner1] finished deploy of {'folders': '1.40/extensions/WikiTideMagic', 'versions': ['']} to [mw1, mw2, jobrunner1] - SUCCESS in 3s [01:48:54] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [01:49:00] !log [@jobrunner1] starting deploy of {'folders': '1.41/extensions/WikiTideMagic', 'versions': ['']} to [mw1, mw2, jobrunner1] [01:49:01] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [01:49:02] !log [@jobrunner1] finished deploy of {'folders': '1.41/extensions/WikiTideMagic', 'versions': ['']} to [mw1, mw2, jobrunner1] - SUCCESS in 2s [01:49:07] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [01:49:14] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [02:11:32] PROBLEM - cp4 Puppet on cp4 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [02:13:18] PROBLEM - phorge1 Puppet on phorge1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [02:13:27] PROBLEM - mon1 Puppet on mon1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [02:13:31] PROBLEM - prometheus1 Puppet on prometheus1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [02:39:33] RECOVERY - cp4 Puppet on cp4 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [02:39:33] RECOVERY - prometheus1 Puppet on prometheus1 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [02:41:18] RECOVERY - phorge1 Puppet on phorge1 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [02:41:28] RECOVERY - mon1 Puppet on mon1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:46:57] PROBLEM - db1 Backups SQL on db1 is CRITICAL: FILE_AGE CRITICAL: /var/log/backup-logs/sql-backup.log is 1209665 seconds old and 1977 bytes [05:45:10] PROBLEM - newcascadia.net - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.404 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [06:45:09] PROBLEM - newcascadia.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - newcascadia.net All nameservers failed to answer the query. [07:15:10] PROBLEM - newcascadia.net - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [08:15:09] PROBLEM - newcascadia.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - newcascadia.net All nameservers failed to answer the query. [09:15:10] PROBLEM - newcascadia.net - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.404 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [09:45:09] PROBLEM - newcascadia.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - newcascadia.net All nameservers failed to answer the query. [17:42:20] PROBLEM - nexttide.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'nexttide.org' expires in 7 day(s) (Sun 14 Jan 2024 05:21:30 PM GMT +0000).