[00:02:20] PROBLEM - cp27 HTTP 4xx/5xx ERROR Rate on cp27 is CRITICAL: CRITICAL - NGINX Error Rate is 75% [00:02:47] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03cvtbot [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/517dd8020147...2c2849b6e1f4 [00:02:50] [02miraheze/puppet] 07Universal-Omega 032c2849b - Add NuGet.Config [00:02:51] [02puppet] 07Universal-Omega synchronize pull request 03#3727: irc: enable cvtbot - 13https://github.com/miraheze/puppet/pull/3727 [00:03:42] miraheze/puppet - Universal-Omega the build has errored. [00:04:19] PROBLEM - cp27 HTTP 4xx/5xx ERROR Rate on cp27 is WARNING: WARNING - NGINX Error Rate is 50% [00:05:16] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03cvtbot [+1/-0/±0] 13https://github.com/miraheze/puppet/compare/2c2849b6e1f4...0f5b9de18e50 [00:05:17] [02miraheze/puppet] 07Universal-Omega 030f5b9de - Add NuGet.Config [00:05:18] [02puppet] 07Universal-Omega synchronize pull request 03#3727: irc: enable cvtbot - 13https://github.com/miraheze/puppet/pull/3727 [00:06:27] miraheze/puppet - Universal-Omega the build has errored. [00:07:13] [02puppet] 07Universal-Omega synchronize pull request 03#3727: irc: enable cvtbot - 13https://github.com/miraheze/puppet/pull/3727 [00:07:15] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03cvtbot [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/0f5b9de18e50...4b27f0daef4b [00:07:17] [02miraheze/puppet] 07Universal-Omega 034b27f0d - Syntax [00:07:47] [02puppet] 07Universal-Omega synchronize pull request 03#3727: irc: enable cvtbot - 13https://github.com/miraheze/puppet/pull/3727 [00:07:48] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03cvtbot [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/4b27f0daef4b...6b21560ec887 [00:07:51] [02miraheze/puppet] 07Universal-Omega 036b21560 - irc user [00:18:06] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/03a5fd8c8eba...a7c7b7a34c80 [00:18:09] [02miraheze/mw-config] 07Universal-Omega 03a7c7b7a - Check for rcid as well [00:19:05] miraheze/mw-config - Universal-Omega the build passed. [00:23:04] !log [@test151] starting deploy of {'config': True} to test151 [00:23:05] !log [@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [00:23:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:23:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:24:40] [02puppet] 07Universal-Omega synchronize pull request 03#3727: irc: enable cvtbot - 13https://github.com/miraheze/puppet/pull/3727 [00:24:41] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03cvtbot [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/6b21560ec887...b2c4a3453d83 [00:24:43] [02miraheze/puppet] 07Universal-Omega 03b2c4a34 - Use SSL [00:30:58] [02puppet] 07Universal-Omega synchronize pull request 03#3727: irc: enable cvtbot - 13https://github.com/miraheze/puppet/pull/3727 [00:31:00] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03cvtbot [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/b2c4a3453d83...204587ebb2bc [00:31:01] [02miraheze/puppet] 07Universal-Omega 03204587e - Use cvtbot password [00:31:26] [02puppet] 07Universal-Omega synchronize pull request 03#3727: irc: enable cvtbot - 13https://github.com/miraheze/puppet/pull/3727 [00:31:28] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03cvtbot [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/204587ebb2bc...dce8875a8350 [00:31:30] [02miraheze/puppet] 07Universal-Omega 03dce8875 - password [00:32:00] !log [@mwtask181] starting deploy of {'config': True} to all [00:32:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:32:08] !log [@mwtask181] finished deploy of {'config': True} to all - SUCCESS in 7s [00:32:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:35:18] [02puppet] 07Universal-Omega synchronize pull request 03#3727: irc: enable cvtbot - 13https://github.com/miraheze/puppet/pull/3727 [00:35:20] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03cvtbot [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/dce8875a8350...3fb8f8c94648 [00:35:21] [02miraheze/puppet] 07Universal-Omega 033fb8f8c - Fix [00:36:27] !log add passwords::irc::cvtbot to private puppet [00:36:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:49:02] [02landing] 07Xenareee opened pull request 03#134: Fix typo - 13https://github.com/miraheze/landing/pull/134 [00:55:17] [02landing] 07paladox closed pull request 03#134: Fix typo - 13https://github.com/miraheze/landing/pull/134 [00:55:18] [02miraheze/landing] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/landing/compare/7c3d4113e9ea...ede42e52bd0f [00:55:20] [02miraheze/landing] 07Xenareee 03ede42e5 - Fix typo (#134) [00:55:46] [02landing] 07paladox closed pull request 03#133: Use full width for nav, footer and for facts col - 13https://github.com/miraheze/landing/pull/133 [00:55:48] [02miraheze/landing] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/landing/compare/ede42e52bd0f...ef1520f310f4 [00:55:49] [02miraheze/landing] 07TreeIsLife2 03ef1520f - Use full width for nav, footer and for facts col (#133) [00:55:50] miraheze/landing - Xenareee the build passed. [00:56:07] miraheze/landing - paladox the build passed. [00:56:33] miraheze/landing - paladox the build passed. [01:07:22] [02miraheze/puppet] 07Universal-Omega pushed 0329 commits to 03cvtbot [+1/-0/±28] 13https://github.com/miraheze/puppet/compare/3fb8f8c94648...f6960ba82e3d [01:07:23] [02miraheze/puppet] 07Universal-Omega 03e3cf987 - irc: enable cvtbot [01:07:25] [02miraheze/puppet] 07Universal-Omega 0336af72a - Add HTTP_PROXY [01:07:28] [02miraheze/puppet] 07Universal-Omega 032186df3 - test [01:07:31] [02miraheze/puppet] ... and 26 more commits. [01:07:33] [02puppet] 07Universal-Omega synchronize pull request 03#3727: irc: enable cvtbot - 13https://github.com/miraheze/puppet/pull/3727 [01:08:00] [02miraheze/puppet] 07github-actions[bot] pushed 031 commit to 03cvtbot [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/f6960ba82e3d...ff562faa0bd7 [01:08:02] [02miraheze/puppet] 07github-actions 03ff562fa - CI: lint puppet code to standards [01:08:05] [02puppet] 07github-actions[bot] synchronize pull request 03#3727: irc: enable cvtbot - 13https://github.com/miraheze/puppet/pull/3727 [01:10:03] RECOVERY - mon181 Puppet on mon181 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [01:14:30] PROBLEM - mon181 CVT Bot on mon181 is CRITICAL: PROCS CRITICAL: 0 processes with args 'cvtbot' [01:15:56] PROBLEM - mon181 Puppet on mon181 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 1 minute ago with 1 failures. Failed resources (up to 3 shown): File[/srv/cvtbot/src/CVTBot/bin/Release/net6.0/.nuget/NuGet/NuGet.Config] [01:17:51] RECOVERY - mon181 Puppet on mon181 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [01:18:18] RECOVERY - mon181 CVT Bot on mon181 is OK: PROCS OK: 2 processes with args 'cvtbot' [01:19:18] [02puppet] 07Universal-Omega closed pull request 03#3727: irc: enable cvtbot - 13https://github.com/miraheze/puppet/pull/3727 [01:19:19] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+1/-0/±4] 13https://github.com/miraheze/puppet/compare/dc0ff8f68b24...865a71aa5072 [01:19:20] [02miraheze/puppet] 07Universal-Omega 03865a71a - irc: enable cvtbot (#3727) [01:19:21] [02miraheze/puppet] 07Universal-Omega deleted branch 03cvtbot [01:19:23] [02puppet] 07Universal-Omega deleted branch 03cvtbot - 13https://github.com/miraheze/puppet [01:31:39] PROBLEM - puppet181 Puppet on puppet181 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_puppet] [01:47:47] [02miraheze/puppet] 07paladox pushed 035 commits to 03strongswan [+8/-0/±8] 13https://github.com/miraheze/puppet/compare/669d39203f32...41ad8c565504 [01:47:50] [02miraheze/puppet] 07paladox 0321d24e3 - Add strongswan (ipsec) module [01:47:51] [02miraheze/puppet] 07paladox 038b7add1 - Update ipresolve for puppet7 [01:47:54] [02miraheze/puppet] 07paladox 03f5cb14f - Fixes [01:47:55] [02miraheze/puppet] ... and 2 more commits. [01:47:57] [02puppet] 07paladox synchronize pull request 03#3730: Add strongswan (ipsec) module - 13https://github.com/miraheze/puppet/pull/3730 [01:48:28] [02miraheze/puppet] 07github-actions[bot] pushed 031 commit to 03strongswan [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/41ad8c565504...1d79effc56f4 [01:48:31] [02miraheze/puppet] 07github-actions 031d79eff - CI: lint puppet code to standards [01:48:33] [02puppet] 07github-actions[bot] synchronize pull request 03#3730: Add strongswan (ipsec) module - 13https://github.com/miraheze/puppet/pull/3730 [02:00:52] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/45dfa53a72d4...33a120935575 [02:00:55] [02miraheze/dns] 07paladox 0333a1209 - Temporary depool lu [02:01:03] RECOVERY - puppet181 Puppet on puppet181 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:18:11] PROBLEM - wiki.thesimswiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.thesimswiki.com' expires in 15 day(s) (Mon 19 Feb 2024 02:16:01 AM GMT +0000). [02:21:27] PROBLEM - www.thesimswiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.thesimswiki.com' expires in 15 day(s) (Mon 19 Feb 2024 02:16:01 AM GMT +0000). [02:21:39] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/d77a36237c40...4c0b7edd1c60 [02:21:41] [02miraheze/ssl] 07MirahezeSSLBot 034c0b7ed - Bot: Update SSL cert for www.thesimswiki.com [02:22:35] PROBLEM - thesimswiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.thesimswiki.com' expires in 15 day(s) (Mon 19 Feb 2024 02:16:01 AM GMT +0000). [02:27:03] PROBLEM - apocrypha.wiki - LetsEncrypt on sslhost is CRITICAL: Name or service not knownHTTP CRITICAL - Unable to open TCP socket [02:36:33] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03cvtbot-update [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/f9d4b4f5cafb [02:36:34] [02miraheze/puppet] 07Universal-Omega 03f9d4b4f - cvtbot: use latest repo and ensure NuGet directories exist [02:36:37] [02puppet] 07Universal-Omega created branch 03cvtbot-update - 13https://github.com/miraheze/puppet [02:36:39] [02puppet] 07Universal-Omega opened pull request 03#3731: cvtbot: use latest repo and ensure NuGet directories exist - 13https://github.com/miraheze/puppet/pull/3731 [02:37:09] [02miraheze/puppet] 07github-actions[bot] pushed 031 commit to 03cvtbot-update [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/f9d4b4f5cafb...b11459ae0eb7 [02:37:10] [02miraheze/puppet] 07github-actions 03b11459a - CI: lint puppet code to standards [02:37:13] [02puppet] 07github-actions[bot] synchronize pull request 03#3731: cvtbot: use latest repo and ensure NuGet directories exist - 13https://github.com/miraheze/puppet/pull/3731 [02:38:52] [02puppet] 07Universal-Omega synchronize pull request 03#3731: cvtbot: use latest repo and ensure NuGet directories exist - 13https://github.com/miraheze/puppet/pull/3731 [02:38:54] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03cvtbot-update [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/b11459ae0eb7...33599ea28214 [02:38:57] [02miraheze/puppet] 07Universal-Omega 0333599ea - Fix [02:51:20] RECOVERY - www.thesimswiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.thesimswiki.com' will expire on Fri 03 May 2024 01:21:33 AM GMT +0000. [02:51:34] RECOVERY - thesimswiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.thesimswiki.com' will expire on Fri 03 May 2024 01:21:33 AM GMT +0000. [02:54:34] PROBLEM - rockrevolution.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'rockrevolution.wiki' expires in 15 day(s) (Mon 19 Feb 2024 02:25:53 AM GMT +0000). [02:54:44] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/4c0b7edd1c60...5001899cd028 [02:54:47] [02miraheze/ssl] 07MirahezeSSLBot 035001899 - Bot: Update SSL cert for rockrevolution.wiki [02:56:44] RECOVERY - apocrypha.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'apocrypha.wiki' will expire on Fri 01 Mar 2024 12:06:59 PM GMT +0000. [03:07:36] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [03:08:46] [02puppet] 07Universal-Omega synchronize pull request 03#3731: cvtbot: use latest repo and ensure NuGet directories exist - 13https://github.com/miraheze/puppet/pull/3731 [03:08:49] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03cvtbot-update [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/33599ea28214...163760c32be5 [03:08:52] [02miraheze/puppet] 07Universal-Omega 03163760c - Add build [03:09:25] [02miraheze/puppet] 07github-actions[bot] pushed 031 commit to 03cvtbot-update [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/163760c32be5...d674c78bc440 [03:09:28] [02miraheze/puppet] 07github-actions 03d674c78 - CI: lint puppet code to standards [03:09:31] [02puppet] 07github-actions[bot] synchronize pull request 03#3731: cvtbot: use latest repo and ensure NuGet directories exist - 13https://github.com/miraheze/puppet/pull/3731 [03:13:04] [02puppet] 07Universal-Omega synchronize pull request 03#3731: cvtbot: use latest repo and ensure NuGet directories exist - 13https://github.com/miraheze/puppet/pull/3731 [03:13:06] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03cvtbot-update [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/d674c78bc440...6780a2ff91df [03:13:09] [02miraheze/puppet] 07Universal-Omega 036780a2f - Require Exec [03:17:18] RECOVERY - wiki.thesimswiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.thesimswiki.com' will expire on Fri 03 May 2024 01:21:33 AM GMT +0000. [03:23:07] RECOVERY - rockrevolution.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'rockrevolution.wiki' will expire on Fri 03 May 2024 01:54:39 AM GMT +0000. [03:28:25] PROBLEM - apocrypha.wiki - LetsEncrypt on sslhost is CRITICAL: Name or service not knownHTTP CRITICAL - Unable to open TCP socket [03:37:05] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['NS.ANKH.FR.eu.org.', 'NS1.eu.org.', 'NS1.ERIOMEM.NET.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [03:38:40] [02puppet] 07Universal-Omega synchronize pull request 03#3731: cvtbot: use latest repo and ensure NuGet directories exist - 13https://github.com/miraheze/puppet/pull/3731 [03:38:43] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03cvtbot-update [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/6780a2ff91df...028d33877ed2 [03:38:44] [02miraheze/puppet] 07Universal-Omega 03028d338 - Update command [03:38:50] PROBLEM - mon181 Puppet on mon181 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 1 minute ago with 1 failures. Failed resources (up to 3 shown): Exec[CVTBot-build] [03:42:50] RECOVERY - mon181 Puppet on mon181 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [03:44:38] [02puppet] 07Universal-Omega synchronize pull request 03#3731: cvtbot: use latest repo and ensure NuGet directories exist - 13https://github.com/miraheze/puppet/pull/3731 [03:44:41] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03cvtbot-update [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/028d33877ed2...0200ab108590 [03:44:43] [02miraheze/puppet] 07Universal-Omega 030200ab1 - test [03:46:08] [02puppet] 07Universal-Omega synchronize pull request 03#3731: cvtbot: use latest repo and ensure NuGet directories exist - 13https://github.com/miraheze/puppet/pull/3731 [03:46:09] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03cvtbot-update [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/0200ab108590...6923909995f5 [03:46:10] [02miraheze/puppet] 07Universal-Omega 036923909 - test [03:47:40] [02puppet] 07Universal-Omega synchronize pull request 03#3731: cvtbot: use latest repo and ensure NuGet directories exist - 13https://github.com/miraheze/puppet/pull/3731 [03:47:43] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03cvtbot-update [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/6923909995f5...d5482ce5119c [03:47:45] [02miraheze/puppet] 07Universal-Omega 03d5482ce - test [03:48:50] PROBLEM - mon181 Puppet on mon181 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 1 minute ago with 1 failures. Failed resources (up to 3 shown): Exec[CVTBot-build] [03:49:39] [02puppet] 07Universal-Omega synchronize pull request 03#3731: cvtbot: use latest repo and ensure NuGet directories exist - 13https://github.com/miraheze/puppet/pull/3731 [03:49:40] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03cvtbot-update [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/d5482ce5119c...1c759dfb4944 [03:49:41] [02miraheze/puppet] 07Universal-Omega 031c759df - Test [03:50:49] RECOVERY - mon181 Puppet on mon181 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [03:58:07] RECOVERY - apocrypha.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'apocrypha.wiki' will expire on Fri 01 Mar 2024 12:06:59 PM GMT +0000. [04:00:51] [02puppet] 07Universal-Omega synchronize pull request 03#3731: cvtbot: use latest repo and ensure NuGet directories exist - 13https://github.com/miraheze/puppet/pull/3731 [04:00:54] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03cvtbot-update [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/1c759dfb4944...f335e57d0ae2 [04:00:55] [02miraheze/puppet] 07Universal-Omega 03f335e57 - Add [04:01:22] [02miraheze/puppet] 07github-actions[bot] pushed 031 commit to 03cvtbot-update [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/f335e57d0ae2...b37cb9433804 [04:01:23] [02miraheze/puppet] 07github-actions 03b37cb94 - CI: lint puppet code to standards [04:01:25] [02puppet] 07github-actions[bot] synchronize pull request 03#3731: cvtbot: use latest repo and ensure NuGet directories exist - 13https://github.com/miraheze/puppet/pull/3731 [04:04:10] [02puppet] 07Universal-Omega synchronize pull request 03#3731: cvtbot: use latest repo and ensure NuGet directories exist - 13https://github.com/miraheze/puppet/pull/3731 [04:04:11] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03cvtbot-update [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/b37cb9433804...12deb43dee90 [04:04:13] [02miraheze/puppet] 07Universal-Omega 0312deb43 - Add directory [04:04:45] [02miraheze/puppet] 07github-actions[bot] pushed 031 commit to 03cvtbot-update [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/12deb43dee90...da677e13f1cb [04:04:47] [02miraheze/puppet] 07github-actions 03da677e1 - CI: lint puppet code to standards [04:04:49] [02puppet] 07github-actions[bot] synchronize pull request 03#3731: cvtbot: use latest repo and ensure NuGet directories exist - 13https://github.com/miraheze/puppet/pull/3731 [04:07:54] [02puppet] 07Universal-Omega synchronize pull request 03#3731: cvtbot: use latest repo and ensure NuGet directories exist - 13https://github.com/miraheze/puppet/pull/3731 [04:07:55] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03cvtbot-update [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/da677e13f1cb...9f0232c79455 [04:07:57] [02miraheze/puppet] 07Universal-Omega 039f0232c - Add before [04:08:33] [02miraheze/puppet] 07github-actions[bot] pushed 031 commit to 03cvtbot-update [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/9f0232c79455...0961c21346d8 [04:08:35] [02miraheze/puppet] 07github-actions 030961c21 - CI: lint puppet code to standards [04:08:36] [02puppet] 07github-actions[bot] synchronize pull request 03#3731: cvtbot: use latest repo and ensure NuGet directories exist - 13https://github.com/miraheze/puppet/pull/3731 [04:10:23] [02puppet] 07Universal-Omega edited pull request 03#3731: cvtbot: fully puppetize install - 13https://github.com/miraheze/puppet/pull/3731 [04:11:29] [02puppet] 07Universal-Omega synchronize pull request 03#3731: cvtbot: fully puppetize install - 13https://github.com/miraheze/puppet/pull/3731 [04:11:32] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03cvtbot-update [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/0961c21346d8...80bc57c7a4b6 [04:11:33] [02miraheze/puppet] 07Universal-Omega 0380bc57c - Increase max_files to 5000 [04:18:11] [02puppet] 07Universal-Omega synchronize pull request 03#3731: cvtbot: fully puppetize install - 13https://github.com/miraheze/puppet/pull/3731 [04:18:12] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03cvtbot-update [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/80bc57c7a4b6...5a973858bea0 [04:18:15] [02miraheze/puppet] 07Universal-Omega 035a97385 - Update [04:18:50] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03cvtbot-update [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/5a973858bea0...212805a5403e [04:18:53] [02miraheze/puppet] 07Universal-Omega 03212805a - Update [04:18:55] [02puppet] 07Universal-Omega synchronize pull request 03#3731: cvtbot: fully puppetize install - 13https://github.com/miraheze/puppet/pull/3731 [04:24:09] [02puppet] 07Universal-Omega closed pull request 03#3731: cvtbot: fully puppetize install - 13https://github.com/miraheze/puppet/pull/3731 [04:24:10] [02miraheze/puppet] 07Universal-Omega deleted branch 03cvtbot-update [04:24:12] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/puppet/compare/865a71aa5072...4c7f2dd6c36c [04:24:15] [02miraheze/puppet] 07Universal-Omega 034c7f2dd - cvtbot: fully puppetize install (#3731) [04:24:17] [02puppet] 07Universal-Omega deleted branch 03cvtbot-update - 13https://github.com/miraheze/puppet [04:29:49] PROBLEM - apocrypha.wiki - LetsEncrypt on sslhost is CRITICAL: Name or service not knownHTTP CRITICAL - Unable to open TCP socket [04:33:09] [02puppet] 07Universal-Omega closed pull request 03#3176: [DNM] bookworm/python: use bookworm-only features, drop buster dependencies - 13https://github.com/miraheze/puppet/pull/3176 [04:33:12] [02puppet] 07Universal-Omega commented on pull request 03#3176: [DNM] bookworm/python: use bookworm-only features, drop buster dependencies - 13https://github.com/miraheze/puppet/pull/3176#issuecomment-1925086941 [04:36:49] [02puppet] 07Universal-Omega reviewed pull request 03#3589 commit - 13https://github.com/miraheze/puppet/pull/3589#discussion_r1476949892 [04:36:50] [02puppet] 07Universal-Omega reviewed pull request 03#3589 commit - 13https://github.com/miraheze/puppet/pull/3589#discussion_r1476949854 [04:36:53] [02puppet] 07Universal-Omega reviewed pull request 03#3589 commit - 13https://github.com/miraheze/puppet/pull/3589#discussion_r1476949939 [04:38:46] [02puppet] 07Universal-Omega closed pull request 03#3444: add google fonts (T11269) - 13https://github.com/miraheze/puppet/pull/3444 [04:38:48] [02puppet] 07Universal-Omega commented on pull request 03#3444: add google fonts (T11269) - 13https://github.com/miraheze/puppet/pull/3444#issuecomment-1925089366 [04:46:17] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 6.66, 6.85, 6.04 [04:48:12] RECOVERY - cp41 Current Load on cp41 is OK: LOAD OK - total load average: 4.82, 6.01, 5.82 [05:29:12] RECOVERY - apocrypha.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'apocrypha.wiki' will expire on Fri 01 Mar 2024 12:06:59 PM GMT +0000. [05:49:42] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 7.18, 6.50, 5.97 [05:51:37] RECOVERY - cp41 Current Load on cp41 is OK: LOAD OK - total load average: 5.89, 6.51, 6.05 [05:55:27] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 7.64, 7.29, 6.49 [06:03:06] RECOVERY - cp41 Current Load on cp41 is OK: LOAD OK - total load average: 6.38, 6.61, 6.49 [06:07:01] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 6.50, 7.61, 7.01 [06:09:01] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 7.62, 8.07, 7.27 [06:11:01] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 6.81, 7.73, 7.26 [06:14:16] !log [agent@mwtask181] starting deploy of {'extension_list': True, 'versions': '1.40'} to all [06:14:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:14:24] !log [agent@mwtask181] finished deploy of {'extension_list': True, 'versions': '1.40'} to all - SUCCESS in 7s [06:14:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:15:01] !log [agent@mwtask181] starting deploy of {'extension_list': True, 'versions': '1.41'} to all [06:15:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:15:09] !log [agent@mwtask181] finished deploy of {'extension_list': True, 'versions': '1.41'} to all - SUCCESS in 7s [06:15:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:17:02] RECOVERY - cp41 Current Load on cp41 is OK: LOAD OK - total load average: 5.70, 6.31, 6.80 [06:21:01] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 6.37, 7.36, 7.19 [06:23:01] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 8.52, 7.80, 7.37 [06:25:01] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 5.43, 6.83, 7.06 [06:27:01] RECOVERY - cp41 Current Load on cp41 is OK: LOAD OK - total load average: 4.52, 6.04, 6.74 [06:35:01] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 9.09, 7.12, 6.73 [06:37:01] RECOVERY - cp41 Current Load on cp41 is OK: LOAD OK - total load average: 6.76, 6.79, 6.64 [06:45:36] [Grafana] !sre FIRING: The mediawiki job queue has more than 2500 unclaimed jobs https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [06:54:05] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/importDump.php --wiki=robloxcriticalstrikewiki --no-updates --username-prefix=wikia:roblox-critical-strike /home/reception/ImportDump/robloxcriticalstrikewiki-20240203030426.xml (START) [06:54:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:58:16] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/5001899cd028...ccc96d95ac47 [06:58:17] [02miraheze/ssl] 07MirahezeSSLBot 03ccc96d9 - Bot: Add SSL cert for holocron.net [07:19:44] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/importDump.php --wiki=robloxcriticalstrikewiki --no-updates --username-prefix=wikia:roblox-critical-strike /home/reception/ImportDump/robloxcriticalstrikewiki-20240203030426.xml (END - exit=0) [07:19:45] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/rebuildall.php --wiki=robloxcriticalstrikewiki (START) [07:19:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:19:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:24:56] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/rebuildall.php --wiki=robloxcriticalstrikewiki (END - exit=0) [07:24:57] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/initSiteStats.php --wiki=robloxcriticalstrikewiki --active --update (END - exit=0) [07:24:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:25:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:28:57] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 10.63, 9.59, 7.72 [07:34:44] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 3.34, 7.03, 7.33 [07:38:22] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.mahdiruiz.line.pm. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [07:40:32] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 8.48, 6.76, 7.08 [07:42:28] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 6.84, 6.56, 6.96 [07:44:23] RECOVERY - cp41 Current Load on cp41 is OK: LOAD OK - total load average: 5.20, 6.01, 6.71 [07:48:16] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 8.69, 7.35, 7.09 [07:50:12] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 5.06, 6.54, 6.83 [07:58:08] RECOVERY - cp41 Current Load on cp41 is OK: LOAD OK - total load average: 5.36, 6.44, 6.75 [08:07:32] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp36.wikitide.net - CNAME OK [08:15:36] [Grafana] !sre RESOLVED: High Job Queue Backlog https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [08:19:30] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 10.10, 7.69, 6.77 [08:21:26] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 7.33, 7.56, 6.84 [08:25:17] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 9.46, 8.27, 7.23 [08:26:15] RECOVERY - cp27 HTTP 4xx/5xx ERROR Rate on cp27 is OK: OK - NGINX Error Rate is 24% [08:27:13] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 6.88, 7.81, 7.20 [08:33:01] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 8.22, 7.70, 7.27 [08:42:39] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 6.36, 7.71, 7.71 [08:44:35] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 8.71, 7.88, 7.75 [08:48:26] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 7.12, 7.92, 7.83 [08:50:22] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 8.51, 7.92, 7.83 [08:52:18] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 7.73, 7.72, 7.76 [08:56:10] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 10.21, 7.65, 7.63 [08:58:08] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 6.18, 6.83, 7.33 [09:02:08] RECOVERY - cp41 Current Load on cp41 is OK: LOAD OK - total load average: 3.89, 5.56, 6.71 [09:07:38] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.andreijiroh.uk.eu.org All nameservers failed to answer the query. [09:08:08] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 9.16, 8.17, 7.49 [09:10:08] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 6.78, 7.58, 7.35 [09:16:08] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 10.40, 7.89, 7.46 [09:18:08] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 7.22, 7.36, 7.31 [09:28:08] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 10.51, 8.36, 7.68 [09:28:34] PROBLEM - apocrypha.wiki - LetsEncrypt on sslhost is CRITICAL: Name or service not knownHTTP CRITICAL - Unable to open TCP socket [09:28:48] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [09:30:08] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 5.77, 7.38, 7.41 [09:34:08] RECOVERY - cp41 Current Load on cp41 is OK: LOAD OK - total load average: 5.49, 5.92, 6.77 [09:37:36] RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp37.wikitide.net - CNAME OK [09:43:56] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 8.87, 7.70, 7.14 [09:51:39] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 7.51, 7.70, 7.40 [09:55:31] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 8.63, 8.34, 7.74 [09:57:31] RECOVERY - apocrypha.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'apocrypha.wiki' will expire on Fri 01 Mar 2024 12:06:59 PM GMT +0000. [09:57:47] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['NS.ANKH.FR.eu.org.', 'NS1.eu.org.', 'NS1.ERIOMEM.NET.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [10:01:18] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 5.43, 7.73, 7.73 [10:09:00] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 8.91, 7.51, 7.51 [10:10:56] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 5.80, 6.79, 7.24 [10:16:44] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 11.14, 9.10, 8.05 [10:28:20] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 7.72, 7.89, 7.93 [10:30:16] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 9.39, 8.49, 8.15 [10:35:34] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query line.pm. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [10:42:08] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 6.47, 7.35, 7.97 [10:46:08] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 9.93, 8.06, 8.10 [10:54:08] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 7.48, 7.31, 7.85 [10:57:20] PROBLEM - apocrypha.wiki - LetsEncrypt on sslhost is CRITICAL: Name or service not knownHTTP CRITICAL - Unable to open TCP socket [11:04:43] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp37.wikitide.net - CNAME OK [11:06:08] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 9.57, 7.98, 7.56 [11:08:08] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 7.50, 7.85, 7.58 [11:12:09] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 8.27, 7.84, 7.60 [11:18:08] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 6.93, 7.86, 7.74 [11:22:08] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 8.16, 8.05, 7.86 [11:26:17] RECOVERY - apocrypha.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'apocrypha.wiki' will expire on Fri 01 Mar 2024 12:06:59 PM GMT +0000. [11:32:08] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 7.48, 7.59, 7.80 [11:40:08] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 8.13, 7.55, 7.68 [11:42:08] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 5.74, 7.12, 7.53 [11:46:08] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 10.83, 8.52, 7.94 [11:48:08] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 6.39, 7.43, 7.59 [11:50:08] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 8.16, 7.76, 7.70 [11:54:08] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 7.62, 7.80, 7.74 [12:02:08] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 9.34, 7.39, 7.44 [12:04:08] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 7.18, 7.51, 7.50 [12:06:08] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 8.83, 8.14, 7.73 [12:08:08] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 6.97, 7.42, 7.50 [12:10:08] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 9.19, 8.07, 7.73 [12:14:18] I'm unable to ping cp36 [12:14:30] nevermind it is back [12:35:22] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.mahdiruiz.line.pm. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [12:37:08] [02miraheze/puppet] 07paladox pushed 036 commits to 03strongswan [+8/-0/±9] 13https://github.com/miraheze/puppet/compare/1d79effc56f4...988ca5f48355 [12:37:10] [02miraheze/puppet] 07paladox 031af6aee - Add strongswan (ipsec) module [12:37:11] [02miraheze/puppet] 07paladox 0398b54ad - Update ipresolve for puppet7 [12:37:13] [02miraheze/puppet] 07paladox 03bb423ee - Fixes [12:37:16] [02miraheze/puppet] ... and 3 more commits. [12:37:17] [02puppet] 07paladox synchronize pull request 03#3730: Add strongswan (ipsec) module - 13https://github.com/miraheze/puppet/pull/3730 [12:41:19] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.andreijiroh.uk.eu.org All nameservers failed to answer the query. [12:55:02] PROBLEM - apocrypha.wiki - LetsEncrypt on sslhost is CRITICAL: Name or service not knownHTTP CRITICAL - Unable to open TCP socket [13:04:33] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp36.wikitide.net - CNAME OK [13:14:36] [02mw-config] 07redbluegreenhat opened pull request 03#5455: T10615: Add SemanticScribunto to ManageWiki - 13https://github.com/miraheze/mw-config/pull/5455 [13:15:49] miraheze/mw-config - redbluegreenhat the build passed. [13:20:38] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [13:22:15] PROBLEM - cp27 HTTP 4xx/5xx ERROR Rate on cp27 is WARNING: WARNING - NGINX Error Rate is 40% [13:23:59] RECOVERY - apocrypha.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'apocrypha.wiki' will expire on Fri 01 Mar 2024 12:06:59 PM GMT +0000. [13:49:36] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['NS.ANKH.FR.eu.org.', 'NS1.eu.org.', 'NS1.ERIOMEM.NET.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [14:10:09] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 6.63, 7.05, 7.67 [14:11:11] RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp36.wikitide.net - CNAME OK [14:12:09] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 10.89, 7.90, 7.87 [14:23:48] PROBLEM - apocrypha.wiki - LetsEncrypt on sslhost is CRITICAL: Name or service not knownHTTP CRITICAL - Unable to open TCP socket [15:21:42] RECOVERY - apocrypha.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'apocrypha.wiki' will expire on Fri 01 Mar 2024 12:06:59 PM GMT +0000. [15:23:14] !log [paladox@mwtask181] /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/1.40/maintenance/updateCollation.php --force [15:23:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:24:01] !log [paladox@mwtask181] /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/1.40/extensions/RottenLinks/maintenance/updateExternalLinks.php [15:24:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:28:08] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 6.27, 6.49, 7.78 [15:32:08] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 17.87, 9.69, 8.59 [15:33:39] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query line.pm. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [15:34:32] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/b24539e2ab8d [15:34:33] [02miraheze/puppet] 07paladox 03b24539e - Allow varnish to use mediawiki port 80 [15:34:34] [02puppet] 07paladox created branch 03paladox-patch-12 - 13https://github.com/miraheze/puppet [15:34:36] [02puppet] 07paladox opened pull request 03#3732: Allow varnish to use mediawiki port 80 - 13https://github.com/miraheze/puppet/pull/3732 [15:48:41] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/b24539e2ab8d...165fdc822c31 [15:48:44] [02miraheze/puppet] 07paladox 03165fdc8 - Update default.vcl [15:48:46] [02puppet] 07paladox synchronize pull request 03#3732: Allow varnish to use mediawiki port 80 - 13https://github.com/miraheze/puppet/pull/3732 [15:49:12] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 208.30 ms [15:49:40] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/165fdc822c31...6ef66ee18437 [15:49:43] [02miraheze/puppet] 07paladox 036ef66ee - Update init.pp [15:49:46] [02puppet] 07paladox synchronize pull request 03#3732: Allow varnish to use mediawiki port 80 - 13https://github.com/miraheze/puppet/pull/3732 [15:49:50] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/6ef66ee18437...06a5956ce29a [15:49:51] [02miraheze/puppet] 07paladox 0306a5956 - Update cp36.yaml [15:49:54] [02puppet] 07paladox synchronize pull request 03#3732: Allow varnish to use mediawiki port 80 - 13https://github.com/miraheze/puppet/pull/3732 [15:49:59] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/06a5956ce29a...0dcf5919d24f [15:50:01] [02miraheze/puppet] 07paladox 030dcf591 - Update cp37.yaml [15:50:04] [02puppet] 07paladox synchronize pull request 03#3732: Allow varnish to use mediawiki port 80 - 13https://github.com/miraheze/puppet/pull/3732 [15:51:15] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 185.64 ms [15:52:33] PROBLEM - apocrypha.wiki - LetsEncrypt on sslhost is CRITICAL: Name or service not knownHTTP CRITICAL - Unable to open TCP socket [15:54:23] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/0dcf5919d24f...93f3c892ca3d [15:54:26] [02miraheze/puppet] 07paladox 0393f3c89 - Update common.yaml [15:54:28] [02puppet] 07paladox synchronize pull request 03#3732: Allow varnish to use mediawiki port 80 - 13https://github.com/miraheze/puppet/pull/3732 [15:58:57] [02puppet] 07paladox synchronize pull request 03#3732: Allow varnish to use mediawiki port 80 - 13https://github.com/miraheze/puppet/pull/3732 [15:58:59] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/93f3c892ca3d...b28727ed50e3 [15:59:01] [02miraheze/puppet] 07paladox 03b28727e - Update mediawiki.conf.erb [16:00:09] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 5.61, 6.98, 7.82 [16:02:09] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 9.09, 7.84, 8.03 [16:03:22] [02puppet] 07paladox synchronize pull request 03#3732: Allow varnish to use mediawiki port 80 - 13https://github.com/miraheze/puppet/pull/3732 [16:03:23] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/b28727ed50e3...89d2088dc82c [16:03:26] [02miraheze/puppet] 07paladox 0389d2088 - Update mediawiki.conf.erb [16:03:49] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/89d2088dc82c...b2441d3402de [16:03:51] [02miraheze/puppet] 07paladox 03b2441d3 - Update mediawiki.pp [16:03:52] [02puppet] 07paladox synchronize pull request 03#3732: Allow varnish to use mediawiki port 80 - 13https://github.com/miraheze/puppet/pull/3732 [16:04:20] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/b2441d3402de...3dce2a9b779e [16:04:22] [02miraheze/puppet] 07paladox 033dce2a9 - Update common.yaml [16:04:23] [02puppet] 07paladox synchronize pull request 03#3732: Allow varnish to use mediawiki port 80 - 13https://github.com/miraheze/puppet/pull/3732 [16:21:18] PROBLEM - mw152 Puppet on mw152 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 14 minutes ago with 0 failures [16:21:30] RECOVERY - apocrypha.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'apocrypha.wiki' will expire on Fri 01 Mar 2024 12:06:59 PM GMT +0000. [16:28:09] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 5.40, 7.04, 7.98 [16:32:04] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp37.wikitide.net - CNAME OK [16:32:08] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 8.40, 7.99, 8.15 [16:34:11] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 6.44, 7.28, 7.85 [16:40:08] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 10.02, 7.85, 7.84 [16:42:08] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 7.17, 7.47, 7.70 [16:46:08] PROBLEM - cp41 Current Load on cp41 is CRITICAL: LOAD CRITICAL - total load average: 11.96, 8.37, 7.92 [16:48:08] PROBLEM - cp41 Current Load on cp41 is WARNING: LOAD WARNING - total load average: 6.64, 7.53, 7.68 [17:01:51] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/33a120935575...74668e5a744b [17:01:54] [02miraheze/dns] 07paladox 0374668e5 - Depool au and ja [17:03:18] RECOVERY - mw152 Puppet on mw152 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [17:04:09] RECOVERY - cp41 Current Load on cp41 is OK: LOAD OK - total load average: 3.06, 5.33, 6.41 [17:28:33] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/a7c7b7a34c80...c606d57f2407 [17:28:36] [02miraheze/mw-config] 07Universal-Omega 03c606d57 - Update bastion servers [17:29:30] miraheze/mw-config - Universal-Omega the build passed. [17:30:49] [02puppet] 07Universal-Omega synchronize pull request 03#3720: Decom servers - 13https://github.com/miraheze/puppet/pull/3720 [17:30:50] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03decom [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/bc0c187e68b2...a07693035a12 [17:30:51] [02miraheze/puppet] 07Universal-Omega 03a076930 - Revert [17:32:16] [02puppet] 07Universal-Omega synchronize pull request 03#3720: Decom servers - 13https://github.com/miraheze/puppet/pull/3720 [17:32:19] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03decom [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/a07693035a12...5f373eba3926 [17:32:20] [02miraheze/puppet] 07Universal-Omega 035f373eb - Readd [17:32:33] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query line.pm. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [17:32:38] !log [@mwtask181] starting deploy of {'config': True} to all [17:32:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:32:45] !log [@mwtask181] finished deploy of {'config': True} to all - SUCCESS in 7s [17:32:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:40:15] RECOVERY - cp27 HTTP 4xx/5xx ERROR Rate on cp27 is OK: OK - NGINX Error Rate is 34% [17:40:30] [02dns] 07Universal-Omega closed pull request 03#483: Decom servers - 13https://github.com/miraheze/dns/pull/483 [17:40:32] [02miraheze/dns] 07Universal-Omega deleted branch 03decom [17:40:33] [02dns] 07Universal-Omega deleted branch 03decom - 13https://github.com/miraheze/dns [17:40:40] [02miraheze/dns] 07Universal-Omega pushed 031 commit to 03decom [+0/-0/±1] 13https://github.com/miraheze/dns/commit/d1aac81f54fe [17:40:42] [02miraheze/dns] 07Universal-Omega 03d1aac81 - Update 0.0.0.0.0.4.7.6.0.1.a.2.ip6.arpa [17:40:43] [02dns] 07Universal-Omega created branch 03decom - 13https://github.com/miraheze/dns [17:40:55] [02dns] 07Universal-Omega opened pull request 03#488: Decom servers - 13https://github.com/miraheze/dns/pull/488 [17:42:44] [02miraheze/dns] 07Universal-Omega pushed 031 commit to 03decom [+0/-0/±1] 13https://github.com/miraheze/dns/compare/d1aac81f54fe...bc3786edca85 [17:42:46] [02miraheze/dns] 07Universal-Omega 03bc3786e - Update miraheze.org [17:42:48] [02dns] 07Universal-Omega synchronize pull request 03#488: Decom servers - 13https://github.com/miraheze/dns/pull/488 [17:46:18] [02puppet] 07Universal-Omega closed pull request 03#3720: Decom servers - 13https://github.com/miraheze/puppet/pull/3720 [17:46:19] [02miraheze/puppet] 07Universal-Omega deleted branch 03decom [17:46:22] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-6/±2] 13https://github.com/miraheze/puppet/compare/4c7f2dd6c36c...bef4519f791f [17:46:23] [02miraheze/puppet] 07Universal-Omega 03bef4519 - Decom servers (#3720) [17:46:26] [02puppet] 07Universal-Omega deleted branch 03decom - 13https://github.com/miraheze/puppet [17:46:54] [02dns] 07Universal-Omega closed pull request 03#488: Decom servers - 13https://github.com/miraheze/dns/pull/488 [17:46:56] [02miraheze/dns] 07Universal-Omega deleted branch 03decom [17:46:59] [02miraheze/dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/dns/compare/74668e5a744b...8c3f10b5c86d [17:47:02] [02miraheze/dns] 07Universal-Omega 038c3f10b - Decom servers (#488) [17:47:05] [02dns] 07Universal-Omega deleted branch 03decom - 13https://github.com/miraheze/dns [17:50:15] PROBLEM - apocrypha.wiki - LetsEncrypt on sslhost is CRITICAL: Name or service not knownHTTP CRITICAL - Unable to open TCP socket [17:53:15] !log [@test151] starting deploy of {'config': True} to test151 [17:53:16] !log [@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [17:53:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:53:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:01:46] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp37.wikitide.net - CNAME OK [18:05:09] PROBLEM - puppet181 Puppet on puppet181 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_srv-ssl] [18:19:12] RECOVERY - apocrypha.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'apocrypha.wiki' will expire on Fri 01 Mar 2024 12:06:59 PM GMT +0000. [18:29:08] RECOVERY - puppet181 Puppet on puppet181 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [18:40:00] Fingerprint for test151 is missing [18:40:11] please ping me on IRC once added [18:42:45] by fingerprint I mean SSH fingerprint ofc [18:43:31] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/8c3f10b5c86d...1e4cebd2af4f [18:43:32] [02miraheze/dns] 07paladox 031e4cebd - Add rdb151/161 and bots171 to dns [18:45:00] [02miraheze/puppet] 07Universal-Omega created branch 03phorge 13https://github.com/miraheze/puppet/commit/bef4519f791f9fa8fc8d87df227ebc9bad2c39e4 [18:45:01] [02puppet] 07Universal-Omega created branch 03phorge - 13https://github.com/miraheze/puppet [18:45:55] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03phorge [+7/-9/±4] 13https://github.com/miraheze/puppet/compare/bef4519f791f...11ea36c5a770 [18:45:56] [02miraheze/puppet] 07Universal-Omega 0311ea36c - Rename phabricator module to phorge [18:46:07] [02puppet] 07Universal-Omega opened pull request 03#3733: Rename phabricator module to phorge - 13https://github.com/miraheze/puppet/pull/3733 [18:46:41] [02miraheze/puppet] 07github-actions[bot] pushed 031 commit to 03phorge [+0/-0/±2] 13https://github.com/miraheze/puppet/compare/11ea36c5a770...8813cdb1dc63 [18:46:44] [02miraheze/puppet] 07github-actions 038813cdb - CI: lint puppet code to standards [18:46:47] [02puppet] 07github-actions[bot] synchronize pull request 03#3733: Rename phabricator module to phorge - 13https://github.com/miraheze/puppet/pull/3733 [19:10:13] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-14 [+1/-0/±0] 13https://github.com/miraheze/puppet/commit/d456f3a49de7 [19:10:14] [02miraheze/puppet] 07paladox 03d456f3a - Install rdb151 [19:10:16] [02puppet] 07paladox created branch 03paladox-patch-14 - 13https://github.com/miraheze/puppet [19:10:17] [02puppet] 07paladox opened pull request 03#3734: Install rdb151 - 13https://github.com/miraheze/puppet/pull/3734 [19:10:50] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-14 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/d456f3a49de7...3704dcfaf464 [19:10:51] [02miraheze/puppet] 07paladox 033704dcf - Update site.pp [19:10:54] [02puppet] 07paladox synchronize pull request 03#3734: Install rdb151 - 13https://github.com/miraheze/puppet/pull/3734 [19:10:56] [02puppet] 07paladox closed pull request 03#3734: Install rdb151 - 13https://github.com/miraheze/puppet/pull/3734 [19:10:57] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-14 [19:10:59] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/puppet/compare/bef4519f791f...a73d62a92fb7 [19:11:00] [02miraheze/puppet] 07paladox 03a73d62a - Install rdb151 (#3734) [19:11:03] [02puppet] 07paladox deleted branch 03paladox-patch-14 - 13https://github.com/miraheze/puppet [19:12:40] [02miraheze/dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/1e4cebd2af4f...1f432d4f3043 [19:12:42] [02miraheze/dns] 07Universal-Omega 031f432d4 - add ownercheck TXT record for ns2.wikitide.net [19:17:13] PROBLEM - rdb151 conntrack_table_size on rdb151 is UNKNOWN: NRPE: Unable to read output [19:17:18] PROBLEM - rdb151 Puppet on rdb151 is UNKNOWN: NRPE: Unable to read output [19:17:54] PROBLEM - rdb151 ferm_active on rdb151 is UNKNOWN: NRPE: Unable to read output [19:17:56] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/1f432d4f3043...ab36f968d0bf [19:17:59] [02miraheze/dns] 07paladox 03ab36f96 - Fix bots171 ip [19:19:12] RECOVERY - rdb151 conntrack_table_size on rdb151 is OK: OK: nf_conntrack is 0 % full [19:19:16] RECOVERY - rdb151 Puppet on rdb151 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [19:19:53] RECOVERY - rdb151 ferm_active on rdb151 is OK: OK ferm input default policy is set [19:34:06] PROBLEM - rdb151 SSH on rdb151 is CRITICAL: connect to address 10.0.15.119 and port 22: No route to host [19:34:07] PROBLEM - rdb151 NTP time on rdb151 is CRITICAL: connect to address 10.0.15.119 port 5666: No route to hostconnect to host 10.0.15.119 port 5666: No route to host [19:34:18] PROBLEM - Host rdb151 is DOWN: CRITICAL - Host Unreachable (10.0.15.119) [19:36:19] RECOVERY - Host rdb151 is UP: PING OK - Packet loss = 0%, RTA = 0.28 ms [19:37:00] PROBLEM - bots171 ferm_active on bots171 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:37:01] PROBLEM - bots171 PowerDNS Recursor on bots171 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:37:03] PROBLEM - bots171 Disk Space on bots171 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:37:04] PROBLEM - bots171 APT on bots171 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:37:14] PROBLEM - bots171 Puppet on bots171 is UNKNOWN: NRPE: Unable to read output [19:37:34] PROBLEM - bots171 NTP time on bots171 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:37:36] PROBLEM - bots171 conntrack_table_size on bots171 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [19:38:08] RECOVERY - rdb151 SSH on rdb151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u2 (protocol 2.0) [19:38:13] RECOVERY - rdb151 NTP time on rdb151 is OK: NTP OK: Offset -0.01583960652 secs [19:38:54] RECOVERY - bots171 ferm_active on bots171 is OK: OK ferm input default policy is set [19:38:57] RECOVERY - bots171 Disk Space on bots171 is OK: DISK OK - free space: / 15180MiB (84% inode=93%); [19:38:58] RECOVERY - bots171 PowerDNS Recursor on bots171 is OK: DNS OK: 0.051 seconds response time. miraheze.org returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206 [19:39:00] RECOVERY - bots171 APT on bots171 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [19:39:12] RECOVERY - bots171 Puppet on bots171 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [19:39:35] RECOVERY - bots171 conntrack_table_size on bots171 is OK: OK: nf_conntrack is 0 % full [19:39:38] PROBLEM - bots171 NTP time on bots171 is WARNING: NTP WARNING: Offset 0.1938095093 secs [19:41:43] RECOVERY - bots171 NTP time on bots171 is OK: NTP OK: Offset -0.00406652689 secs [19:47:44] PROBLEM - apocrypha.wiki - LetsEncrypt on sslhost is CRITICAL: Name or service not knownHTTP CRITICAL - Unable to open TCP socket [20:15:35] PROBLEM - mon181 Puppet on mon181 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:16:51] PROBLEM - ns2 GDNSD Datacenters on ns2 is UNKNOWN: NRPE: Unable to read output [20:17:32] RECOVERY - mon181 Puppet on mon181 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:18:50] [02miraheze/dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/ab36f968d0bf...bcc550e4f8ec [20:18:51] [02miraheze/dns] 07Universal-Omega 03bcc550e - Remove cp3 [20:18:57] PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/usr/share/GeoIP/GeoLite2-Country.mmdb] [20:19:01] PROBLEM - ns2 Auth DNS on ns2 is CRITICAL: No response from DNS 2001:41d0:801:2000::4089 [20:20:11] [02miraheze/dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/bcc550e4f8ec...62f81ecff7af [20:20:14] [02miraheze/dns] 07Universal-Omega 0362f81ec - Update ns2 IP [20:30:09] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.mahdiruiz.line.pm. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [20:32:33] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 3 datacenters are down: 51.79.55.151/cpweb2, 2607:5300:205:200::2aa8/cpweb2, 63.141.240.6/cpweb2 [20:33:00] RECOVERY - ns2 Auth DNS on ns2 is OK: DNS OK: 0.317 seconds response time. miraheze.org returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206 [20:33:01] RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:36:52] [02miraheze/dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/62f81ecff7af...fcffbba55fca [20:36:54] [02miraheze/dns] 07Universal-Omega 03fcffbba - Update ns2 IP [20:45:19] RECOVERY - apocrypha.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'apocrypha.wiki' will expire on Fri 01 Mar 2024 12:06:59 PM GMT +0000. [20:45:38] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/a73d62a92fb7...71a2f584db84 [20:45:41] [02miraheze/puppet] 07Universal-Omega 0371a2f58 - Update ns2 IP [20:45:50] CosmicAlpha: don't forget glue records [20:47:02] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/71a2f584db84...4d78a6b8d5dd [20:47:04] [02miraheze/puppet] 07Universal-Omega 034d78a6b - Decom cloud12 and ns2.mh [20:47:23] RhinosF1: already done [20:47:40] :) [20:47:54] And just like that SCSVG is no more [20:49:13] Thank god [20:50:38] !log remove ns2.miraheze.org and cloud12.miraheze.org from puppet and shutdown ns2 VM on cloud12 [20:50:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:52:06] [02miraheze/dns] 07Universal-Omega pushed 031 commit to 03decom [+0/-1/±0] 13https://github.com/miraheze/dns/commit/2e0f172522e6 [20:52:09] [02miraheze/dns] 07Universal-Omega 032e0f172 - Delete zones/0.0.0.0.0.4.7.6.0.1.a.2.ip6.arpa [20:52:11] [02dns] 07Universal-Omega created branch 03decom - 13https://github.com/miraheze/dns [20:52:13] [02dns] 07Universal-Omega opened pull request 03#489: Decom cloud12 - 13https://github.com/miraheze/dns/pull/489 [20:52:42] [02miraheze/dns] 07Universal-Omega pushed 031 commit to 03decom [+0/-0/±1] 13https://github.com/miraheze/dns/compare/2e0f172522e6...f91cad6b4dcb [20:52:43] [02miraheze/dns] 07Universal-Omega 03f91cad6 - Update miraheze.org [20:52:45] [02dns] 07Universal-Omega synchronize pull request 03#489: Decom cloud12 - 13https://github.com/miraheze/dns/pull/489 [20:53:17] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/extensions/ManageWiki/maintenance/toggleExtension.php --wiki=criticalrolewiki cirrussearch (END - exit=0) [20:53:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:55:45] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/runJobs.php --wiki=criticalrolewiki --procs 24 (START) [20:55:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:58:45] [02dns] 07Universal-Omega closed pull request 03#489: Decom cloud12 - 13https://github.com/miraheze/dns/pull/489 [20:58:48] [02miraheze/dns] 07Universal-Omega deleted branch 03decom [20:58:49] [02miraheze/dns] 07Universal-Omega pushed 031 commit to 03master [+0/-1/±1] 13https://github.com/miraheze/dns/compare/fcffbba55fca...e2ee591cff45 [20:58:52] [02miraheze/dns] 07Universal-Omega 03e2ee591 - Decom cloud12 (#489) [20:58:55] [02dns] 07Universal-Omega deleted branch 03decom - 13https://github.com/miraheze/dns [20:59:23] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/runJobs.php --wiki=criticalrolewiki --procs 24 (END - exit=0) [20:59:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:00:56] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/runJobs.php --wiki=criticalrolewiki --procs 24 (START) [21:00:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:01:41] !log [universalomega@puppet181] Upgraded packages libc6-dbg, libc-devtools, libc6-dev, libc-dev-bin, libc6, libc-bin, libc-l10n, and locales on db181 [21:01:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:01:58] !log [universalomega@puppet181] Upgraded packages libc6-dbg, libc-devtools, libc6-dev, libc-dev-bin, libc6, libc-bin, libc-l10n, and locales on db182 [21:01:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:02:14] !log [universalomega@puppet181] Upgraded packages libc-devtools, libc6-dev, libc-dev-bin, libc6, libc-bin, libc-l10n, and locales on cloud18 [21:02:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:02:31] !log [universalomega@puppet181] Upgraded packages libc6-dbg, libc-devtools, libc6-dev, libc-dev-bin, libc6, libc-bin, libc-l10n, and locales on db151 [21:02:32] RECOVERY - db182 APT on db182 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [21:02:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:02:52] RECOVERY - db181 APT on db181 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [21:02:53] !log [universalomega@puppet181] Upgraded packages libc6-dbg, libc-devtools, libc6-dev, libc-dev-bin, libc6, libc-bin, libc-l10n, and locales on db171 [21:02:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:03:07] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/runJobs.php --wiki=criticalrolewiki --procs 24 (END - exit=0) [21:03:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:03:14] !log [universalomega@puppet181] Upgraded packages libc-devtools, libc6-dev, libc-dev-bin, libc6, libc-bin, libc-l10n, and locales on cp37 [21:03:15] RECOVERY - db151 APT on db151 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [21:03:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:03:21] RECOVERY - cp37 APT on cp37 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [21:03:40] RECOVERY - cloud18 APT on cloud18 is OK: APT OK: 10 packages available for upgrade (0 critical updates). [21:03:55] !log [universalomega@puppet181] Upgraded packages libc-devtools, libc6-dev, libc-dev-bin, libc6, libc-bin, redis-server, redis-tools, libc-l10n, and locales on matomo151 [21:03:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:04:19] RECOVERY - db171 APT on db171 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [21:04:31] !log [universalomega@puppet181] Upgraded packages libc-devtools, libc6-dev, libc-dev-bin, libc6, libc-bin, libc-l10n, locales, openjdk-17-jdk, openjdk-17-jdk-headless, openjdk-17-jre, and openjdk-17-jre-headless on graylog161 [21:04:31] RECOVERY - graylog161 APT on graylog161 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [21:04:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:04:50] !log [universalomega@puppet181] Upgraded packages libc-devtools, libc6-dev, libc-dev-bin, libc6, libc-bin, libc-l10n, and locales on bast181 [21:04:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:05:08] !log [universalomega@puppet181] Upgraded packages libc-devtools, libc6-dev, libc-dev-bin, libc6, libc-bin, libc-l10n, and locales on cloud16 [21:05:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:05:13] RECOVERY - matomo151 APT on matomo151 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [21:05:25] RECOVERY - bast181 APT on bast181 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [21:05:26] !log [universalomega@puppet181] Upgraded packages libc-devtools, libc6-dev, libc-dev-bin, libc6, libc-bin, libc-l10n, and locales on mem151 [21:05:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:05:44] !log [universalomega@puppet181] Upgraded packages libc6-dbg, libc-devtools, libc6-dev, libc-dev-bin, libc6, libc-bin, libc-l10n, and locales on db161 [21:05:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:05:49] RECOVERY - mem151 APT on mem151 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [21:06:01] RECOVERY - cloud15 APT on cloud15 is OK: APT OK: 8 packages available for upgrade (0 critical updates). [21:06:02] !log [universalomega@puppet181] Upgraded packages libc-devtools, libc6-dev, libc-dev-bin, libc6, libc-bin, libc-l10n, and locales on cloud15 [21:06:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:06:06] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 17.31, 20.71, 15.52 [21:06:20] !log [universalomega@puppet181] Upgraded packages libc-devtools, libc6-dev, libc-dev-bin, libc6, libc-bin, redis-server, redis-tools, libc-l10n, and locales on jobchron171 [21:06:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:06:36] RECOVERY - cloud16 APT on cloud16 is OK: APT OK: 2 packages available for upgrade (0 critical updates). [21:06:39] !log [universalomega@puppet181] Upgraded packages libc-devtools, libc6-dev, libc-dev-bin, libc6, libc-bin, libc-l10n, and locales on cloud17 [21:06:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:06:51] RECOVERY - jobchron171 APT on jobchron171 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [21:06:59] !log [universalomega@puppet181] Upgraded packages libc-devtools, libc6-dev, libc-dev-bin, libc6, libc-bin, libc-l10n, and locales on bast161 [21:07:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:07:09] RECOVERY - bast161 APT on bast161 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [21:07:17] !log [universalomega@puppet181] Upgraded packages libc-devtools, libc6-dev, libc-dev-bin, libc6, libc-bin, libc-l10n, and locales on cp36 [21:07:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:07:36] !log [universalomega@puppet181] Upgraded packages libc-devtools, libc6-dev, libc-dev-bin, libc6, libc-bin, libc-l10n, and locales on ldap171 [21:07:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:07:38] RECOVERY - db161 APT on db161 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [21:07:52] RECOVERY - ldap171 APT on ldap171 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [21:08:02] !log [universalomega@puppet181] Upgraded packages libc-devtools, libc6-dev, libc-dev-bin, libc6, libc-bin, libc-l10n, and locales on mem161 [21:08:04] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 9.52, 16.66, 14.64 [21:08:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:08:28] RECOVERY - cloud17 APT on cloud17 is OK: APT OK: 2 packages available for upgrade (0 critical updates). [21:08:34] !log [universalomega@puppet181] Upgraded packages libc-devtools, libc6-dev, libc-dev-bin, libc6, libc-bin, libc-l10n, and locales on cp27 [21:08:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:08:44] RECOVERY - mem161 APT on mem161 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [21:08:46] RECOVERY - cp36 APT on cp36 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [21:08:53] !log [universalomega@puppet181] Upgraded packages libc-devtools, libc6-dev, libc-dev-bin, libc6, libc-bin, libc-l10n, and locales on mon181 [21:08:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:09:06] RECOVERY - cp27 APT on cp27 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [21:09:20] !log [universalomega@puppet181] Upgraded packages libc-devtools, libc6-dev, libc-dev-bin, libc6, libc-bin, libc-l10n, and locales on cp41 [21:09:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:09:38] !log [universalomega@puppet181] Upgraded packages libc-devtools, libc6-dev, libc-dev-bin, libc6, libc-bin, libc-l10n, locales, openjdk-17-jdk, openjdk-17-jdk-headless, openjdk-17-jre, and openjdk-17-jre-headless on puppet181 [21:09:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:09:50] RECOVERY - mon181 APT on mon181 is OK: APT OK: 2 packages available for upgrade (0 critical updates). [21:10:06] !log [universalomega@puppet181] Upgraded packages libc-devtools, libc6-dev, libc-dev-bin, libc6, libc-bin, libc-l10n, and locales on cp51 [21:10:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:10:21] RECOVERY - cp41 APT on cp41 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [21:10:23] !log [universalomega@puppet181] Upgraded packages libc-devtools, libc6-dev, libc-dev-bin, libc6, libc-bin, libc-l10n, locales, locales-all, xserver-common, xserver-xorg-core, and xvfb on mw182 [21:10:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:10:26] RECOVERY - cp51 APT on cp51 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [21:10:47] !log [universalomega@puppet181] Upgraded packages libc-devtools, libc6-dev, libc-dev-bin, libc6, libc-bin, libc-l10n, locales, openjdk-17-jdk, openjdk-17-jdk-headless, openjdk-17-jre, and openjdk-17-jre-headless on os151 [21:10:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:11:03] !log [universalomega@puppet181] Upgraded packages libc-devtools, libc6-dev, libc-dev-bin, libc6, libc-bin, libc-l10n, locales, locales-all, xserver-common, xserver-xorg-core, and xvfb on mw181 [21:11:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:11:20] !log [universalomega@puppet181] Upgraded packages libc-devtools, libc6-dev, libc-dev-bin, libc6, libc-bin, libc-l10n, locales, locales-all, xserver-common, xserver-xorg-core, and xvfb on mw151 [21:11:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:11:32] RECOVERY - puppet181 APT on puppet181 is OK: APT OK: 2 packages available for upgrade (0 critical updates). [21:11:44] RECOVERY - mw182 APT on mw182 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [21:11:46] RECOVERY - os151 APT on os151 is OK: APT OK: 2 packages available for upgrade (0 critical updates). [21:11:47] RECOVERY - mw181 APT on mw181 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [21:11:47] !log [universalomega@puppet181] Upgraded packages libc-devtools, libc6-dev, libc-dev-bin, libc6, libc-bin, libc-l10n, locales, and locales-all on mw162 [21:11:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:12:04] RECOVERY - mw151 APT on mw151 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [21:12:04] !log [universalomega@puppet181] Upgraded packages libc-devtools, libc6-dev, libc-dev-bin, libc6, libc-bin, libc-l10n, locales, locales-all, xserver-common, xserver-xorg-core, and xvfb on mw152 [21:12:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:12:31] !log [universalomega@puppet181] Upgraded packages libc-devtools, libc6-dev, libc-dev-bin, libc6, libc-bin, libc-l10n, and locales on cp26 [21:12:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:12:55] !log [universalomega@puppet181] Upgraded packages libc-devtools, libc6-dev, libc-dev-bin, libc6, libc-bin, libc-l10n, locales, openjdk-17-jdk, openjdk-17-jdk-headless, openjdk-17-jre, and openjdk-17-jre-headless on os161 [21:12:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:12:59] RECOVERY - cp26 APT on cp26 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [21:13:13] !log [universalomega@puppet181] Upgraded packages libc-devtools, libc6-dev, libc-dev-bin, libc6, libc-bin, libc-l10n, locales, and locales-all on mw171 [21:13:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:13:29] !log [universalomega@puppet181] Upgraded packages libc-devtools, libc6-dev, libc-dev-bin, libc6, libc-bin, libc-l10n, and locales on prometheus151 [21:13:29] RECOVERY - mw152 APT on mw152 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [21:13:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:13:33] RECOVERY - os161 APT on os161 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [21:13:35] RECOVERY - mw162 APT on mw162 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [21:13:47] !log [universalomega@puppet181] Upgraded packages libc-devtools, libc6-dev, libc-dev-bin, libc6, libc-bin, libc-l10n, locales, and locales-all on mw161 [21:13:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:13:57] RECOVERY - mw171 APT on mw171 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [21:14:03] RECOVERY - prometheus151 APT on prometheus151 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [21:14:06] !log [universalomega@puppet181] Upgraded packages libc-devtools, libc6-dev, libc-dev-bin, libc6, libc-bin, libc-l10n, locales, and locales-all on mw172 [21:14:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:14:16] RECOVERY - mw172 APT on mw172 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [21:14:28] !log [universalomega@puppet181] Upgraded packages libc-devtools, libc6-dev, libc-dev-bin, libc6, libc-bin, libc-l10n, and locales on phorge171 [21:14:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:14:48] !log [universalomega@puppet181] Upgraded packages libc-devtools, libc6-dev, libc-dev-bin, libc6, libc-bin, libc-l10n, and locales on ns1 [21:14:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:14:51] RECOVERY - ns1 APT on ns1 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [21:15:04] !log [universalomega@puppet181] Upgraded packages libc-devtools, libc6-dev, libc-dev-bin, libc6, libc-bin, libc-l10n, and locales on reports171 [21:15:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:15:21] RECOVERY - mw161 APT on mw161 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [21:15:23] !log [universalomega@puppet181] Upgraded packages libc-devtools, libc6-dev, libc-dev-bin, libc6, libc-bin, redis-server, redis-tools, libc-l10n, locales, locales-all, xserver-common, xserver-xorg-core, and xvfb on test151 [21:15:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:15:38] RECOVERY - reports171 APT on reports171 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [21:15:38] !log [universalomega@puppet181] Upgraded packages libc-devtools, libc6-dev, libc-dev-bin, libc6, libc-bin, libc-l10n, and locales on swiftobject181 [21:15:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:15:48] PROBLEM - apocrypha.wiki - LetsEncrypt on sslhost is CRITICAL: Name or service not knownHTTP CRITICAL - Unable to open TCP socket [21:15:53] RECOVERY - phorge171 APT on phorge171 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [21:15:57] !log [universalomega@puppet181] Upgraded packages libc-devtools, libc6-dev, libc-dev-bin, libc6, libc-bin, libc-l10n, and locales on swiftobject161 [21:15:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:16:16] !log [universalomega@puppet181] Upgraded packages libnghttp2-14, curl, libcurl4, and libcurl3-gnutls on ns2 [21:16:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:16:33] RECOVERY - test151 APT on test151 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [21:16:35] !log [universalomega@puppet181] Upgraded packages libc-devtools, libc6-dev, libc-dev-bin, libc6, libc-bin, libc-l10n, and locales on swiftobject151 [21:16:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:16:43] RECOVERY - swiftobject161 APT on swiftobject161 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [21:16:54] !log [universalomega@puppet181] Upgraded packages libc-devtools, libc6-dev, libc-dev-bin, libc6, libc-bin, libc-l10n, and locales on swiftproxy171 [21:16:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:17:12] !log [universalomega@puppet181] Upgraded packages libc-devtools, libc6-dev, libc-dev-bin, libc6, libc-bin, libc-l10n, and locales on swiftproxy161 [21:17:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:17:23] RECOVERY - swiftobject151 APT on swiftobject151 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [21:17:27] RECOVERY - swiftproxy161 APT on swiftproxy161 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [21:17:32] !log [universalomega@puppet181] Upgraded packages libc-devtools, libc6-dev, libc-dev-bin, libc6, libc-bin, libc-l10n, and locales on swiftac171 [21:17:33] RECOVERY - swiftobject181 APT on swiftobject181 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [21:17:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:17:39] RECOVERY - ns2 APT on ns2 is OK: APT OK: 14 packages available for upgrade (0 critical updates). [21:17:48] !log [universalomega@puppet181] Upgraded packages libc-devtools, libc6-dev, libc-dev-bin, libc6, libc-bin, libc-l10n, locales, locales-all, xserver-common, xserver-xorg-core, and xvfb on mwtask181 [21:17:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:18:07] !log [universalomega@puppet181] Upgraded packages libc-devtools, libc6-dev, libc-dev-bin, libc6, libc-bin, libc-l10n, and locales on swiftobject171 [21:18:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:18:19] RECOVERY - mwtask181 APT on mwtask181 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [21:18:35] RECOVERY - swiftac171 APT on swiftac171 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [21:18:45] RECOVERY - swiftobject171 APT on swiftobject171 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [21:18:45] RECOVERY - swiftproxy171 APT on swiftproxy171 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [21:23:35] PROBLEM - ns2 NTP time on ns2 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H [-4|-6] [-w ] [-c ] [-v verbose] [-o