[00:00:46] PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 1213MiB (6% inode=94%); [00:02:28] [02mw-config] 07AgentIsai closed pull request 03#12: T37: Migrate to Swift - 13https://github.com/WikiTideOrg/mw-config/pull/12 [00:02:31] [02WikiTideOrg/mw-config] 07AgentIsai deleted branch 03AgentIsai-patch-1 [00:02:32] [02WikiTideOrg/mw-config] 07AgentIsai pushed 031 commit to 03main [+0/-0/±3] 13https://github.com/WikiTideOrg/mw-config/compare/0bdb426579de...706e4146a5a6 [00:02:34] [02WikiTideOrg/mw-config] 07AgentIsai 03706e414 - T37: Migrate to Swift (#12) * T37: Enable FileBackend for Swift * Don't load AWS anymore * Delete S3 specific GlobalSettings values [00:02:35] [02mw-config] 07AgentIsai deleted branch 03AgentIsai-patch-1 - 13https://github.com/WikiTideOrg/mw-config [00:03:11] @agentisai that says miraheze [00:03:33] WikiTideOrg/mw-config - AgentIsai the build passed. [00:03:39] lol [00:03:43] will fix later [00:04:06] It will cause db errors on all wikis using wikitidecommons [00:04:38] Because commonswiki does not exist, commonswikitide does [00:13:22] [02WikiTideOrg/mw-config] 07AgentIsai pushed 031 commit to 03main [+0/-0/±1] 13https://github.com/WikiTideOrg/mw-config/compare/706e4146a5a6...059671273d83 [00:13:23] [02WikiTideOrg/mw-config] 07AgentIsai 030596712 - Fix hostnames [00:13:42] [02WikiTideOrg/mw-config] 07AgentIsai pushed 031 commit to 03main [+0/-0/±1] 13https://github.com/WikiTideOrg/mw-config/compare/059671273d83...1f2acaf1557e [00:13:45] [02WikiTideOrg/mw-config] 07AgentIsai 031f2acaf - Fix hostname [00:14:19] WikiTideOrg/mw-config - AgentIsai the build passed. [00:14:39] WikiTideOrg/mw-config - AgentIsai the build passed. [00:15:40] PROBLEM - cp1 HTTP 4xx/5xx ERROR Rate on cp1 is WARNING: WARNING - NGINX Error Rate is 54% [00:16:36] @agentisai "InvalidArgumentException","message":"Cannot register a backend with no name. all over the place again [00:16:56] PROBLEM - swiftobject1 Current Load on swiftobject1 is WARNING: LOAD WARNING - total load average: 0.29, 0.60, 3.62 [00:17:25] nice [00:17:40] PROBLEM - cp1 HTTP 4xx/5xx ERROR Rate on cp1 is CRITICAL: CRITICAL - NGINX Error Rate is 60% [00:17:49] I see why [00:18:37] [02WikiTideOrg/mw-config] 07AgentIsai pushed 031 commit to 03main [+0/-0/±1] 13https://github.com/WikiTideOrg/mw-config/compare/1f2acaf1557e...bfa18793ec22 [00:18:39] [02WikiTideOrg/mw-config] 07AgentIsai 03bfa1879 - Remove loading S3 from WikiTideFunctions [00:18:56] RECOVERY - swiftobject1 Current Load on swiftobject1 is OK: LOAD OK - total load average: 0.22, 0.47, 3.20 [00:19:03] PROBLEM - mw2 MediaWiki Rendering on mw2 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 684 bytes in 0.157 second response time [00:19:28] InvalidArgumentException: Key \"AWSBucketName\" not found in input sources [00:19:31] WikiTideOrg/mw-config - AgentIsai the build has errored. [00:19:35] PROBLEM - jobrunner1 MediaWiki Rendering on jobrunner1 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 684 bytes in 0.196 second response time [00:19:41] WikiTideMagic [00:19:54] In options->get is why I think [00:20:17] @agentisai [00:20:31] oh [00:20:34] PROBLEM - mw1 MediaWiki Rendering on mw1 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 684 bytes in 0.171 second response time [00:20:35] let me see [00:21:54] I don't think so [00:22:18] It is if the config is used and doesn't exist an error will be thrown [00:22:35] Even if not this exact issue that is another fatal that will happen [00:22:48] it is [00:22:48] Key \"AWSBucketName\" [00:23:22] Indeed key in https://github.com/WikiTideOrg/WikiTideMagic/blob/3279cb1dbd2c975ef533f2dfa52f27a733594ca8/includes/WikiTideMagicHooks.php#L77 [00:23:45] [02WikiTideOrg/WikiTideMagic] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/WikiTideMagic/compare/3279cb1dbd2c...f66367312182 [00:23:46] [02WikiTideOrg/WikiTideMagic] 07AgentIsai 03f663673 - Remove AWS config option [00:24:02] !log [agent@jobrunner1] starting deploy of {'versions': ['1.40'], 'upgrade_extensions': ['WikiTideMagic']} to [mw1, mw2, jobrunner1] [00:24:03] !log [agent@jobrunner1] DEPLOY ABORTED: Canary check failed for meta.wikitide.org@localhost [00:24:06] !log [agent@jobrunner1] starting deploy of {'force': True, 'versions': ['1.40'], 'upgrade_extensions': ['WikiTideMagic']} to [mw1, mw2, jobrunner1] [00:24:07] !log [agent@jobrunner1] finished deploy of {'force': True, 'versions': ['1.40'], 'upgrade_extensions': ['WikiTideMagic']} to [mw1, mw2, jobrunner1] - SUCCESS in 0s [00:24:08] https://github.com/WikiTideOrg/WikiTideMagic/blob/f663673121822eb3c193194305f53a9daa3fc06b/includes/WikiTideMagicHooks.php#L148 is another fatal now [00:24:08] !log [agent@jobrunner1] starting deploy of {'force': True, 'versions': ['1.40'], 'upgrade_extensions': ['WikiTideMagic']} to [mw1, mw2, jobrunner1] [00:24:09] !log [agent@jobrunner1] finished deploy of {'force': True, 'versions': ['1.40'], 'upgrade_extensions': ['WikiTideMagic']} to [mw1, mw2, jobrunner1] - SUCCESS in 3s [00:24:11] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [00:24:14] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [00:24:34] PROBLEM - mw1 MediaWiki Rendering on mw1 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 8191 bytes in 0.308 second response time [00:24:35] or at least will be when the hooks are triggered [00:24:39] stuff loads [00:24:40] for now [00:25:00] Yeah you removed the part that will 100% fatal, it will still fatal when hooks are triggered though [00:25:03] PROBLEM - mw2 MediaWiki Rendering on mw2 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 8191 bytes in 0.332 second response time [00:25:18] yeah, we have to fix that [00:25:36] PROBLEM - jobrunner1 MediaWiki Rendering on jobrunner1 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 8191 bytes in 0.433 second response time [00:25:40] PROBLEM - cp1 HTTP 4xx/5xx ERROR Rate on cp1 is WARNING: WARNING - NGINX Error Rate is 43% [00:25:44] You can never use a config in ServiceOptions or ConfigRegistry that does not exist or you will get fatals though with a confusing error about key 'config' [00:27:25] Also you can run a script I created to update static URLs in ManageWiki [00:27:40] RECOVERY - cp1 HTTP 4xx/5xx ERROR Rate on cp1 is OK: OK - NGINX Error Rate is 36% [00:27:48] oh nice [00:27:51] which is that? [00:28:06] I have to find it lol [00:28:22] Are we using private containers? [00:28:35] For private wikis [00:28:39] I think so? [00:29:22] https://github.com/WikiTideOrg/WikiTideMagic/blob/master/maintenance/updatePrivateAuthUrls.php then is one [00:29:43] It updates ManageWiki to use img_auth [00:30:11] It can be modified to update to replace all other static URLs also [00:30:45] !log [@test1] starting deploy of {'config': True, 'versions': ['']} to [test1] [00:30:46] !log [@test1] finished deploy of {'config': True, 'versions': ['']} to [test1] - SUCCESS in 0s [00:30:48] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [00:30:50] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [00:30:58] @cosmicalpha [00:31:00] PROBLEM - test1 MediaWiki Rendering on test1 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 684 bytes in 0.242 second response time [00:31:40] PROBLEM - cp1 HTTP 4xx/5xx ERROR Rate on cp1 is WARNING: WARNING - NGINX Error Rate is 52% [00:31:47] What about it? [00:32:18] Could this possibly work [00:33:01] I don't have any idea what rc max age has to do with this, unless you mean the query I created for that? [00:41:53] I am going to dinner be back in a few hours [00:47:40] RECOVERY - cp1 HTTP 4xx/5xx ERROR Rate on cp1 is OK: OK - NGINX Error Rate is 35% [00:59:25] [02WikiTideOrg/mw-config] 07AgentIsai pushed 031 commit to 03main [+0/-0/±1] 13https://github.com/WikiTideOrg/mw-config/compare/bfa18793ec22...99ef35ce4ee5 [00:59:26] [02WikiTideOrg/mw-config] 07AgentIsai 0399ef35c - End file maintenance [00:59:42] !log [agent@jobrunner1] starting deploy of {'pull': 'config', 'config': True, 'versions': ['1.40']} to [mw1, mw2, jobrunner1] [00:59:45] !log [agent@jobrunner1] finished deploy of {'pull': 'config', 'config': True, 'versions': ['1.40']} to [mw1, mw2, jobrunner1] - SUCCESS in 2s [00:59:45] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [00:59:48] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [01:00:27] WikiTideOrg/mw-config - AgentIsai the build has errored. [01:01:00] !log [@test1] starting deploy of {'config': True, 'versions': ['']} to [test1] [01:01:01] !log [@test1] finished deploy of {'config': True, 'versions': ['']} to [test1] - SUCCESS in 0s [01:01:03] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [01:01:06] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [01:03:08] I did this by the way [01:32:53] PROBLEM - cp1 HTTP 4xx/5xx ERROR Rate on cp1 is WARNING: WARNING - NGINX Error Rate is 40% [01:34:50] RECOVERY - cp1 HTTP 4xx/5xx ERROR Rate on cp1 is OK: OK - NGINX Error Rate is 37% [01:43:40] PROBLEM - cp1 HTTP 4xx/5xx ERROR Rate on cp1 is WARNING: WARNING - NGINX Error Rate is 47% [01:49:40] RECOVERY - cp1 HTTP 4xx/5xx ERROR Rate on cp1 is OK: OK - NGINX Error Rate is 34% [02:14:00] [02WikiTideOrg/WikiTideMagic] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/WikiTideMagic/compare/f66367312182...e9b893262400 [02:14:01] [02WikiTideOrg/WikiTideMagic] 07AgentIsai 03e9b8932 - Add Swift support Based off code from MirahezeMagic [02:16:14] [02WikiTideOrg/WikiTideMagic] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/WikiTideMagic/compare/e9b893262400...143a79d53b24 [02:16:15] [02WikiTideOrg/WikiTideMagic] 07AgentIsai 03143a79d - Update swift-lb URL [02:22:25] [02WikiTideOrg/puppet] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/9b05eaee920d...92f12ab77808 [02:22:27] [02WikiTideOrg/puppet] 07AgentIsai 0392f12ab - roundcubemail: Update logo URL [02:22:48] [02WikiTideOrg/puppet] 07AgentIsai pushed 031 commit to 03master [+0/-1/±3] 13https://github.com/WikiTideOrg/puppet/compare/92f12ab77808...add33d08b2f7 [02:22:50] [02WikiTideOrg/puppet] 07AgentIsai 03add33d0 - Remove S3 mentions from MediaWiki [02:24:15] [02WikiTideOrg/mw-config] 07AgentIsai pushed 031 commit to 03main [+0/-0/±1] 13https://github.com/WikiTideOrg/mw-config/compare/99ef35ce4ee5...7f5a4e6565ae [02:24:16] [02WikiTideOrg/mw-config] 07AgentIsai 037f5a4e6 - Update ImportDump command [02:25:06] WikiTideOrg/mw-config - AgentIsai the build has errored. [02:25:46] @agentisai if you set removed: true to an extension in mediawiki-repos, puppet can handle the removal. [02:26:42] [02WikiTideOrg/mw-config] 07AgentIsai pushed 031 commit to 03main [+0/-0/±1] 13https://github.com/WikiTideOrg/mw-config/compare/7f5a4e6565ae...9570cfb8c63d [02:26:43] [02WikiTideOrg/mw-config] 07AgentIsai 039570cfb - Remove unused $wgExtensionFunctions [02:27:34] [02WikiTideOrg/mw-config] 07Universal-Omega pushed 038 commits to 03remove-checks [+0/-0/±13] 13https://github.com/WikiTideOrg/mw-config/compare/6178b5745d74...dda9351c1ee5 [02:27:35] WikiTideOrg/mw-config - AgentIsai the build passed. [02:27:36] [02WikiTideOrg/mw-config] 07Universal-Omega 03dda9351 - Merge branch 'main' into remove-checks [02:27:37] [02mw-config] 07Universal-Omega synchronize pull request 03#11: Remove unnecessary checks - 13https://github.com/WikiTideOrg/mw-config/pull/11 [02:28:19] [02mw-config] 07Universal-Omega closed pull request 03#11: Remove unnecessary checks - 13https://github.com/WikiTideOrg/mw-config/pull/11 [02:28:20] [02WikiTideOrg/mw-config] 07Universal-Omega deleted branch 03remove-checks [02:28:22] [02WikiTideOrg/mw-config] 07Universal-Omega pushed 031 commit to 03main [+0/-0/±1] 13https://github.com/WikiTideOrg/mw-config/compare/9570cfb8c63d...e06598c60372 [02:28:23] [02WikiTideOrg/mw-config] 07Universal-Omega 03e06598c - Remove unnecessary checks [02:28:24] [02mw-config] 07Universal-Omega deleted branch 03remove-checks - 13https://github.com/WikiTideOrg/mw-config [02:28:26] WikiTideOrg/mw-config - Universal-Omega the build passed. [02:29:18] WikiTideOrg/mw-config - Universal-Omega the build passed. [02:30:52] !log [@test1] starting deploy of {'config': True, 'versions': ['']} to [test1] [02:30:53] !log [@test1] finished deploy of {'config': True, 'versions': ['']} to [test1] - SUCCESS in 0s [02:30:56] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [02:30:59] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [02:36:42] PROBLEM - cp1 HTTP 4xx/5xx ERROR Rate on cp1 is WARNING: WARNING - NGINX Error Rate is 44% [02:38:40] RECOVERY - cp1 HTTP 4xx/5xx ERROR Rate on cp1 is OK: OK - NGINX Error Rate is 39% [02:46:27] PROBLEM - cp1 HTTP 4xx/5xx ERROR Rate on cp1 is WARNING: WARNING - NGINX Error Rate is 44% [02:46:45] sus yo [02:47:43] !log [@jobrunner1] starting deploy of {'config': True, 'versions': ['']} to [mw1, mw2, jobrunner1] [02:47:45] !log [@jobrunner1] finished deploy of {'config': True, 'versions': ['']} to [mw1, mw2, jobrunner1] - SUCCESS in 2s [02:47:46] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [02:47:48] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [02:48:25] RECOVERY - cp1 HTTP 4xx/5xx ERROR Rate on cp1 is OK: OK - NGINX Error Rate is 36% [02:52:18] PROBLEM - cp1 HTTP 4xx/5xx ERROR Rate on cp1 is WARNING: WARNING - NGINX Error Rate is 44% [03:00:10] RECOVERY - cp1 HTTP 4xx/5xx ERROR Rate on cp1 is OK: OK - NGINX Error Rate is 37% [03:04:03] PROBLEM - cp1 HTTP 4xx/5xx ERROR Rate on cp1 is WARNING: WARNING - NGINX Error Rate is 42% [03:11:54] RECOVERY - cp1 HTTP 4xx/5xx ERROR Rate on cp1 is OK: OK - NGINX Error Rate is 37% [03:22:40] PROBLEM - cp1 HTTP 4xx/5xx ERROR Rate on cp1 is WARNING: WARNING - NGINX Error Rate is 40% [03:24:38] RECOVERY - cp1 HTTP 4xx/5xx ERROR Rate on cp1 is OK: OK - NGINX Error Rate is 30% [03:44:10] PROBLEM - cp1 HTTP 4xx/5xx ERROR Rate on cp1 is WARNING: WARNING - NGINX Error Rate is 45% [03:45:29] PROBLEM - m.wikitide.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:45:38] PROBLEM - wc.wikitide.org on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:45:57] PROBLEM - www.lgbtqia.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - www.lgbtqia.wiki reverse DNS resolves to vps-d267e842.vps.ovh.ca [03:47:24] PROBLEM - m.wikitide.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wikitide.org' expires in 12 day(s) (Mon 04 Dec 2023 02:58:50 PM GMT +0000). [03:47:37] PROBLEM - wc.wikitide.org on sslhost is WARNING: WARNING - Certificate 'wikitide.org' expires in 12 day(s) (Mon 04 Dec 2023 02:58:50 PM GMT +0000). [03:48:06] RECOVERY - cp1 HTTP 4xx/5xx ERROR Rate on cp1 is OK: OK - NGINX Error Rate is 15% [04:15:57] PROBLEM - www.lgbtqia.wiki - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['grannbo.ns.cloudflare.com.', 'rodrigo.ns.cloudflare.com.'], 'CNAME': 'cf-lb.wikitide.org.'} [04:53:17] [02WikiTideOrg/WikiTideMagic] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/WikiTideMagic/compare/143a79d53b24...089e578e70d2 [04:53:18] [02WikiTideOrg/WikiTideMagic] 07AgentIsai 03089e578 - Revert changes in order to support Swift [04:56:37] [02WikiTideOrg/puppet] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/add33d08b2f7...0c051f3241c6 [04:56:40] [02WikiTideOrg/puppet] 07AgentIsai 030c051f3 - Update sitemap command [04:56:58] !log [agent@jobrunner1] starting deploy of {'versions': ['1.40'], 'upgrade_extensions': ['WikiTideMagic']} to [mw1, mw2, jobrunner1] [04:56:59] !log [agent@jobrunner1] finished deploy of {'versions': ['1.40'], 'upgrade_extensions': ['WikiTideMagic']} to [mw1, mw2, jobrunner1] - SUCCESS in 0s [04:57:00] !log [agent@jobrunner1] starting deploy of {'versions': ['1.40'], 'upgrade_extensions': ['WikiTideMagic']} to [mw1, mw2, jobrunner1] [04:57:01] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [04:57:01] !log [agent@jobrunner1] finished deploy of {'versions': ['1.40'], 'upgrade_extensions': ['WikiTideMagic']} to [mw1, mw2, jobrunner1] - SUCCESS in 2s [04:57:04] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [04:57:07] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [04:57:10] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [04:57:14] @agentisai you removed version that wont work [04:57:46] nice [04:57:49] will fix [04:57:54] Thanks [04:58:20] [02WikiTideOrg/puppet] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/0c051f3241c6...7060973964cb [04:58:22] [02WikiTideOrg/puppet] 07AgentIsai 037060973 - runner: Add ${version} back to generateSitemapIndex [04:58:28] Yeah it gets you sometimes, we have a lot deferent config here due to multiversion and DB suffix stuff it gets confusing sometimes lol [04:59:44] lol yeah [05:00:51] heck yeah, just need my sitemap to be regen and indexed and it should be good to go- [05:02:47] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/7060973964cb...357f84de979c [05:02:47] [02WikiTideOrg/puppet] 07Universal-Omega 03357f84d - varnish: update static URL [05:03:49] !log [agent@jobrunner1] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases-wikitide.json /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/generateWikiTideSitemap.php (START) [05:03:51] !log [agent@jobrunner1] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases-wikitide.json /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/generateWikiTideSitemap.php (END - exit=2) [05:03:52] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [05:03:55] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [05:04:00] !log [agent@jobrunner1] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases-wikitide.json /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/extensions/WikiTideMagic/maintenance/generateWikiTideSitemap.php (START) [05:04:03] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [05:04:50] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±3] 13https://github.com/WikiTideOrg/puppet/compare/357f84de979c...8d7d3a8ee7d4 [05:04:51] [02WikiTideOrg/puppet] 07Universal-Omega 038d7d3a8 - Update static URL for X-Frame-Options [05:06:25] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/8d7d3a8ee7d4...212590d6102d [05:06:28] [02WikiTideOrg/puppet] 07Universal-Omega 03212590d - Update static redirect [05:08:03] [02WikiTideOrg/ErrorPages] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±11] 13https://github.com/WikiTideOrg/ErrorPages/compare/a3643432087c...07b1a26d0759 [05:08:05] !log [agent@jobrunner1] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases-wikitide.json /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/extensions/WikiTideMagic/maintenance/generateWikiTideSitemap.php (END - exit=0) [05:08:05] [02WikiTideOrg/ErrorPages] 07Universal-Omega 0307b1a26 - Update static URLs [05:08:07] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [05:09:05] [02WikiTideOrg/landing] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/landing/compare/573ac88c3b43...8c8ffa6d14cd [05:09:08] [02WikiTideOrg/landing] 07Universal-Omega 038c8ffa6 - Update static URL [05:11:35] @agentisai https://static.wikitide.net/lhmnwikitide/5/58/Footer.SN.xyz.svg permission issue [05:11:42] Or is it private? [05:13:48] Nope wiki is not private but only https://lhmn.wikitide.org/w/img_auth.php/5/58/Footer.SN.xyz.svg works [05:16:04] [02WikiTideOrg/mw-config] 07Universal-Omega pushed 031 commit to 03main [+0/-0/±3] 13https://github.com/WikiTideOrg/mw-config/compare/e06598c60372...b1631d8c1495 [05:16:05] [02WikiTideOrg/mw-config] 07Universal-Omega 03b1631d8 - Update static URLs [05:16:59] WikiTideOrg/mw-config - Universal-Omega the build passed. [05:18:10] !log [@jobrunner1] starting deploy of {'config': True, 'versions': ['']} to [mw1, mw2, jobrunner1] [05:18:12] !log [@jobrunner1] finished deploy of {'config': True, 'versions': ['']} to [mw1, mw2, jobrunner1] - SUCCESS in 2s [05:18:13] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [05:18:17] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [05:20:01] [02WikiTideOrg/dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±3] 13https://github.com/WikiTideOrg/dns/compare/c4a8c70930ee...b8ff6f5d8aa8 [05:20:04] [02WikiTideOrg/dns] 07Universal-Omega 03b8ff6f5 - Update static URLs [05:21:24] [02WikiTideOrg/WikiTideMagic] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/WikiTideMagic/compare/089e578e70d2...e3d41070af45 [05:21:25] [02WikiTideOrg/WikiTideMagic] 07Universal-Omega 03e3d4107 - Update static URL [05:23:10] [02WikiTideOrg/mw-config] 07Universal-Omega pushed 031 commit to 03main [+0/-0/±1] 13https://github.com/WikiTideOrg/mw-config/compare/b1631d8c1495...b43314292972 [05:23:13] [02WikiTideOrg/mw-config] 07Universal-Omega 03b433142 - Update static URL in wgTemplateStylesAllowedUrls [05:24:09] WikiTideOrg/mw-config - Universal-Omega the build passed. [05:30:54] [02WikiTideOrg/WikiTideMagic] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±30] 13https://github.com/WikiTideOrg/WikiTideMagic/compare/e3d41070af45...17dd09908545 [05:30:56] [02WikiTideOrg/WikiTideMagic] 07Universal-Omega 0317dd099 - Update IRC channels [05:31:38] !log [@test1] starting deploy of {'config': True, 'versions': ['']} to [test1] [05:31:39] !log [@test1] finished deploy of {'config': True, 'versions': ['']} to [test1] - SUCCESS in 0s [05:31:41] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [05:31:44] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [05:33:54] [02WikiTideOrg/dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/dns/compare/b8ff6f5d8aa8...3d4fb19ae1cf [05:33:57] [02WikiTideOrg/dns] 07Universal-Omega 033d4fb19 - Update mail hostname [05:35:09] [02WikiTideOrg/dns] 07Universal-Omega pushed 031 commit to 03master [+0/-1/±0] 13https://github.com/WikiTideOrg/dns/compare/3d4fb19ae1cf...f55c8d8622cd [05:35:11] [02WikiTideOrg/dns] 07Universal-Omega 03f55c8d8 - Delete zone not a part of WikiTide [05:36:04] [02WikiTideOrg/dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±9] 13https://github.com/WikiTideOrg/dns/compare/f55c8d8622cd...102009cab971 [05:36:06] [02WikiTideOrg/dns] 07Universal-Omega 03102009c - Update email [05:37:33] [02WikiTideOrg/mw-config] 07Universal-Omega pushed 031 commit to 03main [+0/-0/±1] 13https://github.com/WikiTideOrg/mw-config/compare/b43314292972...730436be160f [05:37:36] [02WikiTideOrg/mw-config] 07Universal-Omega 03730436b - Update wording [05:38:29] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/212590d6102d...28e9a5ebdd78 [05:38:31] [02WikiTideOrg/puppet] 07Universal-Omega 0328e9a5e - Update host [05:38:31] WikiTideOrg/mw-config - Universal-Omega the build passed. [05:40:30] [02WikiTideOrg/WikiTideMagic] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/WikiTideMagic/compare/17dd09908545...b3b95ff22b5a [05:40:31] [02WikiTideOrg/WikiTideMagic] 07Universal-Omega 03b3b95ff - Update repo [05:41:27] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/28e9a5ebdd78...8da1ed1642f0 [05:41:30] [02WikiTideOrg/puppet] 07Universal-Omega 038da1ed1 - Use WikiTide ErrorPages [05:42:20] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/8da1ed1642f0...2572d2bae38b [05:42:23] [02WikiTideOrg/puppet] 07Universal-Omega 032572d2b - Fix spacing [05:46:45] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/2572d2bae38b...be2809e35773 [05:46:46] [02WikiTideOrg/puppet] 07Universal-Omega 03be2809e - mwdeploy: remove WikiForge [05:48:08] WikiTideOrg/puppet - Universal-Omega the build passed. [05:48:24] !log [@jobrunner1] starting deploy of {'config': True, 'versions': ['']} to [mw1, mw2, jobrunner1] [05:48:26] !log [@jobrunner1] finished deploy of {'config': True, 'versions': ['']} to [mw1, mw2, jobrunner1] - SUCCESS in 2s [05:48:27] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [05:48:29] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [05:49:32] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/be2809e35773...d6205d94897c [05:49:35] [02WikiTideOrg/puppet] 07Universal-Omega 03d6205d9 - Use wikitide option [05:50:41] [02WikiTideOrg/WikiTideMagic] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/WikiTideMagic/compare/b3b95ff22b5a...24fcb5f00df6 [05:50:44] [02WikiTideOrg/WikiTideMagic] 07Universal-Omega 0324fcb5f - Fix Twitter URL [05:51:03] WikiTideOrg/puppet - Universal-Omega the build passed. [06:01:16] !log [@test1] starting deploy of {'config': True, 'versions': ['']} to [test1] [06:01:17] !log [@test1] finished deploy of {'config': True, 'versions': ['']} to [test1] - SUCCESS in 0s [06:01:19] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [06:01:22] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [06:21:40] PROBLEM - wc.wikitide.org on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:23:42] PROBLEM - wc.wikitide.org on sslhost is WARNING: WARNING - Certificate 'wikitide.org' expires in 12 day(s) (Mon 04 Dec 2023 02:58:50 PM GMT +0000). [06:25:44] @cosmicalpha I thought it autorenewed? ^ [06:25:57] I don't think wildcards do [06:26:40] I believe you configured the wikiforge.net wildcard to [06:27:17] No, I just made it work with ssl-certificate not auto renew, it needs acme validation to renew wildcards [07:02:04] RECOVERY - test1 JobChron Service on test1 is OK: PROCS OK: 1 process with args 'redisJobChronService' [07:06:04] PROBLEM - test1 JobChron Service on test1 is CRITICAL: PROCS CRITICAL: 0 processes with args 'redisJobChronService' [07:38:00] [02WikiTideOrg/ssl] 07WikiTideSSL pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/WikiTideOrg/ssl/compare/1f212e28c564...5513d148addf [07:38:03] [02WikiTideOrg/ssl] 07WikiTideSSL 035513d14 - Bot: Adding SSL cert for wiki.burning.art Certificate committed by reception [07:42:48] PROBLEM - cp4 Puppet on cp4 is UNKNOWN: NRPE: Unable to read output [07:48:59] PROBLEM - cp6 Puppet on cp6 is UNKNOWN: NRPE: Unable to read output [07:51:28] PROBLEM - jobrunner1 Puppet on jobrunner1 is UNKNOWN: NRPE: Unable to read output [07:53:17] PROBLEM - cp3 Puppet on cp3 is UNKNOWN: NRPE: Unable to read output [07:55:10] PROBLEM - cp2 Puppet on cp2 is UNKNOWN: NRPE: Unable to read output [07:55:21] PROBLEM - cp5 Puppet on cp5 is UNKNOWN: NRPE: Unable to read output [07:57:32] PROBLEM - cp1 Puppet on cp1 is UNKNOWN: NRPE: Unable to read output [08:05:31] PROBLEM - mw2 Puppet on mw2 is UNKNOWN: NRPE: Unable to read output [08:08:52] RECOVERY - cp4 Puppet on cp4 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [08:10:36] PROBLEM - mw1 Puppet on mw1 is UNKNOWN: NRPE: Unable to read output [08:16:58] RECOVERY - cp6 Puppet on cp6 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [08:19:28] RECOVERY - jobrunner1 Puppet on jobrunner1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [08:21:10] RECOVERY - cp2 Puppet on cp2 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [08:21:17] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [08:23:21] RECOVERY - cp5 Puppet on cp5 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [08:27:32] RECOVERY - cp1 Puppet on cp1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [08:33:31] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [08:38:36] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [09:06:46] PROBLEM - cp3 Disk Space on cp3 is CRITICAL: DISK CRITICAL - free space: / 1148MiB (5% inode=94%); [09:38:31] PROBLEM - cp1 HTTP 4xx/5xx ERROR Rate on cp1 is WARNING: WARNING - NGINX Error Rate is 43% [09:40:29] RECOVERY - cp1 HTTP 4xx/5xx ERROR Rate on cp1 is OK: OK - NGINX Error Rate is 37% [10:01:59] PROBLEM - cp1 HTTP 4xx/5xx ERROR Rate on cp1 is WARNING: WARNING - NGINX Error Rate is 45% [10:09:50] RECOVERY - cp1 HTTP 4xx/5xx ERROR Rate on cp1 is OK: OK - NGINX Error Rate is 39% [10:32:04] RECOVERY - test1 JobChron Service on test1 is OK: PROCS OK: 1 process with args 'redisJobChronService' [10:36:04] PROBLEM - test1 JobChron Service on test1 is CRITICAL: PROCS CRITICAL: 0 processes with args 'redisJobChronService' [12:12:28] !log [agent@jobrunner1] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/importImages.php --wiki=simpleelectronicswikitide images --search-recursive (START) [12:12:30] !log [agent@jobrunner1] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/importImages.php --wiki=simpleelectronicswikitide images --search-recursive (END - exit=256) [12:13:21] !log [agent@jobrunner1] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/importImages.php --wiki=simpleelectronicswikitide images --search-recursively (START) [12:13:22] !log [agent@jobrunner1] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/importImages.php --wiki=simpleelectronicswikitide images --search-recursively (END - exit=0) [12:13:34] !log [agent@jobrunner1] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/importImages.php --wiki=simpleelectronicswikitide images --search-recursively (START) [12:13:40] !log [agent@jobrunner1] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/importImages.php --wiki=simpleelectronicswikitide images --search-recursively (END - exit=0) [12:15:19] !log [agent@jobrunner1] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/importImages.php --wiki=simpleelectronicswikitide images --search-recursively (START) [12:15:20] !log [agent@jobrunner1] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/importImages.php --wiki=simpleelectronicswikitide images --search-recursively (END - exit=0) [12:15:28] !log [agent@jobrunner1] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/importImages.php --wiki=simpleelectronicswikitide images --search-recursively (START) [12:17:06] !log [agent@jobrunner1] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/importImages.php --wiki=simpleelectronicswikitide images --search-recursively (END - exit=2) [12:17:20] !log [agent@jobrunner1] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/importImages.php --wiki=sdiywikitide images --search-recursively (START) [12:21:02] !log [agent@jobrunner1] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/importImages.php --wiki=sdiywikitide images --search-recursively (END - exit=0) [12:23:38] !log [agent@jobrunner1] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/importImages.php --wiki=scruffywikitide images --search-recursively (START) [12:25:17] !log [agent@jobrunner1] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/importImages.php --wiki=scruffywikitide images --search-recursively (END - exit=0) [12:50:32] !log [agent@jobrunner1] starting deploy of {'force': True, 'versions': ['1.40'], 'upgrade_extensions': ['WikiTideMagic']} to [mw1, mw2, jobrunner1] [12:50:33] !log [agent@jobrunner1] finished deploy of {'force': True, 'versions': ['1.40'], 'upgrade_extensions': ['WikiTideMagic']} to [mw1, mw2, jobrunner1] - SUCCESS in 0s [12:50:34] !log [agent@jobrunner1] starting deploy of {'force': True, 'versions': ['1.40'], 'upgrade_extensions': ['WikiTideMagic']} to [mw1, mw2, jobrunner1] [12:50:35] !log [agent@jobrunner1] finished deploy of {'force': True, 'versions': ['1.40'], 'upgrade_extensions': ['WikiTideMagic']} to [mw1, mw2, jobrunner1] - SUCCESS in 2s [12:51:13] !log [agent@jobrunner1] starting deploy of {'force': True, 'versions': ['1.41'], 'upgrade_extensions': ['WikiTideMagic']} to [mw1, mw2, jobrunner1] [12:51:13] !log [agent@jobrunner1] finished deploy of {'force': True, 'versions': ['1.41'], 'upgrade_extensions': ['WikiTideMagic']} to [mw1, mw2, jobrunner1] - SUCCESS in 0s [12:51:14] !log [agent@jobrunner1] starting deploy of {'force': True, 'versions': ['1.41'], 'upgrade_extensions': ['WikiTideMagic']} to [mw1, mw2, jobrunner1] [12:51:16] !log [agent@jobrunner1] finished deploy of {'force': True, 'versions': ['1.41'], 'upgrade_extensions': ['WikiTideMagic']} to [mw1, mw2, jobrunner1] - SUCCESS in 3s [12:59:19] PROBLEM - m.wikitide.org - LetsEncrypt on sslhost is CRITICAL: connect to address m.wikitide.org and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [12:59:25] PROBLEM - wc.wikitide.org on sslhost is CRITICAL: connect to address wikiforge.org and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [13:00:06] PROBLEM - mason.sagan4.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - mason.sagan4.org reverse DNS resolves to vps-d267e842.vps.ovh.ca [13:01:09] PROBLEM - beta.sagan4.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - beta.sagan4.org reverse DNS resolves to vps-d267e842.vps.ovh.ca [13:01:19] PROBLEM - m.wikitide.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wikitide.org' expires in 12 day(s) (Mon 04 Dec 2023 02:58:50 PM GMT +0000). [13:01:25] PROBLEM - wc.wikitide.org on sslhost is WARNING: WARNING - Certificate 'wikitide.org' expires in 12 day(s) (Mon 04 Dec 2023 02:58:50 PM GMT +0000). [13:03:09] PROBLEM - wikitide.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wikitide.org reverse DNS resolves to vps-d267e842.vps.ovh.ca [13:11:49] [02WikiTideOrg/dns] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/dns/compare/102009cab971...5506d08a8a3e [13:11:50] [02WikiTideOrg/dns] 07AgentIsai 035506d08 - Add acme challenge [13:12:26] [02WikiTideOrg/ssl] 07WikiTideSSL pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/ssl/compare/5513d148addf...91a30207f852 [13:12:29] [02WikiTideOrg/ssl] 07WikiTideSSLBot 0391a3020 - Bot: Update SSL cert for wikitide.org [13:15:19] RECOVERY - m.wikitide.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wikitide.org' will expire on Tue 20 Feb 2024 12:12:20 PM GMT +0000. [13:15:29] RECOVERY - wc.wikitide.org on sslhost is OK: OK - Certificate 'wikitide.org' will expire on Tue 20 Feb 2024 12:12:20 PM GMT +0000. [13:15:50] RECOVERY - puppet1 Puppet on puppet1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:19:16] [02WikiTideOrg/ssl] 07WikiTideSSL pushed 031 commit to 03master [+1/-0/±0] 13https://github.com/WikiTideOrg/ssl/compare/91a30207f852...7b7aade64c8e [13:19:18] [02WikiTideOrg/ssl] 07WikiTideSSLBot 037b7aade - Bot: Update SSL cert for *.wikitide.org [13:19:56] [02WikiTideOrg/dns] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/dns/compare/5506d08a8a3e...860554ff7413 [13:19:58] [02WikiTideOrg/dns] 07AgentIsai 03860554f - Update wikitide.org [13:21:21] [02WikiTideOrg/ssl] 07AgentIsai pushed 031 commit to 03master [+0/-1/±0] 13https://github.com/WikiTideOrg/ssl/compare/7b7aade64c8e...6b39517be069 [13:21:22] [02WikiTideOrg/ssl] 07AgentIsai 036b39517 - Delete certificates/*.wikitide.org.crt [13:24:03] [02WikiTideOrg/ssl] 07WikiTideSSL pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/ssl/compare/6b39517be069...52bfe7f4f629 [13:24:04] [02WikiTideOrg/ssl] 07WikiTideSSLBot 0352bfe7f - Update wikitide.org cert [13:26:54] RECOVERY - wikitide.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wikitide.org' will expire on Tue 20 Feb 2024 12:20:10 PM GMT +0000. [13:30:06] PROBLEM - mason.sagan4.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns59.domaincontrol.com.', 'ns60.domaincontrol.com.'], 'CNAME': 'cf-lb.wikitide.org.'} [13:31:09] PROBLEM - beta.sagan4.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns59.domaincontrol.com.', 'ns60.domaincontrol.com.'], 'CNAME': 'cf-lb.wikitide.org.'} [13:32:48] RECOVERY - wikitide.org - reverse DNS on sslhost is OK: SSL OK - wikitide.org reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK [13:34:56] RECOVERY - www.wikitide.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wikitide.org' will expire on Tue 20 Feb 2024 12:20:10 PM GMT +0000. [17:01:59] [02WikiTideOrg/landing] 07Reception123 pushed 031 commit to 03Reception123-patch-1 [+1/-0/±1] 13https://github.com/WikiTideOrg/landing/compare/e5ca75f83354...262275b33183 [17:02:02] [02WikiTideOrg/landing] 07Reception123 03262275b - add i18n lines [17:02:05] [02landing] 07Reception123 synchronize pull request 03#2: make landing page translatable per T71 - 13https://github.com/WikiTideOrg/landing/pull/2 [17:03:12] [02WikiTideOrg/landing] 07Reception123 pushed 031 commit to 03Reception123-patch-1 [+0/-0/±1] 13https://github.com/WikiTideOrg/landing/compare/262275b33183...a62953c4d2f2 [17:03:15] [02WikiTideOrg/landing] 07Reception123 03a62953c - not here [17:03:17] [02landing] 07Reception123 synchronize pull request 03#2: make landing page translatable per T71 - 13https://github.com/WikiTideOrg/landing/pull/2 [17:08:49] [02WikiTideOrg/landing] 07Reception123 pushed 031 commit to 03Reception123-patch-1 [+1/-0/±0] 13https://github.com/WikiTideOrg/landing/compare/a62953c4d2f2...7cdf00d9eb73 [17:08:50] [02WikiTideOrg/landing] 07Reception123 037cdf00d - add qqq [17:08:52] [02landing] 07Reception123 synchronize pull request 03#2: make landing page translatable per T71 - 13https://github.com/WikiTideOrg/landing/pull/2 [17:44:11] hmm, if I try and save an edit to https://meta.wikitide.org/wiki/IRC I get a 500 error which doesn't happen on other pages [17:44:34] I wonder if it could be related to the logbot issue [17:46:05] I do also get 500 trying to edit the SAL [17:46:23] strange that it's only on specific pages though [18:14:57] [02WikiTideOrg/mw-config] 07AgentIsai pushed 031 commit to 03main [+0/-0/±1] 13https://github.com/WikiTideOrg/mw-config/compare/730436be160f...2df1845ba28b [18:14:59] [02WikiTideOrg/mw-config] 07AgentIsai 032df1845 - Set SCHEMA_COMPAT_READ_NEW for comment table [18:15:27] !log [agent@jobrunner1] starting deploy of {'pull': 'config', 'config': True, 'versions': ['1.40']} to [mw1, mw2, jobrunner1] [18:15:29] !log [agent@jobrunner1] finished deploy of {'pull': 'config', 'config': True, 'versions': ['1.40']} to [mw1, mw2, jobrunner1] - SUCCESS in 2s [18:15:53] WikiTideOrg/mw-config - AgentIsai the build passed. [18:26:56] [02WikiTideOrg/mw-config] 07AgentIsai pushed 031 commit to 03main [+0/-0/±1] 13https://github.com/WikiTideOrg/mw-config/compare/2df1845ba28b...4425a4549c3a [18:26:57] [02WikiTideOrg/mw-config] 07AgentIsai 034425a45 - Use SCHEMA_COMPAT_NEW [18:26:59] !log [agent@jobrunner1] starting deploy of {'pull': 'config', 'config': True, 'versions': ['1.40']} to [mw1, mw2, jobrunner1] [18:27:02] !log [agent@jobrunner1] finished deploy of {'pull': 'config', 'config': True, 'versions': ['1.40']} to [mw1, mw2, jobrunner1] - SUCCESS in 2s [18:28:38] WikiTideOrg/mw-config - AgentIsai the build passed. [18:30:54] !log [@test1] starting deploy of {'config': True, 'versions': ['']} to [test1] [18:30:55] !log [@test1] finished deploy of {'config': True, 'versions': ['']} to [test1] - SUCCESS in 0s [18:31:12] [02WikiTideOrg/mw-config] 07AgentIsai pushed 031 commit to 03main [+0/-0/±1] 13https://github.com/WikiTideOrg/mw-config/compare/4425a4549c3a...97ae389937d0 [18:31:14] [02WikiTideOrg/mw-config] 07AgentIsai 0397ae389 - Fix value [18:31:16] !log [agent@jobrunner1] starting deploy of {'pull': 'config', 'config': True, 'versions': ['1.40']} to [mw1, mw2, jobrunner1] [18:31:18] !log [agent@jobrunner1] finished deploy of {'pull': 'config', 'config': True, 'versions': ['1.40']} to [mw1, mw2, jobrunner1] - SUCCESS in 2s [18:32:06] WikiTideOrg/mw-config - AgentIsai the build passed. [18:32:38] !log [agent@jobrunner1] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases-wikitide.json /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/migrateRevisionCommentTemp.php --force (START) [18:42:41] !log [agent@jobrunner1] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases-wikitide.json /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/migrateRevisionCommentTemp.php --force (END - exit=0) [18:56:30] !log test [18:57:31] Hm yeah it still errors but I can edit IRC now (or preview it where I could not before) [18:58:27] Oh it has to do with cp [18:58:32] @Agent [18:58:53] @agentisai [18:59:07] oh [18:59:12] but why? [19:00:12] My guess is because cp1 has no storage to write cache [19:00:40] But could be wrong [19:00:46] It is cp1 it happens on though [19:00:52] cp2 is fine. [19:01:23] !log [@test1] starting deploy of {'config': True, 'versions': ['']} to [test1] [19:01:24] !log [@test1] finished deploy of {'config': True, 'versions': ['']} to [test1] - SUCCESS in 0s [19:01:29] Hence why it worked for me one minute and not the next as my proxy was going one minute and not the next. [19:01:42] @agentisai [19:02:26] let me restart cp1 to flush out the cache then [19:02:31] @cosmicalpha thoughts? ^ [19:02:33] are we talking about the errors when editing [19:02:39] yes [19:03:08] I did some experimenting and it seems to start around 5000 bytes [19:03:11] Is cp1 an NVMe server? [19:03:18] Its on cloud1 [19:03:27] funnily enough, it has the most space of all the cp*'s [19:03:36] [1/3] > (hist) ‎Shimmering Marephasmatises ‎[4,997 bytes] [19:03:36] [2/3] > (hist) ‎Week 27/Ecosystem/Vonnegut ‎[5,009 bytes] [19:03:36] [3/3] of these two it's the second that has the error when I tested [19:04:04] does cloud1 have NVMe disks then @agentisai ? or just hdd? [19:04:10] SSD [19:05:19] Okay, does it have NVMe SSD or regular SSD? NVMe is a still a type of SSD but more efficient for cache servers in particular. [19:05:28] Just wondering lol [19:05:50] it's a regular SSD [19:05:54] not NVMe [19:05:58] Ah okay thanks [19:51:46] PROBLEM - cp5 Disk Space on cp5 is CRITICAL: DISK CRITICAL - free space: / 1148MiB (5% inode=94%); [19:55:22] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±6] 13https://github.com/WikiTideOrg/puppet/compare/d6205d94897c...35a42b5a6f57 [19:55:24] [02WikiTideOrg/puppet] 07Universal-Omega 0335a42b5 - cp*: reduce caching by 5GB [19:59:26] OH I know why Varnish keeps having issues with storage! [20:02:04] RECOVERY - test1 JobChron Service on test1 is OK: PROCS OK: 1 process with args 'redisJobChronService' [20:05:43] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/35a42b5a6f57...58ec98106aab [20:05:44] [02WikiTideOrg/puppet] 07Universal-Omega 0358ec981 - reload-vcl: fix auto discard [20:06:04] PROBLEM - test1 JobChron Service on test1 is CRITICAL: PROCS CRITICAL: 0 processes with args 'redisJobChronService' [20:12:14] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+1/-6/±3] 13https://github.com/WikiTideOrg/puppet/compare/58ec98106aab...da3373af2622 [20:12:17] [02WikiTideOrg/puppet] 07Universal-Omega 03da3373a - Varnish: cleanup and remove stunnel [20:13:06] WikiTideOrg/puppet - Universal-Omega the build passed. [20:13:14] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/da3373af2622...ca85601b2cd4 [20:13:17] [02WikiTideOrg/puppet] 07Universal-Omega 03ca85601 - Fix spacing [20:29:32] PROBLEM - cp1 Puppet on cp1 is UNKNOWN: NRPE: Unable to read output [20:52:15] [02WikiTideOrg/puppet] 07Universal-Omega created branch 03overhaul-varnish 13https://github.com/WikiTideOrg/puppet/commit/ca85601b2cd42ac54a47003313a9f9da203c668a [20:52:16] [02puppet] 07Universal-Omega created branch 03overhaul-varnish - 13https://github.com/WikiTideOrg/puppet [20:58:44] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03overhaul-varnish [+0/-0/±11] 13https://github.com/WikiTideOrg/puppet/compare/ca85601b2cd4...b9448f412d18 [20:58:47] [02WikiTideOrg/puppet] 07Universal-Omega 03b9448f4 - Overhaul Varnish [20:59:02] [02puppet] 07Universal-Omega opened pull request 03#16: Overhaul Varnish - 13https://github.com/WikiTideOrg/puppet/pull/16 [21:00:58] [02puppet] 07Universal-Omega synchronize pull request 03#16: Overhaul Varnish - 13https://github.com/WikiTideOrg/puppet/pull/16 [21:01:00] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03overhaul-varnish [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/b9448f412d18...3f74c2864690 [21:01:02] [02WikiTideOrg/puppet] 07Universal-Omega 033f74c28 - - [21:15:02] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±52] 13https://github.com/WikiTideOrg/puppet/compare/ca85601b2cd4...266a617d2ff1 [21:15:04] [02WikiTideOrg/puppet] 07Universal-Omega 03266a617 - Replace deprecated ensure_packages with stdlib::ensure_packages [21:21:32] [02WikiTideOrg/puppet] 07Universal-Omega pushed 032 commits to 03overhaul-varnish [+0/-0/±12] 13https://github.com/WikiTideOrg/puppet/compare/3f74c2864690...246200075cd7 [21:21:33] [02WikiTideOrg/puppet] 07Universal-Omega 03ccfd97c - Overhaul Varnish [21:21:34] [02WikiTideOrg/puppet] 07Universal-Omega 032462000 - - [21:21:36] [02puppet] 07Universal-Omega synchronize pull request 03#16: Overhaul Varnish - 13https://github.com/WikiTideOrg/puppet/pull/16 [21:25:43] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/266a617d2ff1...399da4479760 [21:25:45] [02WikiTideOrg/puppet] 07Universal-Omega 03399da44 - Fix check_puppet_run [21:32:27] RECOVERY - mon1 Puppet on mon1 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [21:35:22] PROBLEM - mon1 Check correctness of the icinga configuration on mon1 is CRITICAL: Icinga configuration contains errors [21:39:31] PROBLEM - mem1 Puppet on mem1 is UNKNOWN: NRPE: Unable to read output [21:39:38] RECOVERY - mon1 Check correctness of the icinga configuration on mon1 is OK: Icinga configuration is correct [21:41:25] PROBLEM - cp4 Puppet on cp4 is UNKNOWN: NRPE: Unable to read output [21:42:10] PROBLEM - mon1 Puppet on mon1 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 10 minutes ago with 0 failures [21:49:16] PROBLEM - cp6 Puppet on cp6 is UNKNOWN: NRPE: Unable to read output [21:50:47] pdns recursor as tried to restart 8696610 times on cloud1??? @agentisai [21:51:03] I'm not surprised [21:51:13] It tries to restart every second because something is binding on port 53 [21:53:43] PROBLEM - cp3 Puppet on cp3 is UNKNOWN: NRPE: Unable to read output [21:53:45] PROBLEM - cp2 Puppet on cp2 is UNKNOWN: NRPE: Unable to read output [21:55:08] PROBLEM - ns1 Puppet on ns1 is UNKNOWN: NRPE: Unable to read output [21:58:43] PROBLEM - bast1 Puppet on bast1 is UNKNOWN: NRPE: Unable to read output [21:59:37] @agentisai of course it is, cloud1 has two dns resolvers installed [22:10:25] !log [universalomega@cloud1] stop unbound [22:10:35] !log [universalomega@cloud1] sudo chattr -i /etc/resolv.conf [22:11:11] RECOVERY - cloud1 Puppet on cloud1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [22:12:06] Fixed @agentisai [22:17:34] PROBLEM - wikitide.net - LetsEncrypt on sslhost is CRITICAL: connect to address wikitide.net and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [22:18:54] PROBLEM - cp1 Disk Space on cp1 is WARNING: DISK WARNING - free space: / 5103MiB (9% inode=98%); [22:20:33] RECOVERY - cp1 Puppet on cp1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [22:20:50] PROBLEM - wikitide.net - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wikitide.net' expires in 11 day(s) (Mon 04 Dec 2023 03:06:37 PM GMT +0000). [22:22:36] [02WikiTideOrg/puppet] 07Universal-Omega pushed 032 commits to 03overhaul-varnish [+0/-0/±12] 13https://github.com/WikiTideOrg/puppet/compare/246200075cd7...5377c378730d [22:22:39] [02WikiTideOrg/puppet] 07Universal-Omega 0323ad26c - Overhaul Varnish [22:22:40] [02WikiTideOrg/puppet] 07Universal-Omega 035377c37 - - [22:22:41] [02puppet] 07Universal-Omega synchronize pull request 03#16: Overhaul Varnish - 13https://github.com/WikiTideOrg/puppet/pull/16 [22:26:38] PROBLEM - lophocmatngu.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - lophocmatngu.wiki reverse DNS resolves to vps-d267e842.vps.ovh.ca [22:27:54] [02puppet] 07Universal-Omega synchronize pull request 03#16: Overhaul Varnish - 13https://github.com/WikiTideOrg/puppet/pull/16 [22:27:55] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03overhaul-varnish [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/5377c378730d...5d2f4ff90b6e [22:27:58] [02WikiTideOrg/puppet] 07Universal-Omega 035d2f4ff - Remove mon1 for now [22:35:51] [02puppet] 07Universal-Omega synchronize pull request 03#16: Overhaul Varnish - 13https://github.com/WikiTideOrg/puppet/pull/16 [22:35:54] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03overhaul-varnish [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/5d2f4ff90b6e...5923a21321a6 [22:35:55] [02WikiTideOrg/puppet] 07Universal-Omega 035923a21 - Fix [22:39:15] [02puppet] 07Universal-Omega synchronize pull request 03#16: Overhaul Varnish - 13https://github.com/WikiTideOrg/puppet/pull/16 [22:39:17] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03overhaul-varnish [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/5923a21321a6...858fdfff70f7 [22:39:20] [02WikiTideOrg/puppet] 07Universal-Omega 03858fdff - Use tabs [22:41:22] [02puppet] 07Universal-Omega synchronize pull request 03#16: Overhaul Varnish - 13https://github.com/WikiTideOrg/puppet/pull/16 [22:41:24] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03overhaul-varnish [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/858fdfff70f7...7485478187ea [22:41:27] [02WikiTideOrg/puppet] 07Universal-Omega 037485478 - Tabbing [22:41:58] [02puppet] 07Universal-Omega closed pull request 03#16: Overhaul Varnish - 13https://github.com/WikiTideOrg/puppet/pull/16 [22:42:00] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±11] 13https://github.com/WikiTideOrg/puppet/compare/399da4479760...dac39e993d3e [22:42:02] [02WikiTideOrg/puppet] 07Universal-Omega 03dac39e9 - Overhaul Varnish [22:42:04] [02WikiTideOrg/puppet] 07Universal-Omega deleted branch 03overhaul-varnish [22:42:07] [02puppet] 07Universal-Omega deleted branch 03overhaul-varnish - 13https://github.com/WikiTideOrg/puppet [22:46:24] RECOVERY - cp3 Disk Space on cp3 is OK: DISK OK - free space: / 6083MiB (31% inode=94%); [22:47:49] RECOVERY - lophocmatngu.wiki - reverse DNS on sslhost is OK: SSL OK - lophocmatngu.wiki reverse DNS resolves to cp1.wikitide.net - CNAME FLAT [22:48:15] RECOVERY - cp6 Disk Space on cp6 is OK: DISK OK - free space: / 6386MiB (33% inode=94%); [22:51:32] !log test [22:51:34] RECOVERY - cp2 Disk Space on cp2 is OK: DISK OK - free space: / 6340MiB (33% inode=94%); [22:51:35] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [22:51:42] There we go [22:52:14] RECOVERY - cp5 Disk Space on cp5 is OK: DISK OK - free space: / 6257MiB (32% inode=94%); [22:53:36] RECOVERY - cp4 Disk Space on cp4 is OK: DISK OK - free space: / 6347MiB (33% inode=94%); [23:31:08] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/dac39e993d3e...d3e0c17b5848 [23:31:10] [02WikiTideOrg/puppet] 07Universal-Omega 03d3e0c17 - Fix [23:59:01] PROBLEM - mw2 MediaWiki Rendering on mw2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:59:37] PROBLEM - jobrunner1 MediaWiki Rendering on jobrunner1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:59:47] PROBLEM - mw1 MediaWiki Rendering on mw1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:59:54] PROBLEM - db1 MariaDB on db1 is UNKNOWN: check_mysql: Invalid hostname/address - db1.wikitide.netUsage: check_mysql [-d database] [-H host] [-P port] [-s socket] [-u user] [-p password] [-S] [-l] [-a cert] [-k key] [-C ca-cert] [-D ca-dir] [-L ciphers] [-f optfile] [-g group]