[00:17:35] I'm trying to access the new instance but is denied per publickey. Same publickey is used on the first instance created and can access without problems [00:17:43] Any advice? [00:19:36] Globgor: my first advice would be to check the "Log" tab in Horizon for the new instance and make sure that it has finished its first puppet run sucessfully. [00:21:23] If puppet is still running just check back later. If it failed hard in the first run, then the we can either try to debug that failure or just delete the instance and try again. [00:24:02] bd808, this? The last Puppet run was at Sun Aug 15 13:18:01 UTC 2021 [1m(6343 minutes ago)[0m. [00:24:37] hmmm... "6343 minutes ago" sounds bad. It should happen every 30 minutes on average [00:25:25] Globgor: what is the hostname? I can try looking at what's happening via magic super powers [00:25:52] mars.wikisp.eqiad1.wikimedia.cloud [00:26:01] * bd808 pokes [00:26:16] thx [00:27:28] I tried forcing a puppet run and see "Error: request https://puppetmaster.cloudinfra.wmflabs.org:8140//puppet/v3/node/buildvm-f575acf4-c566-4258-9a5f-20b9ec80ccad.admin.eqiad1.wikimedia.cloud failed: SSL_read: sslv3 alert certificate revoked" [00:28:15] The puppet TLS cert for this instance has been marked as revoked on the puppetmaster [00:28:58] Globgor: is this the second or third instance you made with the name "mars" by any chance? [00:29:12] yup, is it [00:29:40] aaand, that's the problem, right? [00:30:06] ok, so my sense making is that when you deleted the instance prior to this one it had not been cleaned up in the puppetmaster before you made the next one. and that led to bad state of the TLS auth certs that let Puppet work [00:30:30] Sounds legit [00:31:25] and yes, we recommend hostnames that are universally unique. These are "cattle not pets" machines. Names like "wikisp-backend-01" are less likely to have problems than "mars" [00:32:59] Globgor: I think the easiest thing to do with this instance is delete it and make a new one with a new name. The first puppet run never actually worked so the instance is in pretty rough shape. It is technically possible to fix but not really worth the effort. [00:33:47] no prob, I'll delete and make one. Is less painful :) [00:33:52] thx bd808! [00:33:55] yw [00:40:46] !log wikisp Destroyed mars instance [00:40:48] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikisp/SAL [00:49:23] !log wikisp Bacchus instance (Limesurvey + Moodle) launched [00:49:25] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikisp/SAL [14:30:00] This morning I got an alert about puppet, after running run-puppet-agent I get this data: https://dpaste.org/x3LJ [14:30:16] Very headache the lamp in puppet heh [14:31:40] lamp role* [14:34:27] Globgor: run "/usr/sbin/a2enmod php7.0" yourself manually and see what the issue is [14:34:40] in this case puppet only runs that command [14:35:41] maybe because you are on bullseye and the role was made for older distro versions with PHP 7.0 [14:36:32] yea, I bet that is it, the code has this: [14:36:33] 23 $apache_php_module = debian::codename::eq('buster') ? { [14:36:33] 24 true => 'php7.3', [14:36:33] 25 default => 'php7.0', [14:36:53] ^ so you are not on buster, either older or newer than buster, but would work on buster [14:37:16] you can either switch to that for now.. or make us add a case for bullseye there [14:37:58] I gotta go, but feel free to remind me to fix it ..and let me know which distro you are on [14:39:25] I'm in bullseye, and got headache yesterday when I installed moodle. The role is settled across both instances, but only one of them used it. Safely remove puppet role and add on instance? [14:47:20] mutante, so, I'll need to switch buster instead of bullseye? [14:56:51] Globgor: if the instance doesn't need that puppet role then remove it. If it does, someone can send a patch to fix it. [14:58:48] The first instance doesn't need it, but the second was installed lamp via puppet role [15:05:56] Globgor: I can send a patch [15:07:59] RhinosF1, thanks! I'm going to rebuild the second instance [15:14:19] https://gerrit.wikimedia.org/r/c/operations/puppet/+/713863 [15:14:21] mutante: ^ [15:16:33] Jenkins seems to like it but need someone to merge it [15:16:55] RhinosF1: codesearch finds at least two more files that could maybe be updated together with this https://codesearch.wmcloud.org/search/?q=php_version%20fact [15:17:43] Lucas_WMDE: not sure how much rack tables is used [15:18:17] But the other I can look at if someone gives me a +1 [15:18:25] Would rather not replace more if it's wrong [15:19:03] I rebuilded the second instance, so I'll need to manually install lamp or wait the patch? [15:21:07] (I don’t know enough puppet to actually review the patch, sorry) [15:22:10] Globgor: wait for the patch [15:22:17] Unless you have a local puppet master [15:22:30] Nou, I'll wait for the patch then [15:22:37] It should just work once someone puppet merged [15:56:04] !log wikisp Bacchus instance rebuilded [15:56:07] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikisp/SAL [18:42:44] !log tools rebuilding php74-sssd/{base,web} to use composer 2 [18:42:49] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [19:10:08] !log tools rebuilding node12-sssd/{base,web} to use debian packaged npm 7 [19:10:12] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [19:46:41] !log tools.notwikilambda ran "git -C extensions/WikiLambda/ submodule update --init --recursive", apparently the function-schemata submodule was missing? [19:46:44] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.notwikilambda/SAL [19:55:07] !log tools.notwikilambda ran "php extensions/WikiLambda/maintenance/reloadBuiltinData.php" at ArthurPSmith’s request [19:55:09] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.notwikilambda/SAL [23:19:53] !log toolsbeta tried renewing all the certs to get certs working again in kubernetes [23:19:55] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL