[07:22:52] <_joe_> inflatador: I guess no one pointed you to the SRE onboarding chats. Some of those concepts are covered in the onboarding videos too [10:15:00] elukey: looks like https://github.com/wikimedia/cloud-instance-puppet/commit/43da43ece6560b0a8c598120a554105c53c03685 broke puppet on deployment-kafka-logging01, I am seeing "Class[Java]: expects a value for parameter 'java_packages'" [10:15:46] taavi: o/ I know my bad, I am waiting for John to discuss what needs to be done. I am planning to fix it today, if it is a problem I can revert the change now [10:16:16] oh thanks, no hurry but just wanted to make sure you were aware since I didn't see you logged in there [10:17:05] yes yes I am, it is to move kafka logging to the PKI cloud infra, the specific bit that I needed was the new jks truststore that doesn't want to collaborate :D [10:52:00] jbond: stupid question, If I want to add a Hosts footer to my commit to make PCC run on an all db hosts, what should I use? does it support cumin aliases? Or should it be something like O:mariadb::core_multiinstance,O:mariadb::misc::analytics::backup,O:mariadb::misc::multiinstance,.... [10:55:17] Amir1: https://wikitech.wikimedia.org/wiki/Help:Puppet-compiler#Host_variable_override [10:55:46] I'm not sure on all teh details of its limitations, IIRC it doesn't know the prod aliases [10:56:23] let's see if cumin:db-all would work [10:57:32] worst case, I run it on all hosts, the difference is not that big :D [10:58:10] that would not do what you want, it will pick one host per site.pp node block, so will test very few DBs ;) [10:58:11] Amir1: when using things like O:mariadb::core_multiinstance pcc tries to test on unique hosts so instead of testing on all hosts in the role it will only test on one. when using the cumin:O.. syntax it will test on all hosts that match [10:58:31] also pcc only supports the puppetdb backend i think [10:58:35] volans: that's fine [10:58:36] so aliases wont work [10:58:54] oh, here's the authoritative answer :) [10:59:04] jbond: thanks [10:59:18] we should really add support for things like O:mariadb::core_multiinstance,O:mariadb::misc::analytics::backup but thats not there at the moment [10:59:33] the best thing to do is tor try and find a common class shared by all hosts [10:59:46] there is not :( [11:00:02] * jbond one sec [11:00:49] Amir1: perhaps C:mariadb::config [11:00:52] not even a define? [11:01:18] volans the uniqueng magic only works on roles, classes and profiles [11:01:29] k [11:01:32] * jbond would accept patches ;) [11:02:24] hmm, that would work and if anything is outside and fails, I'll see it in puppetboard [11:02:38] for now, I need to fix the linter first [11:03:11] ack [11:26:53] Amir1: you can also use hostname wildcards, if you wanted all db* servers [11:31:27] Emperor: aah, nice. So far it looks good but I keep it mind for later [11:33:17] or run cumin 'A:db-all' on a cumin host and copy/paste the output :-P [11:33:45] depending how critical is the change and how much you want to run pcc on all dbs [11:44:34] im not sure that pcc supports wildcards, but it dose support regex e.g. `Hosts: re:db.*` [11:54:28] oh, yes, sorry [13:13:08] _joe_ I'm working my way thru the onboarding chats, haven't watched all of them yet... [14:05:27] I've been fighting with this prometheus query for some minutes until I found what I needed- hopefully it saves time to someone else in the future: https://wikitech.wikimedia.org/w/index.php?title=Prometheus&type=revision&diff=1957008&oldid=1951664 [16:30:54] who knows about stat1007? It's gently cron-spamming root [ /usr/local/bin/published-sync -q is suffering from EPERM ] [16:32:49] Emperor: probably btullis or ottomata [16:33:55] yep [17:18:39] jbond: today I'm going to be quite annoying, since SSO is a hot topic rn, is there a way to make a service accessible without CAS-SSO auth to internal network? to be precise, I want to be able to use https://orchestrator.wikimedia.org/api/cluster/alias/s4 without auth in cumin or some other place. [17:19:00] The alternative is to install orch-client in cumin which I want to avoid [17:20:06] OH ho probably me [17:20:10] i changed the stats user uid today [17:20:14] i bet it is related [17:20:18] looking Emperor [17:20:51] o/ is there a doc about the syntax supported behind the "Hosts: " tag (commit message) when triggering a PCC check [17:21:19] dcausse: haha, we talked about it quite a bit today, some scroll up would be useful [17:21:28] oh looking [17:21:31] https://wikitech.wikimedia.org/wiki/Help:Puppet-compiler#Host_variable_override [17:21:38] thanks! :) [17:21:49] but it's incomplete or some tweaks [17:22:07] Amir1: the way we decided to support script usage via cas is not to use cas. i.e. it is app specific. for netbox we use the django-cas module which hooks into the django auth flow and thus allows us to chose between using some api key via authorization headers or redirecting to cas [17:22:12] I think I just need the basics :) [17:23:21] jbond: I see, it's going to be a bit complicated as this is not our codebase so my leeway is limited to config mostly, but I will give it a try and see what I can do [17:24:09] i see orchestrator is using mod_auth_cas so we would have to so i would say the best thing to do is add the config directly to the apach vhost file [17:24:45] e.g allow cuminhosts [17:25:27] yeah, agreed. I think we already have a similar provision in one of our existing vhosts if I'm remembering correctly [17:25:44] * jbond was just checking [17:25:44] I think (not sure though) it has its own webservice based on go, not apache [17:26:05] if it was wrong, It's Manuel's fault [17:26:07] Amir1: there are apps like that and a common solution is to reverse-proxy them using apache2 and mod_auth_cas [17:26:23] Amir1: im gussing its proxied via apache yes this ^^ [17:26:44] oh better [17:27:46] klaxon effectively runs that way, even though it's just fastcgi [17:28:03] ... or maybe it is gunicorn listening on a unix socket? i forget [17:28:06] klaxton also allows unauthenticated access to health check for an example [17:28:13] ~/git/puppet/modules/profile/templates/idp/client/httpd-klaxon.erb [17:28:20] yeah, to a health check and the front page [17:28:25] only part of the URI space is protected [17:28:43] you're making it too easy for me, you know that? [17:28:46] and the app code itself knows very little about CAS [17:28:50] Emperor: thanks for the ping. Fixed: https://phabricator.wikimedia.org/T291384#7778841 [17:28:52] it just knows the names of a few headers to look for a username [17:30:26] Amir1: yes the klaxon template should be a good reference yuo will just need to change all in `Require all granted` to something a bit more restrictive [17:31:13] thankfully apache2 configs are so easy everyone can write them [17:31:40] * Amir1 was sarcastic [17:31:59] * jbond whispers elu.key is the resident apache expert ;) [17:36:48] Amir1: https://gerrit.wikimedia.org/r/c/operations/puppet/+/770981 this is not workign code but it shows how to pass the cumin masteres through to the template file and is mostly there i think [17:37:24] awesome, thanks [17:37:49] np [17:39:10] ottomata: thanks :) [17:58:17] FYI, mr1-ulsfo is going down for replacement, parent/child is set properly but mgmt will most likely alert