[01:13:49] <wm-bot>	 !log tools.gitlab-account-approval <bd808> Killed stuck pod running for >24h after ping by legoktm on irc
[01:13:53] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.gitlab-account-approval/SAL
[16:32:19] <wm-bot>	 !log tools.nppbrowser <root> webservice restart
[16:32:22] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.nppbrowser/SAL
[18:11:09] <mutante>	 I am seeing a puppet error on a cloud VPS that seems like it's not related to my role but more a global problem. based on the error message.
[18:11:18] <mutante>	  Lookup of key 'lookup_options' failed: cloudlib::httpyaml failed Tried to load unspecified class: Symbol (file: /etc/puppet/manifests/realm.pp, line: 81,
[18:11:37] <mutante>	 that line 81 is where it looks up nameservers:  $_nameservers = lookup('profile::resolving::nameservers')
[18:12:15] <mutante>	 and the cloudlib::httpyaml is a data_hash in hieradata/cloud.yaml
[18:15:17] <andrewbogott>	 mutante: what's the fqdn of the VM?
[18:15:39] <mutante>	 andrewbogott: contint-bullseye.devtools.eqiad1.wikimedia.cloud
[18:16:51] <mutante>	 there is a local puppetmaster in the project but I am not using it. I have  puppetmaster.cloudinfra.wmflabs.org  in puppet conf on this one
[18:18:51] <andrewbogott>	 Did you already try removing the local puppet config?
[18:19:13] <andrewbogott>	 I suspect the part of the error message with the line number is just a generic 'puppet not working' error
[18:20:00] <mutante>	 well, if I remove my puppet role then there is no puppet error
[18:20:14] <andrewbogott>	 that narrows it down at least
[18:20:16] <mutante>	 but the point of applying it was to find the actual things to fix
[18:20:29] <andrewbogott>	 yeah, would be nice if it gave you an actual error message :/
[18:20:31] <mutante>	 usually that tells me things like "lookup couldnt find this hiera value" or something
[18:20:39] <mutante>	 so I fix one of those
[18:20:43] <mutante>	 and then next thing it's this one
[18:21:14] <mutante>	 well, it's a complex role
[18:21:29] <mutante>	 I am not expecting an easy answer now
[18:21:42] <mutante>	 it just seemed so unsual because it talks about realm.pp
[18:21:49] <mutante>	 and that nameserver lookup
[18:22:01] <mutante>	 and nothing in the actual role.. or at least that is obviously part of it
[18:22:04] <mutante>	 I will look more
[18:22:05] <andrewbogott>	 yeah, it's not great
[18:22:29] <mutante>	 I will make some more experiments
[18:22:59] <andrewbogott>	 If we believe the first part of the message then there's something in role:ci that doesn't actually exist. Is that possible?
[18:24:21] <mutante>	 "lookup_options" is a hiera key set in cloud.yaml 
[18:26:16] <mutante>	 anything is possible with this role, it's the one used on prod CI servers:)
[18:27:08] <mutante>	 my only expectation was to find out what error I get and if it's tied to distro version :)
[18:28:46] <mutante>	 dont worry about it.I will play with it some more.  we might get prod test hardware for ci server dist upgrade
[18:30:18] <mutante>	 though the mystery really is why it cant lookup lookup_options when that is clearly in hierdata/cloud.yaml so global
[18:39:55] <mutante>	 the role includes many profiles. applying one profile at a time to see what happens
[18:54:50] <mutante>	 andrewbogott: I found out what causes it, mostly.and it's kind of funny 
[18:55:01] <andrewbogott>	 tell me!
[18:55:17] <mutante>	 so there is this Hiera data:
[18:55:18] <mutante>	 contint::zuul_merger_hosts: - '::1'  # uses quotes for hiera - 127.0.0.1
[18:55:24] <mutante>	 arrg,let me try this again:
[18:55:34] <mutante>	 contint::zuul_merger_hosts:
[18:55:42] <mutante>	     - '::1'  # uses quotes for hiera
[18:55:47] <mutante>	     - 127.0.0.1
[18:55:55] <mutante>	 ^ this is how it exists in a yaml file
[18:56:10] <mutante>	 now if I copy/paste that into Hiera Config in Horizon
[18:56:17] <mutante>	 the ' ' around ::1 disappear
[18:56:20] <andrewbogott>	 uhoh, does Horizon mess up the quoting?
[18:56:53] <mutante>	 Now there is this in Hiera:
[18:56:54] <mutante>	 - ::1
[18:57:01] <mutante>	 and that creates the fun error message
[18:57:21] <mutante>	 if I just remove that v6 line and keep only the - 127.0.0.1 then I'm good 
[18:58:01] <andrewbogott>	 I'm glad you have a fix but now I'm confused about hiera and yaml :(
[18:58:04] <mutante>	 the "# uses quotes foer hiera" comment is appropriate :)
[18:58:43] <mutante>	 :: must be special 
[18:59:01] <andrewbogott>	 yeah
[18:59:30] <mutante>	 the actual bug is you can't put the '' in the web form
[18:59:33] <mutante>	 I guess
[19:00:05] <mutante>	 can also be worked around by just putting it in yaml in a repo
[21:00:37] <wm-bot>	 !log tools.stashbot <bd808> Updated to 3343de0 (T346631)
[21:02:13] <wm-bot>	 !log tools.stashbot <bd808> Updated to 3343de0 (T346631)
[21:02:17] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stashbot/SAL
[21:30:50] <mutante>	 !log devtools contint-bullseye - sudo /usr/sbin/a2dismod mpm_event ; sudo /usr/sbin/a2endmod php74 - the usual issue we have had for years
[21:30:53] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Devtools/SAL