[16:26:42] <dduvall>	 mutante: hiera `%{alias('profile::resolving::nameservers')}` interpolation was the only thing i could get passed the wmf-style linter. fortunately that seems a little more coherent to me. what do you think?
[16:28:08] <dduvall>	 i.e. added a `profile::gitlab::runner::buildkitd_nameservers` parameter and set its default value to `%{alias('profile::resolving::nameservers')}` in the hieradata
[16:28:46] <dduvall>	 side note, i'm thinking buildkitd should probably be broken out into its own profile at some point
[16:29:12] <dduvall>	 (but let's get it working first) :)
[16:30:09] <dduvall>	 ok, on to `http_proxy` support in the blubber buildkit frontend...
[16:48:56] <wikibugs>	 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Priority Backlog 📥): Support http_proxy, https_proxy and other proxy `build-arg:` options in blubber buildkit frontend - https://phabricator.wikimedia.org/T317997 (10dduvall)
[16:56:00] <mutante>	 dduvall: I think "ship it" :) just merged
[16:56:40] <mutante>	 yea, I did not fully expected it would also be picky about just the part that the key path does not match the class name. I considered it just warning level
[16:56:53] <mutante>	 so.. nice workaround and let's get in unblocked. there you
[16:56:56] <mutante>	 go
[16:57:20] <mutante>	 still much better than saving the DNS server info a second time in another place
[16:57:39] <dduvall>	 thank you!
[16:58:01] <mutante>	 running puppet on 6 runners
[16:58:04] <dduvall>	 yeah, it's a pretty good solution overall. i had to go digging in the buildkit source and was surprised it wasn't documented
[16:58:18] <mutante>	 :) kudos
[16:58:18] <dduvall>	 i'll submit an upstream PR for documentation :)
[16:58:28] <mutante>	 great
[16:58:49] <dduvall>	 buildkit upstream has been pretty nice to work with so far
[16:59:06] <mutante>	 deployed on all the prod runners. the docker command line has been edited
[16:59:15] <mutante>	 puppet has "refreshed" it
[16:59:16] <dduvall>	 \o/
[16:59:25] <mutante>	 sometimes the question is if "refresh" is enough
[16:59:30] <dduvall>	 right
[16:59:32] <dduvall>	 the service you mean?
[16:59:47] <mutante>	 the systemd unit , yea
[16:59:48] <mutante>	 Exec[systemd daemon-reload for buildkitd.service]: Triggered 'refresh'
[16:59:57] <dduvall>	 oh, daemon-reload
[17:00:00] <dduvall>	 hmm, i'm not sure
[17:00:04] <mutante>	 like does it need systemctl restart or not
[17:00:08] <mutante>	 for this 
[17:00:13] <mutante>	 probably yes because the command line changed
[17:00:23] <mutante>	 +    --volume /etc/buildkitd.toml:/etc/buildkit/buildkitd.toml \
[17:00:23] <mutante>	 +    --network gitlab-runner \
[17:00:57] <mutante>	 [cumin2002:~] $ sudo cumin 'gitlab-runner*' 'systemctl restart buildkitd'
[17:01:00] <mutante>	 6 hosts will be targeted:
[17:01:09] <mutante>	 looks ok. done
[17:01:33] <dduvall>	 ah, we'll need to add `restart => true` to the systemd::service definition in the buildkitd module it seems
[17:01:49] <dduvall>	 for that works for now. thank you!
[17:01:58] <dduvall>	 *but that works
[17:02:01] <mutante>	 ack
[17:02:26] <dduvall>	 i'll re-run the pipeline job and see what happens
[17:02:31] <mutante>	 cool
[17:03:16] <mutante>	 be back in 5 min
[17:03:42] <dduvall>	 huh. failed
[17:03:45] * dduvall takes a look
[17:05:01] <dduvall>	 oh curses. i did not add `--config /etc/buildkit/buildkitd.toml` to the service file :(
[17:05:47] <dduvall>	 i remember this from my testing. that's supposed to be the default but for some reason it doesn't work without the explicit argument
[17:06:11] <dduvall>	 well, i can go ahead and add the `restart => true` to the service definition as well
[17:07:24] <mutante>	 ok, yea, no problem. I can merge that follow-up of cours
[17:15:42] <dduvall>	 https://gerrit.wikimedia.org/r/c/operations/puppet/+/832694 whenever you have a minute
[17:16:03] * dduvall goes back to hacking on blubber frontend
[17:16:51] <mutante>	 on it, brb
[17:20:10] <mutante>	 dduvall: deployed. by accident I did the manual restart again in addition to puppet run .. but yea
[17:20:21] <mutante>	 puppet running now via cumin..after testing on runner 1002
[17:20:27] <mutante>	 done
[17:21:26] <dduvall>	 \o/ it worked!
[17:21:30] <dduvall>	 https://gitlab.wikimedia.org/repos/releng/gitlab-runner-test/-/jobs/24384
[17:23:37] <mutante>	 :))
[17:24:22] <mutante>	 glad it's unblocked before SRE summit. I gotta travel soon :)
[17:24:32] <mutante>	 but just waiting what they rebook me on
[17:29:39] <dduvall>	 oh, this might be me projecting but i hope you have leg room :)
[17:32:30] <mutante>	 dduvall: thank you. there was an entire thread about it for me because the airline has started charging even for regular seats, not just the ones with more legroom. now it's "$40 or $60" but we can pay and reimburse. except now you have to use Coupa 
[17:33:00] <dduvall>	 for a seat!?
[17:33:08] <dduvall>	 like, you don't get a seat by default?
[17:35:20] <mutante>	 correct :o
[17:35:29] <mutante>	 and it's not a low-budget airline. it's Lufthansa
[17:35:54] <mutante>	 ironically the actual low budget flight on the way back.. I got premium eco and priority check-in, it's all reversed :p
[17:37:00] <mutante>	 no "assigned" seat. so if you dont pay it gets randomly assigned at gate.. so likely middle seat if it's full
[17:43:30] <dduvall>	 oh right. yeah, that's terrible and how it works here for most budget airlines
[17:44:02] <dduvall>	 flying is a very unpleasant way to travel, especially for tall folk
[18:33:05] <wikibugs>	 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Priority Backlog 📥): Support http_proxy, https_proxy and other proxy `build-arg:` options in blubber buildkit frontend - https://phabricator.wikimedia.org/T317997 (10dduvall) 05Open→03In progress p:05Triage→03Medium
[18:33:10] <wikibugs>	 10GitLab (CI & Job Runners), 10Patch-For-Review, 10Release-Engineering-Team (Priority Backlog 📥), 10User-brennen: Deploy buildkitd to trusted GitLab runners - https://phabricator.wikimedia.org/T308271 (10dduvall)
[18:37:06] <wikibugs>	 10GitLab (CI & Job Runners), 10Patch-For-Review, 10Release-Engineering-Team (Priority Backlog 📥): Explicitly config buildkitd with internal DNS nameserver - https://phabricator.wikimedia.org/T317904 (10dduvall) 05Open→03Resolved
[18:37:11] <wikibugs>	 10GitLab (CI & Job Runners), 10Patch-For-Review, 10Release-Engineering-Team (Priority Backlog 📥), 10User-brennen: Deploy buildkitd to trusted GitLab runners - https://phabricator.wikimedia.org/T308271 (10dduvall)
[22:34:53] <wikibugs>	 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Priority Backlog 📥): Buildkitd cannot publish image due to `certificate signed by unknown authority` - https://phabricator.wikimedia.org/T318019 (10dduvall)
[22:35:32] <wikibugs>	 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Priority Backlog 📥): Buildkitd cannot publish image due to `certificate signed by unknown authority` - https://phabricator.wikimedia.org/T318019 (10dduvall) p:05Triage→03Medium
[22:42:30] <wikibugs>	 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Priority Backlog 📥): Buildkitd cannot publish image due to `certificate signed by unknown authority` - https://phabricator.wikimedia.org/T318019 (10Dzahn) In class `profile::base::certificates` there is:  `         $puppet_ssl_dir = puppet_ssldir()...
[22:47:14] <wikibugs>	 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Priority Backlog 📥): Buildkitd cannot publish image due to `certificate signed by unknown authority` - https://phabricator.wikimedia.org/T318019 (10Dzahn) **./modules/profile/files/puppet/ca.production.pem**
[23:01:18] <wikibugs>	 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Priority Backlog 📥): Buildkitd cannot publish image due to `certificate signed by unknown authority` - https://phabricator.wikimedia.org/T318019 (10Dzahn) @dduvall   The file above is the right CA for the cert in question.   cert in private repo, verifie...