[08:50:04] inflatador: yeah correct, any Horizon/OpenStack rules are applied on the hypervisor or router layer, and won't affect traffic inside the VM. [08:50:47] those rules would only come in to play if the traffic somehow routed outside the VM and back in, which in some convoluted, nat-y scenarios could happen, but really shouldn't [08:51:10] the VM's own iptables rules are being controlled by puppet/ferm so that's where the rules needed changing [16:56:37] anyone know why the host key for ganeti01.svc.eqiad.wmnet might have changed, ssh is complaning when I try to get console on a vm [16:58:38] see https://phabricator.wikimedia.org/T309724, you can use "gnt-instance console --show-cmd" and then change the options passed to SSH as workaround [17:01:02] got it, thanks, should have read that ticket closer [18:29:18] mutante: would you have time later today to help merge these two changes for Codesearch? (It uses production puppet, but runs in WMCS only) https://gerrit.wikimedia.org/r/c/operations/puppet/+/1016480/8 [18:31:39] Krinkle: yea, I can do that [18:31:52] Krinkle: btw, there are ideas to actually make that prod [18:32:34] * Krinkle wouldn't mind :) [18:33:06] https://phabricator.wikimedia.org/T268199 [18:33:16] My London habit of working evenings, whilst in SF, means no SREs around, so today I'm trying a bit earlier instead :) [19:22:48] good morning Krinkle :) [19:28:45] Morning hashar :) [19:29:45] I am technically in week-end [19:30:36] as for code search, I agree it should ideally be upped to production :D [19:31:03] reverting a codesearch change right now [19:31:17] got delayed a bit by questions re: down db host [19:31:19] on it [19:31:36] hashar: +1 [19:33:56] codesearch is broken right now - waiting for puppet merge to go through [19:38:28] Krinkle: codesearch UI works again [19:38:48] reverted second change and manual systemctl restart codesearch-frontend [19:39:51] mutante: oh, how silly of me. I forgot to remove --network=host [19:39:58] that was from a different patch [19:40:03] I tested without that [19:40:19] * Krinkle re-submits [19:43:50] mutante: let's try again with https://gerrit.wikimedia.org/r/c/operations/puppet/+/1017355 [19:44:05] I've also re-ran the test via docker run manally on the same host and still works :) (HTTP 200 OK) [19:44:09] ok:) [19:49:51] Krinkle: take 2 is applied. puppet run MUCH much faster.. web UI still working [19:50:18] frontend restarted by me [19:50:44] > x-log: getHttp timeout=3 http://172.17.0.1:3002/wmcs/api/v1/repos [19:50:49] and its using the new url [19:50:55] awesome [19:51:07] sounds good [19:51:28] (thats output from https://codesearch.wmcloud.org/wmcs/?action=excludes&debug=1 although if you gtry it now you'll get a different log message since it's apcu cached) [19:54:33] thanks for the well written ticket too [19:54:47] learning from that about the setup [19:55:55] I see cmooney also tracked it down to that ferm rule I made more restrictive back in 2020 [19:59:10] 172.17.0.0/16 is now allowed to access 3002. I just wonder if there is a name for that [19:59:26] docker host :) [19:59:37] you mean an alias like CACHES? [19:59:42] Probably not yet. [19:59:46] I mean.. does it appear somewhere as an alias, yea [20:00:02] I left some details on the comments as well about how that could be done dynamically [20:00:11] thanks [20:00:13] the way upstream docker treats it makes me think there are cases where it can be a different range [20:00:57] I didn't want to make the puppet code complicated by running ifconfig at provision time. that seemed way to fragile to me, and more likelyh to break the tool 1000 times before the 1 time that it might save a breakage if/when that ever changes. [20:00:58] ack, it would seem nicer if it was a variable that is defined in a central location [20:01:19] maybe you know what could make it differ? [20:01:54] like does it vary by who is first to install their apt package? or is it entirely up to the package to declare it statically and would it thus vary between versions of the package (if ever), or by kernel version? [20:02:01] yea, I agree, let's not run ifconfig at provision time [20:02:05] in practice it seems always the same, but I don't know what makes it that way [20:02:07] `git grep 172.17.0.0` on ops/puppet.git hits: modules/gitlab_runner/templates/docker-ferm.erb, modules/profile/files/ci/docker-ferm, modules/profile/files/docker/docker-ferm, and now modules/profile/manifests/codesearch.pp [20:03:22] it's the default range that is possible to redefine but if we don't.. good enough [20:03:54] looks like it's ""default-address-pools":" in docker daemon config [20:04:00] https://serverfault.com/a/936255 describes how to change the default network's IP range via /etc/docker/daemon.json [20:07:35] mutante: awesome new avatar in Phab you have, btw! [20:11:31] heh, thanks [20:12:09] that was after sabbatical when I head a special avatar to show I was "out of office" [20:17:05] maybe https://gerrit.wikimedia.org/r/c/operations/puppet/+/1017367 .. let's see [20:17:56] it's probably not the last time we need it [23:45:08] my new host, mx-out2001 in codfw doesn't receive syn-acks from puppetserver2002, but reaches puppetserver2001 with no issue, the source ip is 208.80.153.13, anyone know if there is some additional step I need to make to wire up the networking?