[08:50:03] Note: stashbot keeps failing to log due to an issue being looked at to wiki. Might be worth using Sal.toolforge.org until it behaves. [10:06:04] I watched the recording of the Debian installer BoF session at DebConf last night (no slides yet, recording at https://ftp.acc.umu.se/pub/debian-meetings/2022/DebConf22/) [10:06:37] but the TLDR key takeaway is that hacking d-i (and debugging!) will in the mid term become simpler [10:07:19] d-i will no longer be constrained to uclibc, busybox (and it's limited shell implementation) and udebs [10:07:43] but become more like a regular Debian system based on glibc and most probably using standard Debian packages [10:08:47] and the most interesting change, the plan is to piece-wise reimplement d-i in Python (or at least allowing to implement functionality in Python where it makes sense) [10:09:29] this notably also includes Partman, which might either switch to Growlight or a different reimplementation [10:44:33] that's very good news :) [10:56:31] <_joe_> yeah it all makes sense I guess :) [11:01:27] I am sure kormat will rejoice when she hears that :) [13:16:35] do we have the list of LVS IPs that are doing monitoring requests available anywhere where they can easily be used for ferm rules? [13:17:02] yes [13:18:54] * vgutierrez is trying to remember exactly where [13:20:12] hmmm but wait.. [13:20:25] taavi: do you want to restrict L3 traffic just to LVS? [13:20:59] that's not gonna work as expected [13:21:15] user traffic hits the backend server with the original IP [13:23:05] vgutierrez: I want to close down a port that's currently open to everywhere so that only traffic that's coming from the caches is allowed, but I also don't want to break the LVS healthchecks [13:23:20] the first part I can easily do with $CACHES in srange, but the second part seems a bit trickier [13:23:22] errr caches or LVS? [13:23:26] <_joe_> taavi: what is served from that port? [13:23:42] wikitech and other services [13:23:44] <_joe_> vgutierrez: I think we have an XY problem here [13:23:49] so let me backtrack a little [13:24:02] <_joe_> ok [13:25:50] so the default value of profile::tlsproxy::envoy::ferm_srange is undef, which exposes the envoy proxy to the entire world [13:26:39] wikitech servers (labweb1001/2 and now cloudweb1003/4) have public IPs, meaning that the envoy is accessible from the internet [13:27:06] I want to close down that port so that only traffic that's coming via the caches is allowed [13:27:29] but I imagine if I just set ferm_srange to $CACHES, LVS healthchecks would start failing since LVS servers aren't in $CACHES [13:27:32] does that make any sense? [13:27:36] <_joe_> that's all correct [13:27:50] <_joe_> the XY is indeed that cloudweb has a public IP [13:27:54] <_joe_> and still using lvs [13:28:11] <_joe_> so yeah, I think you have the additional problem [13:28:41] <_joe_> that the loadbalancers appear to call the backends from different IPs depending on the VLAN I think [13:29:28] the 'cloudwebs have public ips' problem is a known and a harder problem to solve, since horizon (also hosted there) needs to be able to talk to services hosted in the cloud realm [13:30:55] <_joe_> so for instance [13:31:51] quick tcpdump check validates what _joe_ is saying... lvs1020 hits cloudweb1003 using vl1002-ens1f0np0.lvs1020.eqiad.wmnet [13:32:14] <_joe_> vgutierrez: you know you also have access logs right [13:32:16] <_joe_> :P [13:32:31] _joe_: I'm not used to have access logs :( [13:32:55] <_joe_> cloudweb1003:~$ fgrep Twisted /var/log/apache2/other_vhosts_access.log | awk '{print $3}' | uniq -c [13:32:57] <_joe_> 12826 208.80.154.150 [13:32:59] <_joe_> uhm [13:33:01] we can't afford them on cp* boxes [13:33:25] that's cloud1003 IP [13:33:26] <_joe_> access logs don't help lol [13:33:27] <_joe_> yes [13:33:28] *cloudweb ;P [13:33:28] fwiw 1003/1004 aren't in service yet [13:34:00] <_joe_> vgutierrez: sigh right that's the wrong IP to look at [13:34:16] <_joe_> because that's probably coming via envoy [13:34:46] <_joe_> and for some reason it's not populating X-Client-IP [13:35:11] so.. those IPs are available on lvs::interfaces::vlan_data, but that data is only available for LVS boxes [13:36:48] I'm can also live with using something like $PRODUCTION_NETWORKS if the more specific rules need too much effort [13:42:26] taavi: yep.. that would solve the issue [13:44:38] <_joe_> tbh I think that's the best option atm [13:48:34] i'll do that the, thanks [18:44:52] I have heard of bats before, but never used it. I used it today and it seems pretty nice for adhoc testing, https://github.com/bats-core/bats-core [19:50:03] thanks for the debconf links moritz-m! will have to watch these