[00:53:12] <quiddity>	 Quarry seems to be struggling. It takes ~120 seconds just to load any page (e.g. Main page, e.g. my userpage. Both as a logged-out reader). I'm not sure who can/should be pinged to help with that. https://quarry.wmcloud.org/ 
[07:44:29] <taavi>	 quiddity: i think what you were seeing was a side effect of the tools-static issue that was just being discussed in -cloud
[09:11:57] <topranks>	 folks I will be doing some changes on the cloudsw's in eqiad in the next hour 
[09:12:16] <topranks>	 nothing drastic I just need to rename the bgp 'group' name they are using, which means moving the neighbors 
[09:12:33] <topranks>	 might be some brief alerts for bgp status but traffic won't be interrupted 
[09:12:48] <topranks>	 it's to bring things in line with what we are doing in prod and move it to automated configuration via homer same as we have there 
[14:01:27] <andrewbogott>	 topranks: have time to help me investigate a mystery? Brand new VMs in eqiad1 are coming up with the codfw1dev dns resolver in their resolv.conf and I'm trying to figure out how it got there.
[14:02:17] <topranks>	 i've actually not got a huge amount of time this afternoon, trying to get something over the line before I log off 
[14:02:29] <topranks>	 sorry... not sure what might cause that, has it changed?
[14:02:46] <andrewbogott>	 it changed but I noticed it yesterday, before you did your latest bgp things.
[14:02:52] <andrewbogott>	 If you're busy that's fine, I'll keep digging.
[14:11:09] <andrewbogott>	 https://www.irccloud.com/pastebin/KKhbV4So/
[14:11:16] <andrewbogott>	 that sure is what's happening...
[14:11:20] <topranks>	 it's unrealted to anything network side 
[14:11:29] <topranks>	 a putty/openstack thing I would guess 
[14:11:53] <topranks>	 maybe it's being returned in DHCP by neutron?  VMs use DHCP to get their IPs right?
[14:12:28] <andrewbogott>	 Yeah. So it's a misconfigured dhcp server someplace
[14:34:02] <arturo>	 andrewbogott: check tofu-infra, the DNS that DHCP will use is set on each subnet
[14:34:26] <arturo>	 and there is some copy-paste involved, so potential for typos
[14:34:43] <andrewbogott>	 you are 30 seconds too late! I just found it
[14:34:43] <andrewbogott>	 https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/commit/0cc23ba5119bc762806061f9ebdc3fdee93488ac
[14:35:24] <arturo>	 LGTM
[14:36:35] * andrewbogott tries to remember the proper way to deploy that
[14:49:08] <andrewbogott>	 thanks for appearing arturo! Now I will rebuild a bunch of things :)