[07:25:06] https://icinga.wikimedia.org/alerts doesn't look too bad after the holidays! [07:25:14] and happy new year everyone [07:31:08] * tn pretends to not see the criticals :P [10:19:08] happy new year :) [10:46:49] happy new year dcaro [11:02:09] I have to reboot deploy1002; I see a couple of people have a tmux open running some non-critical things, and those people are offline now [11:03:04] zpapierski, inflatador sorry about that ^^ [11:20:05] (rebooting now) [11:36:23] elukey: hi! do you have any idea how to test https://gerrit.wikimedia.org/r/c/operations/puppet/+/751100/? [11:37:22] moritzm: if you are in the mood of reviewing puppet cleanup patches, feel free to look into any of the ones I'm sending :) (only added you as a backup if I did not find an 'owner', but reviews are always welcome) [11:38:33] (thanks for the ones you already did ;) ) [11:38:46] sure, feel free to send more my way :-) [11:39:15] taavi: o/ I am officially back on the 10th, we can work on it next week if you are ok [11:40:07] sure, works for me, enjoy your holiday [14:09:05] sure, no worries joe [14:53:30] moritzm: jhathaway: hey! ftp.us.debian.org still includes sodium and not mirror1001, which is now causing user reports of outdated mirror data on #wikimedia [14:53:56] I'm guessing someone needs to ask Debian to update that dns record on their side? [14:58:54] silly question maybe but should it be a VIP behind LVS? [15:01:24] XioNoX: I think it's enough to add a cname for now [15:02:04] taavi: that's the long-term solution; short term I think adding a cname from sodium to mirror1001 is a good temporary patch [15:02:19] +1 [15:02:21] joe: you can't do that when they hardcode ip addresses in zone files https://salsa.debian.org/dsa-team/mirror/domains/-/blob/master/debian.org#L1168 [15:03:02] le sigh [15:03:04] ah [15:03:17] note that the IP is still assigned to sodium in https://netbox.wikimedia.org/ipam/ip-addresses/6324/ [15:03:42] so we could add it to mirror1001 to fix the issue temporarily [15:03:51] (and decom sodium) [15:03:54] taavi: thanks [15:04:26] yeah but I cringe every time I see multiple IPs in the same subnet on the same network interface [15:04:37] I've seem my fair share of issues with such things [15:06:38] yeah, it would only be a workaround until DNS is updated upstream [15:06:46] depending on how urgent it is [15:07:48] Ideal pi-in-the-sky would be that all services we expose externally use VIPs so they stay static even if we move the backends [15:08:15] but I agree it's significant work for a single node service [15:10:35] XioNoX: for a single service node, wouldn't it make sense to have the VIPs on the routers? [15:10:39] I am going to kick off a manual sync on sodium, to at least bring it up to date, while we determine a better short term fix [15:10:52] oh if sodium is still up, then sure [15:10:58] I thought it was decommissioned [15:11:27] not yet, I was keeping it around for a bit, just in case [15:11:56] only the debian mirror is triggered via ssh, so the other mirros should still be up to date [15:16:46] I'm going to try to track down how to get the DNS record updated, if anyone has any inside knowledge please pass it my way! [15:27:11] joe: not sure I understand what you mean with VIPs on the routers? (as in the server advertise its VIP to the router via BGP?) [15:27:31] XioNoX: I was thinking NAT :P [15:27:38] but yes bgp via bird would work too [15:29:01] I'm also wondering if it could be a private host, and keep in in sync through the proxies [15:34:42] jhathaway: this looks like the form needed https://www.debian.org/mirror/submit [15:35:32] jbond: thanks, I'll try that route [15:40:13] I wonder why they didn't use a cname to point to mirrors.wikimedia.org [15:43:15] you can't have multiple CNAMEs in there [15:43:40] (for the same record) [15:45:33] taavi: right, right [15:49:41] I sent an email to the mirrors team, as was suggested in IRC [17:57:40] sodium, is now up to date [21:09:15] paravoid: someone noticed the sodium IP's earlier today [21:09:31] Debian have been asked to update, see scrollback [21:10:10] ah didn't see that, thanks [21:10:17] I believe mirrors1001 updates are working as they don't check host keys [21:10:24] Strictly [21:10:41] It's on one of the gerrit patches