[08:08:40] If anyone is willing to do a proof reading of the new reimage process draft page, any feedback is welcome! https://wikitech.wikimedia.org/wiki/Server_Lifecycle/Reimage [08:15:06] "Ensure the host to reimage is a physical host" could use a pointer to the Ganeti page what to do for VMs [08:15:58] is in https://wikitech.wikimedia.org/wiki/Server_Lifecycle/Reimage#Virtual_hosts [08:16:22] "Image of a new freshly racked host" -> This also applies for servers which are installed under a new name, so maybe rephrase [08:17:04] true, I was planning to integrate the rename part of Server_Lifecycle afterwards but yeah I can do that [08:20:09] read through it and looks good to me, I think the only thing that could be added [08:20:24] is under What to do if... -> The reimage fails [08:20:49] had a quick look, seems fine to me, thx great stuff! [08:21:09] thanks both! [08:21:12] point people to the /var/log/spicerack/foo directory where to look for further information [08:21:14] moritzm: what you would add there? [08:21:17] ahh ok [08:21:29] there is a small caveat [08:21:45] and maybe a pointer where to diagnose DHCP issues (daemon.log on the install host)? [08:22:52] in case of first puppet run due to a limitation in cumin we're not saving the output but it's viewable in puppetboard (and the cookboks gives the user the link to it) [08:22:56] ack, will add those [11:23:21] 10Puppet, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review, 10User-jbond: Puppet clean up Parent task - https://phabricator.wikimedia.org/T267395 (10jbond) [11:24:02] 10Puppet, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: Puppet Proposal to remove require_package - https://phabricator.wikimedia.org/T266479 (10jbond) 05Open→03Resolved a:03Dzahn I was bold and have now removed require_packages everywhere and dropped it from the repo. The reason i did th... [15:51:37] 10SRE-tools, 10Infrastructure-Foundations, 10SRE, 10serviceops-radar: SVC DNS zonefiles and source of truth - https://phabricator.wikimedia.org/T270071 (10akosiaris) >>! In T270071#7404837, @akosiaris wrote: >>>! In T270071#7404620, @ayounsi wrote: >> About the Ganeti, as I understand it, the issue is that... [15:55:59] 10SRE-tools, 10Infrastructure-Foundations, 10SRE, 10serviceops-radar: SVC DNS zonefiles and source of truth - https://phabricator.wikimedia.org/T270071 (10Volans) @akosiaris my bad, I didn't understood you meant that there ;) Yeah i think that if we could just rename the ganeti DNS record to be ouside of t... [15:57:02] 10SRE-tools, 10Infrastructure-Foundations, 10SRE, 10serviceops-radar: SVC DNS zonefiles and source of truth - https://phabricator.wikimedia.org/T270071 (10akosiaris) >>! In T270071#7409199, @Volans wrote: > @akosiaris my bad, I didn't understood you meant that there ;) Yeah i think that if we could just re... [16:07:36] 10netbox, 10Infrastructure-Foundations: Manage DHCP from Netbox - https://phabricator.wikimedia.org/T269855 (10Volans) p:05Triage→03Medium a:03Volans [16:13:43] topranks: XioNoX: fyi im seeing a routing loop in HE AS13335 (getting a report now) [16:14:25] ok [16:15:13] where are you seeing this? from your home connection? [16:16:26] Sorry, AS133335 is Politeknik Negeri Padang [16:17:14] possibly CF [16:17:25] topranks: actually I can't ssh to prod [16:17:41] and pushing a gerrit change just failed for me [16:17:46] https://phabricator.wikimedia.org/P17435 i did wonder i think he came from mtr and did think hu i thought he was 6963 or something like that [16:17:56] Working for me..... can you do a traceroute volans ? [16:18:18] sure [16:18:24] I can open the wikis though [16:18:28] Through AS6939 is working for me (via tunnel here) [16:18:57] topranks: ignore the HE thing that was a red herring it is as13335 [16:19:09] jbond: what was the target in that trace you posted? [16:19:16] gerrit [16:19:31] me appolagises mtr is being wierd [16:20:10] ok np [16:20:19] so, I can ssh into bast3005, not to bast1003 [16:20:29] bast2002 is ok [16:20:32] hijack then? why should it go to AS13335 ? [16:21:23] AS13335 is CF [16:21:35] sry how'd I get that wrong [16:21:38] apologies [16:22:10] topranks: fyi updated https://phabricator.wikimedia.org/P17435 so its a bit more readable [16:22:19] added an extra '3' when I looked it up, trying to work out what random university in Indonesia had to do with it! [16:22:32] I endup into CF [16:22:54] and get stuck there [16:23:27] XioNoX, topranks: https://phabricator.wikimedia.org/P17437 the last IP is CF [16:23:54] for codfw I go through NTT and reach us normally [16:24:26] volans: XioNoX: has disabled CF so it should fix its self soon [16:24:45] Nothing on their status page I see [16:24:49] yep read it now sorry [16:24:52] for the noise [17:51:23] 10netops, 10Infrastructure-Foundations, 10SRE: Eqiad Expansion - LVS Connectivity Options - https://phabricator.wikimedia.org/T292630 (10cmooney) @BBlack @mark @joanna_borun @ayounsi I've put together a doc outlining some options here. Purpose of the doc is to facilitate collaboration / discussion on the op... [18:19:36] 10Puppet, 10Infrastructure-Foundations, 10GitLab (Infrastructure), 10Patch-For-Review, and 3 others: Puppetise gitlab-ansible playbook - https://phabricator.wikimedia.org/T283076 (10Dzahn) @Jelto As agreed we reimaged gitlab2001 together in a call. We confirmed a few things: - Arnold can run wmf-auto-re... [18:21:02] 10Puppet, 10Infrastructure-Foundations, 10GitLab (Infrastructure), 10Patch-For-Review, and 3 others: Puppetise gitlab-ansible playbook - https://phabricator.wikimedia.org/T283076 (10Dzahn) Also we spoke about the !log command and Arnold used that though we still need to solve privileges so that he can sche... [20:11:55] 10Puppet, 10Infrastructure-Foundations, 10GitLab (Infrastructure), 10Patch-For-Review, and 3 others: Puppetise gitlab-ansible playbook - https://phabricator.wikimedia.org/T283076 (10brennen) Nice work, all! > if you have a user on gitlab-replica.wikimedia.org could you check some features which require lo... [20:29:19] 10Puppet, 10Infrastructure-Foundations, 10GitLab (Infrastructure), 10Patch-For-Review, and 3 others: Puppetise gitlab-ansible playbook - https://phabricator.wikimedia.org/T283076 (10Dzahn) @brennen The VM has been reinstalled and puppet reinstalled gitlab but the actual gitlab data needs to be manually imp...