[07:32:24] 10Puppet, 10Infrastructure-Foundations, 10Project-Admins, 10PM: Clarify Puppet tag - https://phabricator.wikimedia.org/T295221 (10Aklapper) @joanna_borun: Could you please answer? Thanks in advance! :) [08:35:36] 10netops, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: Rebuild ping* hosts with 10G disks - https://phabricator.wikimedia.org/T295767 (10MoritzMuehlenhoff) >>! In T295767#7529602, @ayounsi wrote: > All 3 VMs got rebuilt with larger disks, but with the default Debian Buster. > > @MoritzMuehlenh... [09:09:24] 10Puppet, 10DBA, 10Data-Engineering, 10Infrastructure-Foundations: Split mariadb::dbstore_multiinstance into 2 separate roles (backup sources and analytics) - https://phabricator.wikimedia.org/T296285 (10Marostegui) a:03jcrespo Assigning to Jaime to reflect current status as I believe he's working on it... [09:32:57] 10puppet-compiler, 10Infrastructure-Foundations, 10Cloud-Services-Origin-Team, 10Cloud-Services-Worktype-Project, and 2 others: PCC: add a fast fail option to bail out when the first error is encountered - https://phabricator.wikimedia.org/T295028 (10dcaro) [11:13:15] 10netops, 10Infrastructure-Foundations: Enable NTP for drmrs network devices - https://phabricator.wikimedia.org/T296623 (10cmooney) p:05Triage→03Low [11:21:37] 10netops, 10Infrastructure-Foundations: Enable NTP for drmrs network devices - https://phabricator.wikimedia.org/T296623 (10cmooney) Ok so on the switches I can see requests hitting the dns servers and they are responding: ` cmooney@dns1001:~$ sudo tcpdump -i ens2f0np0 -l -p -nn host 10.136.128.4 tcpdump: ver... [11:30:29] 10Puppet, 10DBA, 10Data-Engineering, 10Infrastructure-Foundations: Split mariadb::dbstore_multiinstance into 2 separate roles (backup sources and analytics) - https://phabricator.wikimedia.org/T296285 (10Kormat) Split the alias, have a new `db-backup-source` alias. [12:00:18] 10netops, 10Infrastructure-Foundations, 10SRE: Enable NTP for drmrs network devices - https://phabricator.wikimedia.org/T296623 (10cmooney) Ok yes it seems to be the loopback filter alright, testing the change on asw1-b13-drmrs adding a new term as advised in the KB article fixed it: ` cmooney@asw1-b13-drmrs... [12:48:38] volans, XioNoX: one of you may be able to help me. [12:48:46] yep? [12:48:52] I'm trying again to get homer working to our routers from my laptop. [12:49:26] (I gave up during my first weeks, but it's useful to sanity check potential config changes before submitting CRs). [12:49:30] I'm hitting this though: [12:49:32] DEBUG:homer.transports.junos:Connecting to device cr1-eqiad.wikimedia.org (user ssh_config None) [12:49:56] If I use the normal ssh client it works ok. Undoubtedly something to do with my ssh config file. [12:50:18] I think the issue is the "None" for user [12:50:37] I subsequently get this and it throws an exception: [12:50:37] DEBUG:ncclient.transport.ssh:[host cr1-eqiad.wikimedia.org session 0x7ff279436730] private key file is encrypted [12:51:13] I see in some other github issues people getting same if they submit wrong user using paramiko [12:52:47] Sorry I think I might have fix... for my lap setup I define "username" in /etc/homer/config.yaml under "transports" [12:52:50] let me give that a go [12:52:56] #rubberduckdebugging [12:53:01] :) [12:56:02] Ok well I'm still getting the same, but the changed error gives me more insight [12:56:06] DEBUG:homer.transports.junos:Connecting to device cr1-eqiad.wikimedia.org (user cmooney ssh_config None) [12:57:07] let me set the ssh_config var think I'll be set. [12:57:10] sorry for the spam :) [12:57:35] it's fine, not sure what to recommend though [13:02:23] hmm ok, I've made some progress [13:03:24] But I think fundamentally it isn't working cos it doesn't have passphrase for my ssh key [13:03:46] still get this, and I'm pretty sure it's looking in the right place now [13:03:46] DEBUG:ncclient.transport.ssh:[host cr1-eqiad.wikimedia.org session 0x7f4e39007af0] private key file is encrypted [13:04:11] topranks: do you use an ssh agent? [13:04:26] no [13:04:44] I'm currently pointhing homer at this config file: [13:04:47] https://www.irccloud.com/pastebin/w8TCVdbd/ [13:06:17] https://github.com/paramiko/paramiko/blob/c648199836db920cf9828d66880100f9d67dd359/paramiko/pkey.py#L246 [13:08:05] nevermind, that's paramiko, not ncclient [13:08:46] ncclient uses paramiko I think though? [13:09:51] what's the error you get? [13:10:17] DEBUG:ncclient.transport.ssh:[host cr1-eqiad.wikimedia.org session 0x7f4e39007af0] private key file is encrypted [13:10:26] then [13:10:29] ncclient.transport.errors.AuthenticationError: PasswordRequiredException('private key file is encrypted') [13:10:39] and [13:10:41] jnpr.junos.exception.ConnectAuthError: ConnectAuthError(cr1-eqiad.wikimedia.org) [13:16:04] OK I got it working. [13:16:22] ah? [13:16:26] Removing the "IdentityFile" (line 5) from the config.homer file made it work ?? [13:16:50] So I guess if they unlocked key is in memory it forwards it, but specifying the filename made it try to open that instead? [13:17:04] or something like that (excuse my terrible terminology!) [13:17:12] that could make sens, yeah [13:19:41] now you say it it makes sens [13:19:44] https://github.com/ncclient/ncclient/blob/master/ncclient/transport/ssh.py#L413 [13:20:13] it tries the keys it finds in the config files directly, then return an exception if nothing is found [13:20:37] and only after that it tries the local agent [13:22:17] cool [13:22:19] mystery solved! [13:24:12] we usually use ssh agents [13:29:25] I've been super cautious in my config tbh [13:29:41] Given all the direction to avoid forwarding prod agent to anything in cloud realm I disabled the forwarding everywhere really. [13:52:28] 10CAS-SSO, 10Infrastructure-Foundations, 10SRE: Deprecation of U2F API in Chrome / Enable web auth in CAS - https://phabricator.wikimedia.org/T296629 (10MoritzMuehlenhoff) [13:57:10] 10netops, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: Enable NTP for drmrs network devices - https://phabricator.wikimedia.org/T296623 (10cmooney) ^^ apologies ignore above used incorrect task ref. [13:59:57] 10netops, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review, 10Sustainability (Incident Followup): Use next-hop-self for iBGP sessions - https://phabricator.wikimedia.org/T295672 (10cmooney) ^^ ignore above - pasted wrong task ID. and sorry for spam. [14:27:29] 10CAS-SSO, 10Infrastructure-Foundations, 10SRE, 10User-jbond: Deprecation of U2F API in Chrome / Enable web auth in CAS - https://phabricator.wikimedia.org/T296629 (10jbond) [14:47:35] 10CAS-SSO, 10Infrastructure-Foundations: [chrome] MFA in idp.wikimedia.org will not be usable past Feb 2022 - https://phabricator.wikimedia.org/T296591 (10Volans) [14:47:37] 10CAS-SSO, 10Infrastructure-Foundations, 10SRE, 10User-jbond: Deprecation of U2F API in Chrome / Enable web auth in CAS - https://phabricator.wikimedia.org/T296629 (10Volans) [15:01:00] topranks: re:homer changes. tl;dr is yes i would like to remain on them but i should be considered more a CC then a blocking reviewr. [15:03:39] ok no probs at all [15:03:41] I just thought I'd ask as I seen where that is configured and it occurred to me it was possibly from when Arzhel was only net. eng on the team. [15:03:43] more expanded before you started i was a bit more active in actully revieiing changes for XioNoX however i stepped back from that when you started. i am of course happy to started activly reviewing again however i tend to think its more adding noise theses days. however it is stil usefull for me to keep an eye on the changes as one intention is for me to look at helping with the dev of [15:03:49] the actuall homer software, and also to keep a ... [15:03:52] ... genral eyey on how the network is changing [15:04:27] as to where its configuered its this magic wiki page https://www.mediawiki.org/wiki/Git/Reviewers [15:04:55] yeah that came up on irc during the week and I was like "ah so that's why John gets cc'd on these" [15:05:03] ack [15:05:14] More than happy to have your input and advice into network stuff at any stage though [15:06:37] ack thanks i genrally try to look and would of course ask if anything seemed wrong but like i said probably best to just think of me as a CC and ping if oyu need an extra set of eyes (if someone is on vacation etc) [15:07:54] oh yeah that's fine - no expectation you'll be able to look at everything. [15:18:46] cdanis: i pinged last week when you where out but could you look at https://gerrit.wikimedia.org/r/c/operations/puppet/+/740818 to make sure the limits seem sound to you [15:19:09] and there is also https://gerrit.wikimedia.org/r/c/operations/puppet/+/740828/10 which should let is control the same block via hiera [15:20:18] 10netops, 10Infrastructure-Foundations, 10SRE, 10Traffic-Icebox, 10Patch-For-Review: Collect netflow data for internal traffic - https://phabricator.wikimedia.org/T263277 (10cmooney) Seems like a sane proposal. The use of sflow and a different pipeline will keep a clean separation between it and data fr... [16:28:22] jbond: yeah sorry I'm definitely getting to that early this week :) [16:34:59] cdanis: great thanks <3 [17:27:38] Should we merge Network, tooling and infra security in one I/F section? [17:27:48] good question [17:27:50] not sure [17:28:09] as long as we keep the right people to talk for them fine for me :D [17:29:53] I've asked that same question a couple of times [17:30:10] I think everyone was kind of neutral about it so I didn't press it :) [17:30:47] * jbond remains fairly neutrual about it ;) [17:31:21] paravoid: I guess is a more a decision on your side if you prefer team or topic updates :D [17:31:24] currently it's a mix [17:31:41] everything seems to be merging toward teams [17:31:41] I'm not that prescriptive [17:32:40] or better, it's team based with the exception of I/F :D