[11:20:14] !log gitlab-runners enable flavor g3.cores8.ram24.disk20.ephemeral40.4xiops (T293832) [11:20:18] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Gitlab-runners/SAL [11:20:18] T293832: Request increased quota for gitlab-runners Cloud VPS project - https://phabricator.wikimedia.org/T293832 [15:35:04] !log tools add mail.tools.wmcloud.org to the tools mail tls certificate alternative names [15:35:07] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [15:35:48] !log tools remove "^tools-k8s-master-[0-9]+\.tools\.eqiad\.wmflabs$" from authorized_regexes for the main certificate [15:35:50] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [16:38:25] !log paws deploy https://github.com/toolforge/paws/pull/91 [16:38:27] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Paws/SAL [19:29:40] anyone having trouble with Toolforge mail recently [19:30:06] I can send mail from tools fine, including to tool addresses [19:30:48] but when I send mail from my gmail account to a tool address, nothing happens. no bounceback and no mail [19:31:13] tried anticompositebot.maintainers@ and tool.anticompositebot@ and with both domains [19:32:12] it's tools.foo@, not tool.foo@, but I'll have a look [19:32:33] yeah I used tools. [19:38:39] AntiComposite: the most recent incoming email I see from you was at 19:15:21 and exim thinks it was forwarded successfully to the email you have in ldap (same gmail with +wmcs added) [19:39:05] alright, that should be correct [19:39:52] but I never got anything (nothing in Spam, nothing in All Mail, nothing in Trash) [19:41:32] maybe gmail does not accept it when it was sent from the same mailbox? [19:43:38] my past experience was that Gmail dedupes mails that it already has copies of (this is mostly why I stopped using Gmail) [19:51:45] mkay [19:58:01] o_O I can’t SSH to gerrit, permission denied (publickey) [19:58:05] with -v it says: send_pubkey_test: no mutual signature algorithm [19:58:32] do I have such a weird SSH key? [19:58:51] lucaswerkmeister: https://phabricator.wikimedia.org/T240266 [19:59:07] I don’t have access to that [19:59:36] lucaswerkmeister: nevermind, it's something different I think [19:59:51] but apparently I still have an ssh-rsa key, let me try generating an ed25519 one [19:59:51] but the sshd included in gerrit is not openssh [19:59:53] those are nicer anyways [20:03:26] ed25519 key worked 🤷 [20:03:34] https://phabricator.wikimedia.org/T276486 might be the task [20:03:54] looks like it at a glance [20:04:18] oh, so it’s my OpenSSH that doesn’t want ssh-rsa anymore [20:04:25] yeah now I remember reading something like that in the release notes [20:04:38] yea, it's more like you are current and the other side is old [20:04:40] yup, and the gerrit sshd refuses to negotiate (properly) [20:05:52] oh right, because the issue isn’t directly the key, it’s the signature scheme [20:05:55] according to https://lwn.net/Articles/866926/ [20:06:10] so ideally it would still be able to negotiate rsa-sha2-256 or rsa-sha2-512 [20:06:57] well, I’m happy with my new key anyways ^^ [20:10:17] lucaswerkmeister: https://github.com/apache/mina-sshd [20:10:26] that is what yo are talking to, if you are curious [20:10:35] what gerrit uses as internal sshd [20:10:50] so it's not like the regular ones we use on servers [20:11:25] but now I can’t even connect to Gerrit anymore from my desktop o_O [20:11:32] but it works from my server so I guess that’s not Gerrit being down [20:11:49] probably a network issue, see -operations [20:12:26] * lucaswerkmeister tries to look at the -operations log on wm-bot.wmcloud.org, can’t connect to that either [20:12:32] I’ll just believe you that it’s known and be patient [20:12:58] unless they need more info from affected users over there [20:18:50] lucaswerkmeister: where in world are you? [20:19:03] Berlin, Germany [20:21:32] Issues connecting to eqiad known [20:23:20] I'm getting DNS errors on tools too [20:25:04] I can’t even ssh into the wm-bot server or any other wmcloud vms [20:31:27] Seems like outgoing connections are fine though since all the bots are still online [20:31:43] yeah, it's a problem between you and eqiad [20:31:44] It's been worked on by SRE [20:32:00] a North American Fiber-Seeking Backhoe got hungry [20:36:37] there was a fiber cut that probably affected other things [20:51:29] AntiComposite: oh, I know those, we had a problem with them in my parents' yard when I was younger [20:51:34] darn things are quite persistent [20:53:41] we found out there was another issue, unrelated :P [20:56:09] guys, are the issues gone? [20:56:21] AntiComposite: actually a different reason [20:56:29] User side yes [20:56:36] I can access everything I tried firm [20:56:56] thanks, we think it should be doe [20:56:58] done [22:36:17] !log wikistats - creating new instance wikistats-bullseye for testing upgrade to next release [22:36:19] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikistats/SAL [22:36:36] !log wikistats - deleted simplelamp temp test instance [22:36:36] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikistats/SAL [22:40:06] * chicocvenancio looks disapprovingly at `git: 'switch' is not a git command. See 'git --help'` [22:40:36] git config alias.switch checkout /s [22:40:53] LOL [22:41:24] dammit, I tried SSHing into the dev-buster instance but that one doesn’t have `git switch` either :/ [22:43:16] looks like just buster and up [22:43:37] *bullseye [22:44:20] backport ALL the things! [22:44:28] it is actually in buster-backports too [22:47:10] in production: cd ~/puppet/modules/admin/files/home/yourusername ; echo "git config alias.switch checkout" > .bash_profile && git add .bash_profile ;; git commit .. ; git review .. ping me to merge it.. then you have your new global alias [22:49:25] !log tools.video2commons restart to run fix from PR#141 fixes T294150 [22:49:28] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.video2commons/SAL [22:49:28] T294150: Video2commons fails with Object of type bytes is not JSON serializable - https://phabricator.wikimedia.org/T294150 [22:49:34] urbanecm: ^