[07:35:19] <elukey>	 hi folks
[07:35:43] <elukey>	 toolhub.wikimedia.org is down, there are some problems with its pods in wikikube eqiad
[07:36:14] <elukey>	 I don't see a clear issue, but afaics the main container fails the readiness probe (it is monday morning so it may be me being slow)
[07:36:41] <elukey>	 the service is active/passive, not sure if there is a procedure to failover but in case the codfw pods are up
[07:54:05] <dcaro>	 elukey: I have little experience with it, but I can try to give it a look, how can I get to the logs?
[07:54:53] <elukey>	 dcaro: o/ I filed a change a min ago, I think it is a dbproxy
[07:55:00] <elukey>	 I am deploying now, let's see if it solves
[07:55:16] <dcaro>	 👍 thanks!
[07:55:33] <elukey>	 yep all up! https://toolhub.wikimedia.org/
[07:55:38] <elukey>	 credits to taavi <3
[07:58:08] <dcaro>	 \o/
[08:01:11] <elukey>	 so I think that there are some follow ups to do
[08:01:40] <elukey>	 like, better logging, the crawler gave me some hints but the other pods didn't really emit any meaningful logs, they only failed the probes
[08:01:52] <elukey>	 what is the best tag/poc to use for toolhub?
[08:02:15] <dcaro>	 I think just 'toolhub'
[09:24:40] <dcaro>	 !log tools deploy envvars-api 0.0.20
[09:24:42] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[09:58:45] <elukey>	 dcaro: sorry for the delay, opened https://phabricator.wikimedia.org/T340950, thanks!
[12:01:13] <dcaro>	 !log tools deploy builds-api 0.0.74
[12:01:15] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[12:12:02] <taavi>	 !log tools deploy jobs-api 0.1.5
[12:12:04] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[12:21:33] <elukey>	 taavi: thanks a lot!
[13:17:06] <dcaro>	 !log tools deploy envvars-admission 0.0.8
[13:17:08] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[13:17:15] <dcaro>	 (no real changes, just ci flows change)
[13:29:50] <dcaro>	 !log tools deploy builds-api 0.0.75 (no real changes, just ci rebuild)
[13:29:53] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[13:48:59] <dcaro>	 !log tools deploy envvars-api 0.0.21 (no real changes, ci rebuild)
[13:49:01] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[14:29:37] <Cyberpower678>	 Can I get an update on being able to send emails again on Toolforge.  I've had a ticket reporting that the SMTP server on Toolforge is rate limiting my tool to send 0 emails.
[14:30:42] <Cyberpower678>	 After following the documentation and implementing the new SMTP server on Toolforge, I have yet to be able to successfully send a single email, and am instead getting a message claiming I am being rate limited.
[14:31:29] <taavi>	 looking now into that. sorry for the delay, forgot to do that that after the hackathon
[14:35:48] <taavi>	 Cyberpower678: which sender address are you using? I have a feeling that something is setting the sender address as an empty string
[14:36:14] <taavi>	 and I'd still like to see the FULL smtp logs, including what commands your client is sending to the server
[14:36:15] <Cyberpower678>	 No worries.
[14:37:28] <Cyberpower678>	 taavi: mind if I PM?
[14:38:27] <taavi>	 go ahead
[17:15:05] <yannfo>	 Hi, I can't log in to login.toolforge.org using https://wikitech.wikimedia.org/wiki/Help:Toolforge/Quickstart
[17:15:40] <wm-bb>	 <jeremy_b> define "log in to"
[17:16:06] <yannfo>	 short answer: yann@login.toolforge.org: Permission denied (publickey,hostbased).
[17:16:30] <yannfo>	 ssh yann@login.toolforge.org
[17:16:53] <wm-bb>	 <jeremy_b> add a `-v` to your `ssh` and pastebin
[17:17:15] <yannfo>	  ssh -v yann@login.toolforge.org
[17:17:36] <yannfo>	 OpenSSH_for_Windows_8.1p1, LibreSSL 3.0.2
[17:17:37] <yannfo>	 debug1: Connecting to login.toolforge.org [185.15.56.66] port 22.
[17:17:37] <yannfo>	 debug1: Connection established.
[17:17:38] <yannfo>	 debug1: identity file C:\\Users\\yannf/.ssh/id_rsa type 0
[17:17:38] <yannfo>	 debug1: identity file C:\\Users\\yannf/.ssh/id_rsa-cert type -1
[17:17:39] <yannfo>	 debug1: identity file C:\\Users\\yannf/.ssh/id_dsa type -1
[17:17:39] <yannfo>	 debug1: identity file C:\\Users\\yannf/.ssh/id_dsa-cert type -1
[17:17:40] <yannfo>	 debug1: identity file C:\\Users\\yannf/.ssh/id_ecdsa type -1
[17:17:40] <yannfo>	 debug1: identity file C:\\Users\\yannf/.ssh/id_ecdsa-cert type -1
[17:17:41] <yannfo>	 debug1: identity file C:\\Users\\yannf/.ssh/id_ed25519 type 3
[17:17:41] <yannfo>	 debug1: identity file C:\\Users\\yannf/.ssh/id_ed25519-cert type -1
[17:17:42] <yannfo>	 debug1: identity file C:\\Users\\yannf/.ssh/id_xmss type -1
[17:17:42] <yannfo>	 debug1: identity file C:\\Users\\yannf/.ssh/id_xmss-cert type -1
[17:17:43] <yannfo>	 debug1: Local version string SSH-2.0-OpenSSH_for_Windows_8.1
[17:17:43] <yannfo>	 debug1: Remote protocol version 2.0, remote software version OpenSSH_7.9p1 Debian-10+deb10u2
[17:17:44] <yannfo>	 debug1: match: OpenSSH_7.9p1 Debian-10+deb10u2 pat OpenSSH* compat 0x04000000
[17:17:44] <yannfo>	 debug1: Authenticating to login.toolforge.org:22 as 'yann'
[17:17:45] <yannfo>	 debug1: SSH2_MSG_KEXINIT sent
[17:17:56] <yannfo>	 debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521>
[17:17:56] <yannfo>	 debug1: SSH2_MSG_SERVICE_ACCEPT received
[17:17:57] <yannfo>	 debug1: Authentications that can continue: publickey,hostbased
[17:17:57] <yannfo>	 debug1: Next authentication method: publickey
[17:17:58] <yannfo>	 debug1: Offering public key: C:\\Users\\yannf/.ssh/id_rsa RSA SHA256:fR7u7fHiRpFdTUt3h3hT23HJDdhMBo6zZrmgCIR+Gic
[17:17:58] <yannfo>	 debug1: Authentications that can continue: publickey,hostbased
[17:17:59] <yannfo>	 debug1: Trying private key: C:\\Users\\yannf/.ssh/id_dsa
[17:17:59] <yannfo>	 debug1: Trying private key: C:\\Users\\yannf/.ssh/id_ecdsa
[17:18:00] <yannfo>	 debug1: Offering public key: C:\\Users\\yannf/.ssh/id_ed25519 ED25519 SHA256:ftFWRBsN6RARXfzFp4Vzr6XcyvfjW93b/i2lI6N1R3o
[17:18:00] <yannfo>	 debug1: Authentications that can continue: publickey,hostbased
[17:18:01] <yannfo>	 debug1: Trying private key: C:\\Users\\yannf/.ssh/id_xmss
[17:18:01] <wm-bb>	 <jeremy_b> PASTEBIN!! not paste it here.
[17:18:01] <yannfo>	 debug1: No more authentication methods to try.
[17:18:02] <yannfo>	 yann@login.toolforge.org: Permission denied (publickey,hostbased).
[17:18:15] <wm-bb>	 <jeremy_b> PASTEBIN!! not paste it here.
[17:21:03] <yannfo>	 https://pastebin.com/T0uPxe6g
[17:21:27] <yannfo>	 the error is pubkey_prepare: ssh_get_authentication_socket: No such file or directory
[17:24:54] <yannfo>	 I suppose it should ask for my password, at least the first time, before I copy my public key on the server
[17:25:32] <wm-bb>	 <jeremy_b> no
[17:26:43] <wm-bb>	 <jeremy_b> if you actually followed the instructions you already linked to then it would already have your key.
[17:28:50] <yannfo>	 Yes, I did that. My key can be seen on https://toolsadmin.wikimedia.org/profile/settings/ssh-keys/
[17:32:35] <taavi>	 yannfo: the key you have configured (RSA SHA256:hMiUN+3M87A8sEb50ldQp+FG2hpurTtmcFtYJz3McVA) does not seem to match what your ssh client is actually trying to use (RSA SHA256:fR7u7fHiRpFdTUt3h3hT23HJDdhMBo6zZrmgCIR+Gic or ED25519 SHA256:ftFWRBsN6RARXfzFp4Vzr6XcyvfjW93b/i2lI6N1R3o). double check you have the correct key configured?
[17:33:31] <yannfo>	 OK, I got it
[17:37:35] <yannfo>	 Now it seems the local files do not have the right permissions.
[17:39:02] <yannfo>	 However drwx------+ 1 Yann            Aucun               0 Jul  3 18:49  .ssh
[17:39:29] <yannfo>	 -rw-------+ 1 Yann Aucun  411 Jul  3 17:40 id_ed25519
[17:39:48] <wm-bb>	 <jeremy_b> what's wrong with those permissions?
[17:39:49] <yannfo>	 -rw-------+ 1 Yann Aucun 2610 Jul  3 17:16 id_rsa
[17:40:07] <wm-bb>	 <jeremy_b> also I thought we agreed not to paste?
[17:40:33] <yannfo>	 Permissions for 'C:\\Users\\yannf/.ssh/id_rsa' are too open.
[17:40:33] <yannfo>	 It is required that your private key files are NOT accessible by others.
[17:40:34] <yannfo>	 This private key will be ignored.
[17:40:34] <yannfo>	 Load key "C:\\Users\\yannf/.ssh/id_rsa": bad permissions
[17:42:27] <wm-bb>	 <jeremy_b> error message is straightforward but I don't see the problem in I assume that's output from `ls`. anyway now you have an error message to run with.
[17:45:12] <yannfo>	 ok thanks
[18:46:14] <wm-bot>	 !log tools.stewardbots <anticomposite> start webservice since it appeared to not be running for some reason
[18:46:17] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stewardbots/SAL