[09:25:47] <wm-bb>	 <yochayco> Hi, I'm having issues connecting with ssh to the server (bastion and glamswikidashboard servers)
[10:26:51] <andre>	 please elaborate what "issues" means
[10:27:25] <andre>	 and provide the exact command being used so we don't have to guess
[10:27:27] <andre>	 "ssh -vvv" provides more verbose output, in case it helps
[11:14:55] <wm-bb>	 <yochayco> Sorry, I wasn't sure if this will be a bot or a human.
[11:14:56] <wm-bb>	 <yochayco> I mean it does not connect when I try to use ssh to connect to bastion server. It seems that the authentication is successful but the authorization is not.
[11:14:57] <wm-bb>	 <yochayco> - it's not my first time using ssh
[11:14:58] <wm-bb>	 <yochayco> - it's my first time trying to connect to the wiki network
[11:15:00] <wm-bb>	 <yochayco> - I've already consulted with someone who succesfully connects to the server and he also thinks that my configurations are fine.
[11:15:01] <wm-bb>	 <yochayco> Here are some highlights of the verbose output I got:
[11:15:03] <wm-bb>	 <yochayco> ```
[11:15:04] <wm-bb>	 <yochayco> ssh bastion.wmcloud.org -vvv 
[11:15:06] <wm-bb>	 <yochayco> OpenSSH_9.6p1 Ubuntu-3ubuntu13.5, OpenSSL 3.0.13 30 Jan 2024 
[11:15:07] <wm-bb>	 <yochayco> debug1: Reading configuration data /home/yochayc/.ssh/config 
[11:15:09] <wm-bb>	 <yochayco> debug1: /home/yochayc/.ssh/config line 1: Applying options for bastion.wmcloud.org 
[11:15:10] <wm-bb>	 <yochayco> ...
[11:15:12] <wm-bb>	 <yochayco> debug1: /etc/ssh/ssh_config line 21: Applying options for * 
[11:15:13] <wm-bb>	 <yochayco> debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/home/yochayc/.ssh/known_hosts' 
[11:15:15] <wm-bb>	 <yochayco> debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/home/yochayc/.ssh/known_hosts2' 
[11:15:16] <wm-bb>	 <yochayco> debug2: resolving "bastion.wmcloud.org" port 22 
[11:15:18] <wm-bb>	 <yochayco> debug3: resolve_host: lookup bastion.wmcloud.org:22 
[11:15:19] <wm-bb>	 <yochayco> debug3: channel_clear_timeouts: clearing 
[11:15:21] <wm-bb>	 <yochayco> debug3: ssh_connect_direct: entering 
[11:15:22] <wm-bb>	 <yochayco> debug1: Connecting to bastion.wmcloud.org [185.15.56.87] port 22. 
[11:15:24] <wm-bb>	 <yochayco> debug3: set_sock_tos: set socket 3 IP_TOS 0x10 
[11:15:26] <wm-bb>	 <yochayco> debug1: Connection established. 
[11:15:27] <wm-bb>	 <yochayco> debug1: identity file /home/yochayc/.ssh/wmil type 3 
[11:15:28] <wm-bb>	 <yochayco> debug1: identity file /home/yochayc/.ssh/wmil-cert type -1 
[11:15:30] <wm-bb>	 <yochayco> debug1: Local version string SSH-2.0-OpenSSH_9.6p1 Ubuntu-3ubuntu13.5 
[11:15:31] <wm-bb>	 <yochayco> debug1: Remote protocol version 2.0, remote software version OpenSSH_8.4p1 Debian-5+deb11u3 
[11:15:33] <wm-bb>	 <yochayco> debug1: compat_banner: match: OpenSSH_8.4p1 Debian-5+deb11u3 pat OpenSSH* compat 0x04000000 
[11:15:34] <wm-bb>	 <yochayco> debug2: fd 3 setting O_NONBLOCK 
[11:15:36] <wm-bb>	 <yochayco> debug1: Authenticating to bastion.wmcloud.org:22 as 'yochayco' 
[11:15:38] <wm-bb>	 <yochayco> debug3: record_hostkey: found key type ED25519 in file /home/yochayc/.ssh/known_hosts:6 
[11:15:40] <wm-bb>	 <yochayco> debug3: load_hostkeys_file: loaded 1 keys from bastion.wmcloud.org
[11:16:19] <wm-bb>	 <yochayco> My config file:
[11:16:19] <wm-bb>	 <yochayco> ```
[11:16:21] <wm-bb>	 <yochayco> Host bastion.wmcloud.org  HostName bastion.wmcloud.org
[11:16:22] <wm-bb>	 <yochayco>   User yochayco  IdentityFile ~/.ssh/wmil
[11:16:24] <wm-bb>	 <yochayco>   IdentitiesOnly yes
[11:16:25] <wm-bb>	 <yochayco> Host  serviceapp.glamwikidashboard.eqiad1.wikimedia.cloud  HostName  serviceapp.glamwikidashboard.eqiad1.wikimedia.cloud
[11:16:27] <wm-bb>	 <yochayco>   User yochayco  Port 22
[11:16:28] <wm-bb>	 <yochayco>   ProxyJump bastion.wmcloud.org:22  IdentityFile ~/.ssh/wmil
[11:16:30] <wm-bb>	 <yochayco> Host webapp.glamwikidashboard.eqiad1.wikimedia.cloud
[11:16:31] <wm-bb>	 <yochayco>   HostName webapp.glamwikidashboard.eqiad1.wikimedia.cloud  User yochayco
[11:16:33] <wm-bb>	 <yochayco>   IdentityFile ~/.ssh/wmil  Port 22
[11:16:35] <wm-bb>	 <yochayco>   ProxyJump bastion.wmcloud.org:22```
[11:18:47] <dhinus>	 does the connection hang, or does it fail with an error?
[11:20:53] <wm-bb>	 <yochayco> it just says that the connect closed and returns to my normal shell, no hang and no real connection
[11:50:40] <dhinus>	 I found this in the bastion logs: Failed publickey for yochayc
[11:53:50] <dhinus>	 can you check the SSH key you are using matches the one shown at https://idm.wikimedia.org/keymanagement/ ?
[11:56:07] <wm-bb>	 <yochayco> It matches. Yesterday I had some worse attempts so maybe it's one of them. Is this failed log from the last 5 hours?
[11:57:44] <dhinus>	 let me check
[11:58:22] <dhinus>	 there are several, the last one is at 10:58 UTC
[11:58:36] <dhinus>	 1 hour ago
[11:59:28] <wm-bb>	 <yochayco> I'll double check in a few minutes
[12:00:26] <dhinus>	 thanks, it's also possible the bastion is not checking the right key, but that's weird
[12:26:32] <wm-bb>	 <yochayco> It's the same key.
[12:26:33] <wm-bb>	 <yochayco> The public key is: `ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIJ6uavBYkSe5Yuo1fQN7z/2CKo95lPi8pGk2ByVkX5U2 yochai.cohenor@gmail.com`
[13:30:34] <AntiComposite>	 tools-static.wmflabs.org appears to be timing out
[13:31:09] <AntiComposite>	 https://tools-static.wmflabs.org/anticompositetest/flickr.jpg is a very small file (not actually a jpeg) but I get "This web service cannot be reached. Please contact a maintainer of this project."
[14:04:25] <dcaro>	 looking
[14:10:37] <dcaro>	 yep, it's blocked on NFS, will reboot, we are looking to see if there's any other issues going around, it might have been a network hiccup
[14:10:57] <dcaro>	 !log tools reboot tools-static-15 due to nginx stuck on nfs
[14:11:00] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[14:24:48] <dcaro>	 AntiComposite: should be fixed now (now the error is a bit different though, but seems to be due to the image not being valid, not static not responding)
[15:44:17] <wm-bb>	 <yochayco> Could you please see if bastion is checking the right key? 🙏 (re @wmtelegram_bot: <dhinus> thanks, it's also possible the bastion is not checking the right key, but that's weird)
[15:45:08] <dhinus>	 yochayco yep sorry, I had to concentrate on other things but I will try to find out what's wrong!
[15:45:34] <dhinus>	 or maybe someone else has an idea of what could be wrong?
[15:46:07] <dhinus>	 I can ssh just fine with ssh fnegri@bastion.wmcloud.org
[15:49:31] <dhinus>	 !log admin cumin 'P:base::cloud_production' 'rm /var/lib/prometheus/node.d/kernel-panic.prom' T382961
[15:49:37] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[15:49:38] <stashbot>	 T382961: Kernel error metrics have overlapping definitions - https://phabricator.wikimedia.org/T382961
[16:13:33] <bd808>	 I was curious so I checked the #wikimedia-cloud-feed channel and I can see that an automated alert for tools-static fired at 12:18Z. I was hoping that we actually had some monitoring there still and that seems to be true.
[17:12:47] <dhinus>	 yochayco: the logs show the SSH key that's failing is a different one: ED25519 SHA256:ky6y0xeZoP+ysaq1j2lCjj8icSoQ1ppvNdQ2mwS1mIM
[17:13:09] <dhinus>	 maybe your client is sending another key you have in your computer?
[17:14:08] <wm-bb>	 <yochayco> I see. I'll check that in an hour. Thanks 🙏
[19:50:38] <MrBABUCHACKO56>	 MrBABUCHACKO/T186061/53/17
[19:57:39] <AntiComposite>	 MrBABUCHACKO56, can we help you?
[20:38:01] <MrBABUCHACKO>	 Wikimedia Cloud, can it computing factual base on time-bound real factors?
[20:39:35] <MrBABUCHACKO>	 For one travelling at a speed of 1000 miles/hr while some others account facts without skipping at a speed of 90miles/hr.
[20:41:13] <wm-bb>	 <yochayco> Hi dhinus, thank you for your patience and help.
[20:41:14] <wm-bb>	 <yochayco> It seems that the key which the log referred to is the fingerprint of my public key.
[20:41:15] <wm-bb>	 <yochayco> I verified it this way:
[20:41:17] <wm-bb>	 <yochayco> ```
[20:41:18] <wm-bb>	 <yochayco> ssh-keygen -lf ~/.ssh/wmil.pub 256 SHA256:ky6y0xeZoP+ysaq1j2lCjj8icSoQ1ppvNdQ2mwS1mIM yochai.cohenor@gmail.com (ED25519)```
[20:43:18] <wm-bot>	 !log jeanfred@tools-sgebastion-10 tools.integraality Deploy 9268348 (Fix  unit tests in test_app)
[20:43:20] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.integraality/SAL
[20:43:30] <wm-bot>	 !log jeanfred@tools-sgebastion-10 tools.integraality Deploy 9268348 (Fix 'QueriesTests' unit tests in test_app)
[20:43:31] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.integraality/SAL
[20:43:37] <wm-bot>	 !log jeanfred@tools-sgebastion-10 tools.integraality Deploy 5941f3f (Upgrade all Python dev-dependencies)
[20:43:38] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.integraality/SAL
[20:43:43] <wm-bot>	 !log jeanfred@tools-sgebastion-10 tools.integraality Deploy a76ceec (Switch from Pipenv to uv as project manager)
[20:43:45] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.integraality/SAL
[21:36:06] <MrBABUCHACKO>	 https://github.com/apache/seatunnel.git/T186061
[21:50:46] <MrBABUCHACKO>	 content/authentication/managing-commit-signature-verification/index.md/T186061
[21:54:54] <MrBABUCHACKO>	 $ gpg --list-secret-keys --keyid-format=long
[21:54:55] <MrBABUCHACKO>	 ------------------------------------
[21:54:55] <MrBABUCHACKO>	 sec   4096R/3AA5C34371567BD2 2016-03-10 [expires: 2017-03-10]
[21:54:56] <MrBABUCHACKO>	 uid                          Hubot <hubot@zygnetcorealf9.com>
[21:54:56] <MrBABUCHACKO>	 ssb   4096R/4BB6D45482678BE3 2016-03-10
[21:58:07] <bd808>	 !kick MrBABUCHACKO 
[22:00:30] <bd808>	 `!kb` I guess is what I meant to use from wmopbot 
[22:01:03] <bd808>	 thanks AntiComposite 
[22:04:14] <bd808>	 !kick is Did you mean `!kb $1` $infobot_nick ?
[22:04:14] <wm-bot>	 Key was added
[22:04:29] <bd808>	 !kick jsut-a-test
[22:04:30] <wm-bot>	 Did you mean `!kb jsut-a-test` bd808 ?
[22:05:59] <bd808>	 !highfive
[22:05:59] * wm-bot highfives bd808!
[22:14:06] <MrBABUCHACKO>	 +ibabucha
[22:14:15] <bd808>	 kb MrBABUCHACKO 
[22:14:18] <bd808>	 !kb MrBABUCHACKO 
[23:53:13] <wm-bb>	 <forzagreen> Hi, I maintain a tool in Toolforge (python Flask API + MariaDB). We would like to have some insights and analytics about the API usage. In particular, the number of requests.
[23:53:14] <wm-bb>	 <forzagreen> Is there a tool that provides this information ?
[23:53:15] <wm-bb>	 <forzagreen> (I found this Grafana dashboard, but it's doesn't have the information:  https://grafana.wmcloud.org/d/TJuKfnt4z/kubernetes-namespace)
[23:53:17] <wm-bb>	 <forzagreen> Or should I parse the logs myself ? If yes, do we have a Prometheus or equivalent to store metrics ?