[04:08:29] <Baskerville>	 !log wikisp mars: Updating dependences
[04:08:32] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikisp/SAL
[04:14:16] <Baskerville>	 !log wikisp apollo: Updating zammad to 6.1
[04:14:19] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikisp/SAL
[04:14:27] <AmandaNP>	 Not sure if we are aware, but i'm getting no route to host on one of my WMCS instances
[04:14:57] <AmandaNP>	 172.16.6.23
[04:24:35] <AmandaNP>	 https://phabricator.wikimedia.org/T347661
[05:36:44] <audiodude>	 My instance is also down and I don't know why: https://grafana.wmcloud.org/d/0g9N-7pVz/cloud-vps-project-board?orgId=1&var-project=mwoffliner&var-instance=mwcurator
[06:43:41] <pintoch>	 hello, some of my tools are down apparently because they cannot connect to the tools-db MySQL server anymore:
[06:43:48] <pintoch>	 "Can't connect to MySQL server on 'tools.db.svc.eqiad.wmflabs' (115)"
[06:44:33] <pintoch>	 connecting manually (with the "mysql" command run as the tool account) gives me "ERROR 2002 (HY000): Can't connect to MySQL server on 'tools-db' (115)"
[06:55:45] <RhinosF1>	 I see lots of is down in -feed
[06:55:48] <RhinosF1>	 Including tools-db
[06:55:50] <RhinosF1>	 !help
[06:55:51] <wm-bot>	 If you don't get a response in 15-30 minutes, please create a phabricator task -- https://phabricator.wikimedia.org/maniphest/task/edit/form/1/?projects=wmcs-kanban
[06:55:57] * RhinosF1 makes UBN task anyway
[06:57:15] <RhinosF1>	 https://phabricator.wikimedia.org/T347665
[07:06:39] <dcaro>	 hi! looking
[07:07:29] <RhinosF1>	 dcaro: thanks
[07:32:17] <taavi>	 !status network issues with bullseye based instances
[07:39:51] <Danny_B>	 i can't connect to my instance suddenly, any known issues atm?
[07:42:38] <dcaro>	 Danny_B: yes, working on it T347665
[07:42:38] <stashbot>	 T347665: Multiple CloudVPS instances down - https://phabricator.wikimedia.org/T347665
[07:44:35] <Danny_B>	 thanks. any eta?
[08:07:13] <dcaro>	 we are slowly getting things up, I'd say 1h tops (vms that lost network need to be manually fixed through console)
[08:36:19] <taavi>	 !log admin start script to fix networking on broken bullseye instances T347665
[08:36:24] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[08:36:24] <stashbot>	 T347665: Multiple CloudVPS instances lost their ips (unreachable) - https://phabricator.wikimedia.org/T347665
[08:39:01] <pintoch>	 tools-db is already up again, yay! thank you all! that seems to be a tricky situation to get out of.
[08:40:47] <arturo>	 pintoch: cool, thanks I just fixed the VM
[09:14:50] <dcaro>	 !status most projects already back online, working on the few left
[09:57:28] <Wurgl>	 Something is odd on sgebastin-11 NFS-Problem?
[09:58:16] <arturo>	 yes, we're working on it
[09:58:23] <Wurgl>	 okay
[09:59:21] <Wurgl>	 Just in case … I was saving a file in vi when this happens … is there some backup?
[10:00:23] <dhinus>	 vim usually writes a temp file that you can use to recover what you were editing, but it might depend on the config
[10:01:29] <Wurgl>	 "usually" … if something goes wrong, it usually goes terrible wrong :-(
[10:03:21] <RhinosF1>	 Wurgl: sadly everything has gone terribly wrong today
[10:03:29] <RhinosF1>	 All of WMCS bullseye instances lost network
[10:03:35] <RhinosF1>	 They are working super hard
[10:06:52] <taavi>	 the bastions should be back up at this point
[10:28:53] <arturo>	 !log testlabs shutdown testabscookbook-nfs-[1,2]
[10:28:56] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Testlabs/SAL
[10:33:01] <Wurgl>	 Lucky me. File is not corrupted
[10:38:24] <dhinus>	 yay!
[10:49:55] <arturo>	 !log cloudvirt-canary force-reboot all servers
[10:49:57] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Cloudvirt-canary/SAL
[11:40:43] <taavi>	 !status Ok, hopefully
[11:42:10] <anomie>	 `toolforge-jobs run` is giving me "502 Server Error: Bad Gateway for url: https://api.svc.tools.eqiad1.wikimedia.cloud:30003/jobs/api/v1/run/". Run from login.toolforge.org.
[11:42:44] <taavi>	 yep, just saw that
[11:43:40] <taavi>	 try now?
[11:45:54] <anomie>	 Worked now, thanks
[11:46:12] <taavi>	 !status Ok, hopefully
[12:18:09] <taavi>	 @lucaswerkmeister: is it intentional that notwikilambda still has a restart cron job? I thought that tool was retired already
[12:18:18] <wm-bb>	 <lucaswerkmeister> uh
[12:18:20] <wm-bb>	 <lucaswerkmeister> nope
[12:18:24] <wm-bb>	 <lucaswerkmeister> let me remove that then
[12:19:03] <taavi>	 thanks
[12:19:12] <wm-bot>	 !log tools.notwikilambda <lucaswerkmeister> kubectl delete cronjob restart (T314880)
[12:19:16] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.notwikilambda/SAL
[12:19:44] <wm-bb>	 <lucaswerkmeister> now `kubectl get all` only shows pod, service, deployment and replicaset (1 each), all from the webservice I assume
[12:20:15] <wm-bb>	 <lucaswerkmeister> yeah lexeme-forms has the same resources (except more replicasets because of the rolling restarts I do there)
[12:39:46] <wm-bb>	 <lucaswerkmeister> thanks for spotting it taavi!
[12:40:14] <taavi>	 !log admin taavi@cloudcontrol1005 ~ $ os subnet set a69bdfad-d7d2-4cfa-8231-3d6d3e0074c9 --no-dns-nameservers --dns-nameserver 172.20.255.1
[12:43:14] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[13:28:55] <elukey>	 hi folks! One quick question - I have some difficulties to ssh to vms like deployment-kafka-jumbo-9.deployment-prep.eqiad1.wikimedia.cloud, could it be related to today's outage or something different? I noticed that we are recovered but before starting to dig deep into it I wanted to double check :)
[13:29:35] <dcaro>	 let me look
[13:30:02] <elukey>	 <3
[13:30:22] <dcaro>	 the machine has network and an ip
[13:33:23] <dcaro>	 elukey: can you try now, I ifdown+ifup the interface
[13:33:31] <dcaro>	 also what error do you get?
[13:36:45] <dcaro>	 hmm, I can't ssh either xd let me look
[13:37:06] <elukey>	 dcaro: thanks! It basically hangs, afaict when doing
[13:37:07] <elukey>	 Executing proxy command: exec ssh -a -W deployment-kafka-jumbo-9.deployment-prep.eqiad1.wikimedia.cloud:22 restricted.bastion.wmcloud.org
[13:39:34] <dhinus>	 can you ssh to restricted.bastion.wmcloud.org?
[13:40:06] <taavi>	 let me guess: that instance is failing to run puppet, and so has an outdated firewall configuration which blocks access from the new restricted bastion
[13:40:25] <dhinus>	 ha, that's possible
[13:40:45] <dcaro>	 feels like it yes
[13:40:53] <dcaro>	 ssh packets arrive, but get dropped
[13:41:18] <dhinus>	 the old bastion is still running, but I already removed the config in operations/puppet so I don't think we can use it
[13:41:23] <dcaro>	 seems to be applying puppet
[13:41:38] <dcaro>	 nope
[13:41:42] <taavi>	 ha. not the first time I've seen that on deployment-prep :P
[13:41:43] <dcaro>	 Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Evaluation Error: Error while evaluating a Resource Statement, Evaluation Error: Error while evaluating a Function Call, Class[Profile::Kafka::Broker]: parameter 'statsd' expects a String value, got Undef (file: /etc/puppet/modules/role/manifests/kafka/jumbo/broker.pp, line: 10, column: 5) on node 
[13:41:43] <dcaro>	 deployment-kafka-jumbo-9.deployment-prep.eqiad1.wikimedia.cloud
[13:41:49] <dcaro>	 I saw that error before too :)
[13:41:55] <dcaro>	 elukey: ^ 
[13:42:53] <elukey>	 lol thanks
[13:43:07] <elukey>	 to note that I was working on a different vm, not doing any change to kafka atm
[13:43:16] <elukey>	 but lemme see if I can fix :D
[13:43:30] <dcaro>	 I think it has been broken for a while (that's why id did not update the bastion firewall rule)
[13:44:56] <dcaro>	 elukey: you should be able to ssh to it to debug, added the ferm rule manually, will get properly fixed on the first successful puppet run
[13:45:13] <elukey>	 ah wow nice!
[13:46:20] <dcaro>	 otherwise fixing puppet is going to be a bit harder xd
[13:47:49] <dhinus>	 but how can you SSH if it doesn't accept connections from the bastion?
[13:48:37] <elukey>	 I can ssh now to jumbo-9 thanks
[13:51:16] <dcaro>	 dhinus: I added the ferm rule to accept connections
[13:51:48] <dhinus>	 did you ssh through the non-restricted bastions?
[13:59:26] <dcaro>	 I did not, I used the console
[13:59:51] <dcaro>	 can you ssh from the non-restricted bastions directly? let me try
[14:01:13] <dcaro>	 hmm, does not seem to work
[14:01:20] <dcaro>	 dcaro@urcuchillay$ ssh -vv -a -W deployment-kafka-jumbo-9.deployment-prep.eqiad1.wikimedia.cloud:22 bast_eqiad
[14:07:09] <dhinus>	 I think you can do "ssh -J fnegri@bastion.wmcloud.org root@deployment-kafka-jumbo-9.deployment-prep.eqiad1.wikimedia.cloud"
[14:07:20] <dhinus>	 (s/fnegri/yourusername/)
[14:07:54] <dhinus>	 jumping through bastion.wmcloud.org instead of restricted.bastion.wmcloud.org
[14:09:25] <dcaro>	 let me try
[14:09:35] <dhinus>	 seems to work for me
[14:09:57] <dcaro>	 oh yes, that works like a charm :), I'll save it in my history for the future
[14:10:15] <dhinus>	 but I only thought about it because you wrote "I'll use the prod one" in the other channel, speaking of something completely unrelated :D
[14:10:44] <dhinus>	 but in my mind you made me think "oh yeah we have another bastion!"
[14:11:12] <dcaro>	 that's the one the cloudcumin uses right?
[14:13:21] <dhinus>	 nope, cloudcumin uses the restricted one
[14:14:00] <dcaro>	 oh, what's that one for then?
[14:14:20] <dhinus>	 https://wikitech.wikimedia.org/wiki/Help:Accessing_Cloud_VPS_instances#Setup
[14:14:34] <dhinus>	 according to that page "restricted" is for SREs
[14:14:44] <dhinus>	 and non-restricted for everyone else
[14:15:02] <dhinus>	 not sure if we really need that separation
[14:15:44] <dcaro>	 oh, hmm, yep, seems prone to errors of "this is up for me" when checking connectivity for users
[14:18:01] <dhinus>	 we actually have two non-restricted ones if you look at the instances in the "bastion" project, I'm not sure if they are load-balanced or not
[14:21:16] <taavi>	 they're not, there's just some docs telling that you can use secondary.bastion.wmcloud.org instead if you want
[14:23:01] <elukey>	 totally unblocked folks, thanks for the help!
[14:23:52] <dcaro>	 👍
[14:24:14] <dhinus>	 taavi: thanks!
[14:59:37] <wm-bb>	 <lucaswerkmeister> is there a way to find out whether / when a tool was deleted? I found a tool that’s linked from a Commons template and has some years-old SAL entries but it no longer has a home dir and doesn’t appear in toolsadmin
[14:59:55] <wm-bb>	 <lucaswerkmeister> (not sure if I should say which tool it is – maybe it was vanished for a reason, idk 🤐)
[15:00:26] <wm-bb>	 <lucaswerkmeister> aha, found in in the general tools sal. nevermind :)
[15:01:32] <dcaro>	 should be there yes
[15:25:23] <bd808>	 @lucaswerkmeister: Another thing that can be checked is the /srv/tools/archivedtools directory on tools-nfs.svc.tools.eqiad1.wikimedia.cloud. When tools are removed by the "Disable tool" process the final archive ends up there.
[15:26:23] <wm-bb>	 <lucaswerkmeister> got it, thanks! (it’s indeed there too)
[16:36:31] <wm-bb>	 <kristbaum> Hi, I'm trying to understand the OAUTH procedure following the (outdated) https://wikitech.wikimedia.org/wiki/Help:Toolforge/My_first_Django_OAuth_tool and it recommends testing your OAUTH application on the beta cluster here:
[16:36:32] <wm-bb>	 <kristbaum> https://meta.wikimedia.beta.wmflabs.org/wiki/Special:OAuthConsumerRegistration/propose
[16:36:34] <wm-bb>	 <kristbaum> But to get access to it, it prompts me to confirm my e-mail address, with out ever sending a confirmation code...
[16:36:35] <wm-bb>	 <kristbaum> Is there another way?
[18:06:08] <bd808>	 @kristabaum: A proposed consumer at meta will work for the user that proposes it before it is approved. That would be the method I personally recommend.
[18:53:11] <audiodude>	 Good morning. I'm trying to add a user to our Cloud VPS server. They're setting up their Wikitech account, but I can't find any docs on how to add them to the project. I've looked at https://wikitech.wikimedia.org/wiki/Help:Cloud_VPS_user_roles_and_rights and https://wikitech.wikimedia.org/wiki/Help:Accessing_Cloud_VPS_instances. The project is `mwoffliner`: https://openstack-browser.toolforge.org/project/mwoffliner
[18:53:36] <audiodude>	 slightly related: I see a list of project "members" on that page, and I'm not one of them, even though I can ssh to one of the machines and deploy code there
[19:01:30] <wm-bb>	 <lucaswerkmeister> yes, you’re under “Viewers” (collapsed by default), which means you don’t have permission to add more members IIUC
[19:01:44] <wm-bb>	 <lucaswerkmeister> (more viewers or members, that is. confusing terminology)