[00:31:40] <wm-bb>	 <bd808> tgr_: a floating ip would be needed, or we would need to setup something new/custom with the shared proxy. Floating ip would be easier.
[00:38:31] <tgr_>	 Thanks! Filed T315198 about it.
[00:38:32] <stashbot>	 T315198: Request floating IP for matrix Cloud VPS project - https://phabricator.wikimedia.org/T315198
[15:39:49] <Penelope>	 Hello.
[15:40:02] <Penelope>	 I've got a stuck job and it has no queue name.
[15:40:16] <taavi>	 hi! which tool?
[15:42:16] <Penelope>	 dbreps is the "become" tool.
[15:42:41] <Penelope>	 "qstat -xml" shows the pending job.
[15:42:45] <Penelope>	 But with no queue name.
[15:42:58] <Penelope>	 So I can't SSH to a host to kill it manually like the docs say.
[15:43:20] <Penelope>	 It's also been stuck for like four days. I'm wondering if there's an easy way to add a timeout.
[15:43:34] <Penelope>	       <JB_owner>tools.dbreps</JB_owner>
[15:43:34] <Penelope>	       <state>Eqw</state>
[15:43:34] <Penelope>	       <JB_submission_time>2022-08-11T10:00:12</JB_submission_time>
[15:43:34] <Penelope>	       <queue_name></queue_name>
[15:47:36] <bd808>	 It looks like there are 2 jobs in the Eqw state for that tool and both failed to queue due to a transient LDAP lookup error.
[15:48:07] <bd808>	 Penelope: have you tried to `qdel` those yourself yet?
[15:48:55] <wm-bot>	 !log tools.bridgebot <lucaswerkmeister> Double IRC messages to other bridges
[15:48:56] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.bridgebot/SAL
[15:48:59] <Penelope>	 I have not. I didn't see that under https://wikitech.wikimedia.org/wiki/Help:Toolforge/Grid#Stuck_jobs
[15:49:05] <Penelope>	 But I see it in the section above now!!
[15:49:49] <Penelope>	 qdel worked, thank you!!
[15:49:52] <bd808>	 awesome
[15:50:21] <Penelope>	 Any way to add a timeout to jsub? Or do people wrap in `timeout`?
[15:50:30] <Penelope>	 legoktm is using this:
[15:50:33] <Penelope>	 @hourly jsub -once -quiet -mem 2G -N rusty /data/project/dbreps/src/database-reports/target/release/dbreps2
[15:50:58] <Penelope>	 Which then iterates through and runs reports that need updating. But if the "rusty" job gets stuck, the reports back up.
[15:51:33] <Penelope>	 I see `timeout` is installed so maybe wrapping in that is easiest?
[15:52:00] <bd808>	 the issue with both of these was that the submission failed from some kind of LDAP lookup hiccup. I don't think there is any way to tell grid engine "try to submit this, but if submission fails then automatically delete the failed submission tombstone after N minutes"
[15:52:56] <bd808>	 the tombstone would kept additional jobs from firing because of the `-once` flag on the submission
[15:53:54] <Penelope>	 -l h_rt= ?
[15:54:04] <Penelope>	 I wonder if that would help.
[15:54:08] <Penelope>	 From https://bioinformatics.mdc-berlin.de/intro2UnixandSGE/sun_grid_engine_for_beginners/how_to_submit_a_job_using_qsub.html and such.
[15:54:48] <Penelope>	 I constantly forget about "qstat -xml" btw and I find the truncated job names in regular "qstat" output infuriating.
[15:55:07] <bd808>	 that would stop a job that was running and kept running, but I don't think it would do anything to change an Eqw failure
[15:55:25] <Penelope>	 Gotta set up another job engine that watches the o.g. ;-)
[15:55:27] <bd808>	 everything about grid engine is infuriating ;)
[15:55:36] <Penelope>	 Hah, for sure.
[15:55:53] <Penelope>	 At least the XML output is verbose.
[15:55:59] <Penelope>	 Thanks for the qdel tip.
[15:57:45] <legoktm>	 Penelope: I think we should be able to move this over to kubernetes (`toolforge-jobs`), which hopefully lets us ignore all the grid engine issues
[15:57:57] <bd808>	 https://wikitech.wikimedia.org/wiki/Help:Toolforge/Jobs_framework is our beta quality replacement for grid engine with Kubernetes as the backend runtime. It can take over for many grid jobs today, but not all yet. Jobs that need multiple runtime languages (like PHP & Ruby at the same time) or external binaries/libraries (like image processing things) are not supported yet.
[15:58:20] <legoktm>	 yep, that
[15:58:25] <Penelope>	 I know Kubernetes fairly well from my professional life, so no objection from me.
[15:58:32] <Penelope>	 Tho it also has dumb behavior.
[15:58:43] <legoktm>	 I'm pretty sure that this is all pure Rust so it should work fine in the `standalone` container
[15:58:44] <Penelope>	 Where like you gotta specify "-o json" to get useful info.
[15:58:57] <Penelope>	 Err, "-o yaml", whatever.
[16:19:13] <mutante>	 SSLV3_ALERT_CERTIFICATE_EXPIRED] sslv3 alert certificate expired (_ssl.c:2622)'))': /hosts/clouddumps1001.wikimedia.org/update
[16:22:22] <firefly_wp>	 bd808 just had the same issue with one of my tasks (stuck in Eqw state)
[16:22:26] <firefly_wp>	 Had to be deleted manually
[17:24:57] * dcaro off
[22:50:02] <Guest35>	 I need help with my communication
[23:43:27] * Platonides suggests as a first step, that he should stay longer in the channel