[07:23:34] <dcausse>	 inflatador: good to know! but if it's for operator flink last time we checked nomad was not supported by the flink ecosystem
[07:31:10] <dcausse>	 s/operator/operating/
[10:17:18] <dcausse>	 lunch+errand
[13:30:44] <gehel>	 pfischer: I did not ping you explicitly, but I sent a meeting invite to chat with you and dcausse about the Search Update Pipeline. Happening now in https://meet.google.com/auy-phue-ntw (sorry for the last minute)
[13:38:48] <gehel>	 pfischer: moved to tomorrow instead
[15:02:04] <gehel>	 inflatador: triage is starting in https://meet.google.com/eki-rafx-cxi
[15:41:42] <moritzm>	 hi, I've build a forward port of the latest Java 8 security fixes for Buster, used by wdqs/wcqs, how do you want to go about rolling them out? shall I deploy the updated debs initially on wdqs-test or right away on all wdqs Blazegraph clusters?
[15:53:15] <ryankemper>	 moritzm: I think all clusters is fine. let us know when those are ready and inflatador/I can get those out this week
[16:01:53] <moritzm>	 sure thing, I'll go update those in a bit and will let you know when they are ready
[16:07:24] <moritzm>	 ryankemper: wcqs* and wdqs* both upgraded to the new JRE packages
[16:07:43] <ryankemper>	 moritzm: ty! will circle back with ya when we've got those out
[16:08:08] <inflatador>	 +!
[16:08:10] <inflatador>	 +1
[16:13:13] <moritzm>	 thx
[16:39:02] <ebernhardson>	 hmm, unfortunate.  The thing i setup on friday to ingest simplewiki with some vectors into vespa is still running. But on friday it was maxing out all the cores, now it's still ingesting but server is 95% idle :(  will have to poke another day
[16:40:04] <dcausse>	 :/ might need more heap or something?
[16:40:36] <ebernhardson>	 maybe, it's not clear. I took ~30min poking logs and metrics but i can't see anything where it's obviously complaining. It claims to have a 5G heap and is maintaining 1G free
[16:52:56] <ebernhardson>	 ci failing more and more often with `no space left on device` :(
[17:10:44] <inflatador>	 Haven't read this but lokos interesting https://aclanthology.org/2022.flp-1.5.pdf
[17:11:06] <inflatador>	 ebernhardson can you tell which CI hosts you're getting? I can try a quick cleanup
[17:12:03] <ebernhardson>	 inflatador: hmm, i re-ran the pipeline (it's gitlab ci) and it worked, not sure how to find the old failed rnu, sec
[17:14:14] <ebernhardson>	 inflatador: yea unfortunately, i'm not seeing where i can get the log of the failed run after it's been re-run
[17:15:31] <inflatador>	 ebernhardson understood. If you wanna link the PR I can run it down with releng and/or IF
[17:19:14] <ebernhardson>	 inflatador: it was this one: https://gitlab.wikimedia.org/repos/search-platform/discolytics/-/merge_requests/27
[17:19:30] <ebernhardson>	 i find it curious that gitlab seems to remove any mention of the failed run, at least in the obvious places
[17:20:15] <inflatador>	 I think I found it: https://gitlab.wikimedia.org/repos/search-platform/discolytics/-/pipelines/21197/builds
[17:20:29] <ebernhardson>	 that didn't take you long :P
[17:20:34] <inflatador>	 if you click "failed" in the status column
[17:20:37] <ebernhardson>	 indeed that looks right
[17:20:56] <inflatador>	 nut, blind squirrel, etc etc ;)
[17:44:49] <dcausse>	 dinner
[17:52:16] <inflatador>	 re: gitlab CI disk space. per convo w/mutante , it's a known issue. Added here as requested: https://phabricator.wikimedia.org/T326558
[18:01:30] <ebernhardson>	 well, at least something will happen some day :)
[18:01:50] <ebernhardson>	 i guess i was somehow optimistic that a container based ci would have a much easier time doing auto-cleanup
[18:22:29] <inflatador>	 Lunch, back in ~40
[19:14:43] * ebernhardson still doesn't understand how all these sparql dags tie together
[19:15:05] <ebernhardson>	 every time i need to fix them i can't quite follow how all the things actually work together
[19:20:58] <inflatador>	 back
[20:06:49] * ebernhardson tracks down how it all works...i should save this somewhere: https://phabricator.wikimedia.org/P49481
[20:08:33] * ebernhardson wonders if there would be some way to make our airflow emit this kind of info directly instead of having to parse it all out...but airflow doesn't know inputs/outputs they are all random cli argumnets
[20:25:35] <ottomata>	 inflatador: dcausse pfischer, do you think you all will want to try kafka stretch for multi DC search update pipeline, so we can do huge events without messing with kafka main?
[20:40:01] <ottomata>	 made a task for ya: https://phabricator.wikimedia.org/T340492
[20:40:02] <ottomata>	 :)
[20:56:41] <inflatador>	 moritzm do you have a phab task for the java updates? Happy to make one if not
[21:01:19] <inflatador>	 ^^ we ended up making one https://phabricator.wikimedia.org/T340482
[21:03:40] <moritzm>	 ack, I hadn't created one indeed
[21:43:54] <inflatador>	 ottomata excellent, I'd like to help out on that