[11:20:08] <dcausse>	 lunch
[12:29:52] <pfischer>	 o/ dcausse: my test-flink.sh fails because of Caused by: java.lang.NoSuchMethodError: 'org.apache.flink.streaming.api.datastream.DataStream org.apache.flink.streaming.api.datastream.SingleOutputStreamOperator.getSideOutput(org.apache.flink.util.OutputTag)' at org.wikimedia.discovery.cirrus.updater.producer.graph.ProducerGraphFactory.createStreamGraph(ProducerGraphFactory.java:191) - I tried to reproduce that locally by 
[12:29:52] <pfischer>	 using flink 1.16.0 (the version provided by docker-registry.wikimedia.org/flink:1.16.0-wmf5-20230326 but that does not seem to be the reason
[12:30:22] <pfischer>	 Does that ring a bell?
[12:34:11] <pfischer>	 Another problem I run into is, that the kafka brokers cannot be resolved from within docker buildx/blubber/gitlab ci. So I don’t know how much more effort I should put into that kind of integration test.
[12:41:23] <dcausse>	 pfischer: I don't think we should do much testing at image building, just a quick smoke test to just see that flink can start
[12:42:11] <dcausse>	 the org.apache.flink.streaming.api.datastream.SingleOutputStreamOperator.getSideOutput is interesting...
[12:42:34] <dcausse>	 sounds like a mixed version (compile time vs runtime)
[12:44:16] <dcausse>	 or perhaps we embark flink-streaming-java in the fat jar?
[12:44:36] <dcausse>	 flink-streaming-java should be provided I believe
[12:51:55] <dcausse>	 the test-flink.sh I wrote does not even start flink really, it just builds the stream graph and exit
[12:52:24] <dcausse>	 but I guess that's sufficient enough to capture small dependencies issues
[13:10:20] <pfischer>	 flink-streaming-java is not part of either of the jar(s)-with-dependencies
[13:11:02] <pfischer>	 I’ll add the build-only-the-graph flag and see what happens.
[13:18:01] <inflatador>	 <o/
[13:19:51] <dcausse>	 o/
[13:20:06] <dcausse>	 inflatador: did you publish an image with flink 1.16.1?
[13:23:15] <inflatador>	 dcausse I thought I did? Did it not show up?
[13:23:38] <dcausse>	 inflatador: can't find it in https://docker-registry.wikimedia.org/flink/tags/
[13:25:09] <inflatador>	 build logs says: `2023-04-12 16:22:37,129 [docker-pkg-build] INFO - Successfully tagged docker-registry.discovery.wmnet/flink:1.16.0-wmf6`
[13:25:21] <inflatador>	 Maybe I should've changed a tag somewhere?
[13:25:24] <dcausse>	 ah ok
[13:25:38] <dcausse>	 yes should have been flink:1.16.1-wmf0
[13:25:42] <dcausse>	 cf the changelog
[13:26:41] <inflatador>	 https://gerrit.wikimedia.org/r/c/operations/docker-images/production-images/+/908256/1/images/flink/flink/changelog
[13:27:09] <inflatador>	 I can push a new patch if you like
[13:27:57] <dcausse>	 this is this patch I think: https://gerrit.wikimedia.org/r/c/operations/docker-images/production-images/+/908486
[13:29:19] <inflatador>	 Interesting, the git log does not have that entry. Let me try that again
[13:32:10] <dcausse>	 perhaps this repo, unlike deployment-prep, needs an explicit fetch+rebase?
[13:32:24] <inflatador>	 It looks like one of those git directory errors
[13:33:01] <inflatador>	 it's refusing to update git on the build server
[13:33:08] <inflatador>	 fatal: detected dubious ownership in repository at '/srv/images/production-images'
[13:34:03] <dcausse>	 this issue rings a bell
[13:34:24] <dcausse>	 ottomata: ^
[13:34:35] <inflatador>	 yeah, it was causing problems before...I can still do a `git pull` as root though...one sec
[13:35:39] <inflatador>	 OK, looks like it's building. The directions at https://wikitech.wikimedia.org/wiki/Kubernetes/Images#Production_images don't work; you have to be root as opposed to using sudo
[13:43:04] <inflatador>	 dcausse OK, the image is built. The docker-registry web UI doesn't show it yet though. let's give it a few minutes
[13:43:24] <dcausse>	 inflatador: thanks!
[13:53:18] <dcausse>	 errand
[14:55:06] <ebernhardson>	 \o
[14:56:08] <dcausse>	 o/
[15:57:43] <ottomata>	 inflatador:  ya sudo -i, if instructions don't say that, can you fix?
[16:21:04] <inflatador>	 ottomata will do,  was waiting to see if it worked first
[16:24:43] <inflatador>	 working out first though ;) . Back in ~40
[16:48:44] <ryankemper>	 \o
[16:54:01] <inflatador>	 <o/
[16:54:03] <inflatador>	 back
[18:56:04] <inflatador>	 errand, back in ~30
[19:35:43] <inflatador>	 back
[20:36:23] <ebernhardson>	 huh, unexpected...it works.  41 passed, 1 skipped, 42 total (100% completed) in 00:09:46 
[20:36:41] <ebernhardson>	 (it actually takes 14m, including setting up the environment)
[20:43:21] <Trey314159>	 Good news!