[07:20:11] <gehel>	 ryankemper: (for later) looking at https://integration.wikimedia.org/ci/blue/organizations/jenkins/wikidata-query-rdf-maven-release-docker/detail/wikidata-query-rdf-maven-release-docker/58/pipeline/#log-17 (line 17 and around), there is a failed data transfer. I wonder if this is the retrieval of the cache (not super clear from the logs), which could explain the long build time.
[07:20:21] <gehel>	 maybe something to check with rel-eng
[08:08:01] <dcausse>	 going to re-enable the mjolnir dag, I've left it disabled last week to catch-up
[08:17:48] <zpapierski>	 dcausse: can this be abandoned - https://gerrit.wikimedia.org/r/c/wikidata/query/rdf/+/670242 ?
[08:18:35] <dcausse>	 zpapierski: no it might be usefull for the data-transfer cookbook
[08:18:55] <zpapierski>	 ok
[08:19:56] <dcausse>	 the old updater used the triplestore itself to store offsets but I'm not a fan of this solution, using files is not great either but at least makes it indenpendent from the store
[09:01:32] <zpapierski>	 hmm, I think wikiId in cindy that runs the test isn't cirrustest
[09:01:46] <zpapierski>	 dcausse: what's the wikiId for a test runner on cindy?
[09:03:16] <dcausse>	 zpapierski: it depends on the wiki you target from the test
[09:03:22] <dcausse>	 there are multiple wikis
[09:03:26] <zpapierski>	 ah, ok
[09:03:53] <dcausse>	 but most tests [should ]use cirrustestwiki
[09:05:32] <zpapierski>	 mine as well - so id is cirrustestwiki, not cirrustest?
[09:06:24] <dcausse>	 yes
[09:06:30] <zpapierski>	 ok, thx
[09:07:05] <dcausse>	 it's the same as the --wiki option passed to maintenance scripts
[09:07:30] <zpapierski>	 ah, right
[09:20:52] <addshore>	 hows the streaming updater stuff going? =]
[09:21:29] <zpapierski>	 ironing out the last wrinkles, hopefuly
[09:21:37] <dcausse>	 addshore: we're deploying it to k8s at the moment (still resolving small issues)
[09:21:51] <dcausse>	 envoy/certificates and the like
[09:22:03] <addshore>	 sounds like fun :D
[09:22:19] <dcausse>	 otherwise it's been running for a while from yarn and populating https://query-preview.wikidata.org
[09:22:38] <addshore>	 I was just chatting over in #wikimedia-analytics with joal, and excited what other things this updater type system could enable us to do
[09:22:54] <dcausse>	 yes saw the discussion
[09:23:33] <dcausse>	 I'd love to unify some parts of it (e.g. reordering events) and make the "fetch content" part tunable
[09:23:43] <zpapierski>	 and I just noticed I got disconnected from that channel :(
[09:24:04] <dcausse>	 so that one could just concentrate on fetching some data out of the mw-api
[09:26:17] <addshore>	 zpapierski: https://phabricator.wikimedia.org/P16834
[09:26:44] <zpapierski>	 thx!
[09:26:50] <addshore>	 all sounds exciting :) I'll leave yall to it!
[09:43:11] <dcausse>	 tempted to switch to java 11 for the flink image
[09:43:26] <zpapierski>	 should work 
[09:44:02] <zpapierski>	 rdf compiles in full on jdk11 and if there's no code from blazegraph, it should be jdk 11 compliant, AFAIR
[09:44:37] <zpapierski>	 it doesn't change a thing for a code itself, but binary might benefit from the switch
[09:47:36] <dcausse>	 yes, and scala 2.11.12 (which we seem to use) claims to support 11
[09:53:39] <gehel>	 change the runtime? Or the compiler?
[09:54:23] <gehel>	 if we have different targets for flink vs blazegraph/wdqs, we should probably split the project
[09:56:40] <dcausse>	 I think we inherit from the disco-parent source==target==1.8
[09:56:59] <dcausse>	 is it bad to deploy a target == 1.8 on java 11?
[10:00:00] <dcausse>	 for context: I need a package "wmf-certificates" that's only available in buster but only a openjdk-11 image is available on buster yet
[10:00:35] <dcausse>	 so I'm trying to lazy solution here: migrate the flink image to openjdk11
[10:00:44] <dcausse>	 without changing anything else
[10:06:11] <gehel>	 there are a few potential issues (none are blocking)
[10:06:48] <zpapierski>	 I don't think that deploying older binaries on 11 would fail - after all, many libraries are compiled with older jdks 
[10:06:56] <gehel>	 if we compile on JDK8 with target=1.8 and deploy on JDK11, we're not running the same JDK for test and for production. That's a potential issue.
[10:07:19] <gehel>	 s/compile/build/
[10:07:40] <gehel>	 if we build on JDK11 with target=1.8 and deploy on JDK11, I see no issues
[10:07:45] <zpapierski>	 We can build/target on 11  with a 8 target on blazegraph
[10:08:16] <gehel>	 having different targets for different modules is easy
[10:08:30] <gehel>	 having different JDK for the build of different modules is hard
[10:08:43] <zpapierski>	 You don't need to, I think 
[10:08:50] <zpapierski>	 Different level should be enough 
[10:08:58] <dcausse>	 target = 11 requires a java 11 to build
[10:09:04] <zpapierski>	 Yep 
[10:09:28] <gehel>	 you don't _need_ to, but if you use a different JDK for your build and your production, then you're not testing the same thing
[10:10:22] <zpapierski>	 Right, that may be a potential issue 
[10:11:21] <gehel>	 not the end of the world, but one of the potential issue is using method / classes from JDK11 in the blazegraph side and having those not available in production
[10:11:24] <dcausse>	 I'm leaning toward keeping target=1.8 for now and do an exception for the flink app (deploying binaries to 11)
[10:11:52] <gehel>	 testing on 8 and running on 11 is less of an issue, backward compatibility is usually very good
[10:11:54] <zpapierski>	 But this would still require you to use jdk11 
[10:12:07] <dcausse>	 to build or run?
[10:12:26] <dcausse>	 to run yes
[10:12:31] <gehel>	 to build and to run the flink part. blazegraph would still be running on 8
[10:12:49] <gehel>	 Oh, sorry, I miss read.
[10:13:00] <zpapierski>	 Me too, I think 
[10:13:12] <gehel>	 so compile everything with target=1.8 and JDK8
[10:13:14] <zpapierski>	 So, build on 8,run on 11
[10:13:14] <dcausse>	 to build unless I'm mistaken we still use java8 all over the place
[10:13:20] <gehel>	 deploy flink on 11, and blazegraph on 8
[10:13:26] <dcausse>	 yes
[10:13:30] <gehel>	 that's unlikely to cause any issue
[10:13:31] <zpapierski>	 Should be fine anyway 
[10:13:47] <dcausse>	 so: https://gerrit.wikimedia.org/r/c/wikidata/query/flink-rdf-streaming-updater/+/705633
[10:13:57] <gehel>	 we do have a java 11 image ready in jenkins, so if we want to move the build to 11 that's easy (not that we should)
[10:15:11] <gehel>	 it is surprising that wmf-certificates are not available for stretch
[10:15:25] <dcausse>	 it's quite new
[10:15:25] <gehel>	 and we should also have a buster + jdk8 image
[10:15:47] <dcausse>	 do we have production on buster+jdk8?
[10:15:55] <zpapierski>	 honestly, I think we should push our blazegraph stuff to a walled garden of jdk8 and push everything else to at least jdk11
[10:15:58] <zpapierski>	 break
[10:16:02] <gehel>	 probably cassandra (if it has migrated already)
[10:16:18] <gehel>	 wdqs is running buster + 11
[10:16:23] <gehel>	 but not on docker
[10:16:29] <dcausse>	 really?
[10:16:37] <dcausse>	 you mean buster+8
[10:17:05] <gehel>	 sorry, yes, buster+8
[10:17:34] <gehel>	 that version matrix already has too many dimensions for me
[10:17:43] <dcausse>	 true :)
[10:18:28] <gehel>	 lunch
[10:19:05] <dcausse>	 meh.... Error: release 6u6um6ds failed: roles.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:ci:tiller" cannot create resource "roles" in API group "rbac.authorization.k8s.io" in the namespace "ci"
[10:19:07] <gehel>	 (and at some point, we'll migrate everything to Java  16!)
[10:19:50] <dcausse>	 that seems far away :)
[10:19:58] <dcausse>	 17
[10:20:26] <gehel>	 it's not out yet! but targeting an LTS would make sense
[10:21:44] <dcausse>	 yes
[10:23:53] <dcausse>	 lunch
[13:41:09] <zpapierski>	 dcausse: can I take 7.45min of you r time to discuss 4 tickets from streaming updater epic?
[14:05:15] <dcausse>	 zpapierski: sure
[14:05:34] <zpapierski>	 https://meet.google.com/sxm-afao-gef
[14:09:21] <zpapierski>	 gehel: can we close this https://phabricator.wikimedia.org/T280579 ?
[14:14:27] <zpapierski>	 dcausse: I also assumed we can close this one - https://phabricator.wikimedia.org/T247058
[14:15:04] <dcausse>	 zpapierski: I think so
[14:19:10] <zpapierski>	 mpham: I took care of remaining opened tickets on the epic - some were added to "out of scope", some I closed and others I'm addressing seperately
[14:19:34] <zpapierski>	 if I didn't touch some it's because there are already being worked on, or have been addressed yesterday already
[15:01:44] <zpapierski>	 code coverage metric on Cirrussearch from Sonar is a bit unhelpful
[15:02:28] <dcausse>	 code coverage is only for "real" unit test (the ones that extend CirrusTestCase)
[15:02:40] <gehel>	 zpapierski:  for T280579, let me check with service ops. It is part of the process requirement on their side to get a new service deployed
[15:02:41] <stashbot>	 T280579: New service request: WDQS Flink based Streaming Updater - https://phabricator.wikimedia.org/T280579
[15:03:08] <zpapierski>	 gehel: thanks!
[15:03:25] <gehel>	 zpapierski: about coverage, that's by design, we want to promote unit tests over integration tests
[15:03:26] <zpapierski>	 dcausse: yeah, I noticed, bugs me everytime, though
[15:03:46] <topranks>	 gehel, ryankemper: we are about to commence T286069, reconfig of switch buffers on eqiad row D.
[15:03:46] <stashbot>	 T286069: Switch buffer re-partition - Eqiad Row D - https://phabricator.wikimedia.org/T286069
[15:03:47] <zpapierski>	 it would be cool if it included what cindy tests
[15:03:50] <topranks>	 Just a heads up.
[15:03:53] <gehel>	 and IMHO, branch / line coverage is meaningful for unit tests, but integration test should be measured in functional coverage
[15:03:59] <ryankemper>	 topranks: thanks
[15:04:02] <gehel>	 topranks: thanks!
[15:04:27] <topranks>	 Updates in #wikimedia-sre, hopefully be uneventful :)
[15:18:20] <razzi>	 Hi #wikimedia-search, our planned hadoop maintenance will take place right after the row D maintenance is done, and there are 2 running mjolnir jobs on the hadoop cluster we'd like to drain. Would that be an issue?
[15:19:36] <dcausse>	 razzi: looking
[15:22:09] <ryankemper>	 (apparently the row d maintenance is done btw, looks like we didn't really lose connectivity at all)
[15:22:36] <dcausse>	 razzi: feel free to kill the 2 running mjolnir jobs when you start
[16:51:03] <dcausse>	 giving up for today, still getting failed on exception: javax.net.ssl.SSLHandshakeException: PKIX path building failed: even with the wmf-certificates package installed
[16:51:29] <dcausse>	 have to run, will update the standup notes later tonight