[12:20:43] late lunch [14:13:39] o/ [15:09:39] https://gerrit.wikimedia.org/r/c/operations/docker-images/production-images/+/1008486 CR for fixing the flink operator img if anyone has time to look [15:43:08] \o [15:43:24] curious on thoughts... the flink RichFunctionSource is super simple, but deprecated. I spent friday working out how to use the new source API and think i've finally figured out the control flow and checkpoints, but this is 350 lines in 10 classes [15:44:33] i guess the question is...should we use the new apis? Or assume this is so cumbersome flink will have to provide a middle ground before fully deprecating RichSourceFunction [15:45:06] ebernhardson: 350 lines/10 classes ouch :( [15:45:41] to be fair, 1 class and 45 lines implements the actual bits that calculate what to emit when :P [16:00:41] TIL that the name for the twö döts ön söme vöwels, "diaeresis/dieresis/diaresis" (also known as "trema" among French speakers and "umlaut" to fans of German), is pronounced "di-ER-e-sis" not "di-a-RE-sis"—sorry if I led you astray, gehel ! (Why do two dots need three names? I prefer "rock dots" anyway....) [16:01:58] dcausse, pfischer: triage meeting: https://meet.google.com/eki-rafx-cxi [16:02:27] oops [18:01:53] lunch, back in ~40 [18:25:02] inflatador: mo.ritzm asked me to reboot flink-zk* for https://phabricator.wikimedia.org/T356239. I'll kick those off in a little bit [18:46:41] ryankemper cool, as long as we do 1 node at a time, should be fine [18:47:34] there are some zkcli commands you can use to check the cluster state, 1 sec [18:47:56] that also reminds me that we don't have flink-zk on the typical ZK dashboard https://grafana.wikimedia.org/d/000000261/zookeeper?orgId=1&refresh=5m&var-cluster=main-eqiad&var-datasource=eqiad%20prometheus%2Fops&var-zookeeper_hosts=All [18:50:21] the command is `/usr/share/zookeeper/bin/zkCli.sh config` , it should spit out the cluster members. Need to figure out if that's updated on the fly, maybe we can play around w/that during pairing [19:24:11] inflatador: ack, yeah let's do the restarts during pairing [19:24:56] * ebernhardson realizes that the problem with not implementing boundedness of a source...is that the test utilities really prefer something that ends [19:28:53] ebernhardson: things are looking good with the reindexing (through afwiki). If you want to kick off the cloudelastic reindexing today or tomorrow, that would be great! [19:32:04] gehel: having some issues joining the meet, one sec [19:32:12] ack [19:32:18] Trey314159: excellent [20:09:32] another small CR to for production docker images...forgot to update the changelog last time https://gerrit.wikimedia.org/r/c/operations/docker-images/production-images/+/1008534 [20:33:38] * ebernhardson now realizes this reindexing is going to spam #wikimedia-operations with reindexing invocations [20:39:16] ebernhardson ACK, will work on that...we can probably move it into #wikimedia-data-platform . b-tullis has done yeoman's work on that [20:40:06] also, small CR to move some elastic hosts back to insetup. Apparently they can't reimage if they're in the prod role ;( https://gerrit.wikimedia.org/r/c/operations/puppet/+/1008528 [21:10:20] dr0ptp4kt you might wanna hit up DC Ops for https://phabricator.wikimedia.org/T358727#9592401 , I'm not sure about the next steps [21:51:21] we have an UBN on the cross cluster search settings: https://phabricator.wikimedia.org/T358541 . I believe we fixed this last week, so will take another look [21:56:46] hmm, the seeds look updated but it's not working :( [21:57:37] inflatador: ahh, i think i see. The prod clusters specify the inter-node transport port in the cluster settings for cross-cluster, on cloudelastic you have the http port [21:58:02] inflatador: you should be able to check if it works via: curl 'https://cloudelastic.wikimedia.org:9243/*:*_content/_search' [21:58:37] ebernhardson thanks, will fix shortly [22:47:36] CCS is fixed, working on improved monitoring as well