[06:08:59] <elukey>	 ryankemper: thanks for the merge!
[06:15:43] <elukey>	 (good morning :)
[06:16:20] <elukey>	 wdqs1013 seems to have some issue with blazegraph, afaics it should be safe to depool/restart but lemme know if there is anything ongoing
[06:45:33] <zpapierski>	 dcausse: it seems we are down to wdqs1007, wdqs1012 and wdqs1013
[06:48:06] <zpapierski>	 do we have any we can proceed in CEST without MrG?
[06:53:06] <zpapierski>	 elukey: nothing going on with the wdqs1013 atn\
[06:53:12] <zpapierski>	 s/atn/atm
[07:00:54] <elukey>	 zpapierski: o/ so ok to depool/restart blazegraph in there?
[07:03:15] <zpapierski>	 sure
[07:04:11] <zpapierski>	 and thanks
[07:32:22] <dcausse>	 zpapierski: I'm afraid we'll have to wait for Ryan to continue
[07:32:50] <dcausse>	 elukey: thanks for the patch over the week-end! :)
[07:32:54] <dcausse>	 and sorry for the noise
[07:35:29] <elukey>	 dcausse: np! Happy to help :) I am restart bz on wdqs1013
[07:35:39] <dcausse>	 thanks!
[07:35:53] <elukey>	 (all done :)
[07:36:03] <zpapierski>	 thx!
[07:36:32] <zpapierski>	 anyway - 3 servers left, we - and by we I mean Ryan - should probably be able to finish today
[07:36:46] <zpapierski>	 sounds weird
[07:37:19] <dcausse>	 yes should be done today hopefully
[09:14:01] <zpapierski>	 ejoseph: how are things?
[09:14:45] <zpapierski>	 let me know when you have time, so we can sync up via meet
[09:32:38] <ejoseph>	 To join the video meeting, click this link: https://meet.google.com/ati-hniw-bod
[09:32:38] <ejoseph>	 Otherwise, to join by phone, dial +27 10 823 0514 and enter this PIN: 821 382 084#
[09:32:38] <ejoseph>	 To view more phone numbers, click this link: https://tel.meet/ati-hniw-bod?hs=5
[09:34:09] <zpapierski>	 wait a sec, I'm having issues with my bt headphones
[09:34:25] <volans|off>	 this way anyone can join that call, you probably want to create a new one, this is a public channel ;)
[09:34:28] <zpapierski>	 it seems ubuntu upgrade might've broken something after all
[09:34:43] <ejoseph>	 Oh
[09:34:49] <zpapierski>	 volans: sure?
[09:35:08] <volans>	 I think phone + pin should be enough no?
[09:35:29] <zpapierski>	 oh right
[09:36:02] <zpapierski>	 ejoseph: post only links next time, any non-wmf person would need to be permitted antry
[09:36:07] <zpapierski>	 s/antry/entry
[09:36:28] <ejoseph>	 oh ok
[10:30:24] <dcausse>	 lunch
[10:41:33] <zpapierski>	 break
[13:51:13] <elukey>	 hey folks super sorry for the dse k8s meeting, totally spaced out doing maintenance on kafka :(
[13:59:18] <dcausse>	 elukey: np!
[14:22:17] <ejoseph>	 The SSH command responded with a non-zero exit status. Vagrant
[14:22:17] <ejoseph>	 assumes that this means the command failed. The output for this command
[14:22:17] <ejoseph>	 should be in the log above. Please read the output to determine what
[14:22:17] <ejoseph>	 went wrong.
[14:25:45] <ejoseph>	 the last log i got while setting up cirruss search
[14:26:30] <dcausse>	 ejoseph: while doing "vagrant up" ?
[14:27:16] <zpapierski>	 ejoseph: provide the "red" logs from the command
[14:27:55] <ejoseph>	 yes
[14:30:35] <ejoseph>	 https://usercontent.irccloud-cdn.com/file/NXtta9yU/Screenshot%202021-10-19%20at%2004.21.19.png
[14:31:00] <zpapierski>	 not enough, there has to be more issues before that
[14:33:15] <dcausse>	 hm... if the catalog was "Applied" it means that puppet did provision properly...
[14:33:45] <dcausse>	 sorry my bad
[15:08:31] <Trey314159>	 ryankemper: are you still having trouble connecting?
[15:09:00] <ryankemper>	 Trey314159: yeah, I’m trying restarting
[15:09:08] <Trey314159>	 cool
[16:20:47] <elukey>	 folks qq - given https://grafana.wikimedia.org/d/000000489/wikidata-query-service?orgId=1&viewPanel=8&refresh=1m&from=now-12h&to=now&var-cluster_name=wdqs for wdqs1013, can I repool it? (depooled this morning after restarting blazegraph)
[16:21:04] <dcausse>	 elukey: I repooled it :)
[16:21:07] <elukey>	 ah!
[16:21:12] <elukey>	 nevermind then :)
[16:27:42] <elukey>	 ah also congrats all for the new streaming updater
[16:27:48] <elukey>	 I know it was a loong project :)
[16:28:38] <dcausse>	 thanks :)
[16:59:11] <ebernhardson>	 does try-with-resources do anything (useful) with ByteArrayOutputStream? Maybe it's just how i'm using it, but to get the content from the baos i have to close the thing wrapping it, which closes the byte array
[16:59:37] <ebernhardson>	 i guess not a big deal, but something about this feels awkward and i'm probably doing it wrong :) https://phabricator.wikimedia.org/P17502
[17:03:54] <dcausse>	 ByteArrayOutputStream no but I think ObjectOutputStream may have some buffers to flush, but in your example you should declare them in the same try( block ) I think
[17:04:54] <dcausse>	 you can have multiple declarations in there
[17:07:16] <dcausse>	 hm this won't work
[17:07:25] <ebernhardson>	 yea the ObjectOutputStream has to be closed, otherwise it doesn't finish writing
[17:08:02] <ebernhardson>	 yea can't declare in the same block, or can but then have to flush/close the oos manually. Ok maybe this is fine then :) Something about it just felt off
[17:08:22] <dcausse>	 it definitely feels weird
[17:09:06] <dcausse>	 might just that I rarely used ByteArrayOuputStream with try-with-res..?
[17:09:28] <ebernhardson>	 that was my other thought, maybe try-with is useful for OutputStream generally, but pointless in the ByteArray case
[17:10:05] <dcausse>	 yes I think so
[17:10:45] <ebernhardson>	 ok, without the nested try blocks it looks less silly :)
[17:18:17] <dcausse>	 dinner
[20:36:41] <mutante>	 hey, is this where work on search-loader2001  is happening as well?
[20:36:56] <mutante>	 just wanted to report this:  The following units failed: wmf_auto_restart_mjolnir-kafka-bulk-daemon.service
[20:37:08] <mutante>	 but maybe it's still WIP ?
[20:37:21] <ebernhardson>	 mutante: hmm, we have some graphs, lemme check.
[20:37:32] <ebernhardson>	 This should be used, but we might not notice codfw breaking since users don't read from codfw
[20:37:34] <mutante>	 ebernhardson: thanks, just going through Icinga web UI
[20:38:02] <mutante>	 we can ACK it but even nicer would be to change puppet so it only monitors where users read from I guess :)
[20:38:53] <ebernhardson>	 well, we still care that codfw is broken. I'm just saying we usually find out things are broken because someone tells us they aren't getting the data. They wont notice if they don't read it :)  Indeed something is wrong in codfw...
[20:39:10] <mutante>	 ah, *nod* yea
[20:39:14] <mutante>	 alright
[20:41:19] <ebernhardson>	 hmm, curious only codfw failed. It's having trouble with a wiki it doesn't know anything about (shnwiki). It should probably fail more gracefully
[20:42:24] <ebernhardson>	 mutante: certainly ack for now, i'll see if i can make any easy fix to log instead of fail
[20:44:27] <mutante>	 ebernhardson: ACK, will ACK :) thanks
[22:21:03] <ebernhardson>	 to follow-up on above, it looks like the root cause is a change to codfw configuration broke the cross-cluster autodetection. mjolnir finds 3 clusters from search.svc.eqiad.wmnet, but only 1 from search.svc.codfw.wmnet