[07:25:12] <ryankemper>	 gehel: brian and I paired on the wdqs1010 stuff; that alert's resolved now. I reimaged elastic1068 so that alert's cleared as well. there's two other elastic hosts with PSU failures so I opened up dc-ops tickets for them
[07:25:49] <gehel>	 ryankemper: thanks for the update ! Go to bed now !
[07:26:14] <ryankemper>	 wrt the elastic dependency issues, I've been getting pcc failures on https://gerrit.wikimedia.org/r/c/operations/puppet/+/757700. usually the `requests.exceptions.TooManyRedirects: Exceeded 30 redirects` means a transient pcc failure, but not sure if it's an actual problem with the patch potentially
[07:26:16] <ryankemper>	 gehel: yessir
[07:42:08] <gehel>	 inflatador: ping me when around later today (tomorrow for you?), I can give you a few pointers on how to get more comfortable with our puppet code.
[08:56:58] <gehel>	 re-reading https://puppet.com/docs/puppet/7/lang_containment.html (and a few other resources), I think we should use the `contain` function a lot more. But that also requires thinking more precisely about what we mean in each class.
[08:57:12] <gehel>	 There might be a need for a larger conversation with our SREs.
[09:28:28] <gehel>	 errand, back in a few
[09:48:13] <ejoseph>	 Hello Guys
[09:48:20] <ejoseph>	 I was told to come back next week
[09:48:26] <ejoseph>	 So i am back today
[09:49:27] <dcausse_>	 o/
[09:55:14] <gehel>	 ejoseph: damn!
[09:55:26] <gehel>	 So you did not get the papers you need yet?
[10:55:37] <ejoseph>	 Nope, I couldn't get it
[10:55:48] <ejoseph>	 I have to go back next week
[11:19:33] <ejoseph>	 dcausse: if you want to continue with cirrussearch i am available now
[11:20:15] <dcausse>	 ejoseph: sure we can for a bit
[11:21:16] <dcausse>	 ejoseph: meet.google.com/zoy-hguj-nqe
[11:34:57] <taavi>	 whoever has local uncommitted changes on deployment-puppetmaster04 /var/lib/git/labs/private, please commit or undo your changes so that the auto-updating script gets unstuck. thanks!
[13:32:06] <gehel>	 Errand, back in a fee
[13:52:19] <gehel>	 taavi: super sorry. I've resetted the repo
[13:52:43] <gehel>	 I should have better explained how this all works
[15:11:36] <inflatador>	 taavi that's on me, I got sidetracked and forgot to add that
[15:11:56] <inflatador>	 also greetings
[15:17:40] <gehel>	 inflatador: ping me if you have time for a few pointers on working with puppet
[15:24:51] <inflatador>	 gehel sure, will hit you up shortly
[15:27:21] <gehel>	 inflatador: when you're ready: meet.google.com/ody-rxur-xgy
[15:27:32] <gehel>	 and for all: did we decide on a game for today?
[15:30:22] <inflatador>	 joining now
[15:30:57] <dcausse>	 o/
[16:00:26] <ebernhardson>	 \o
[16:02:28] <dcausse>	 o/
[16:05:03] <ebernhardson>	 dcausse: one problem i'm having, we pass all oauth params to jetty over the command line, and usernames can have spaces and other such things which can go through bash scripts, but seems far from ideal. Is there a better pattern for providing all these properties than the bash script?
[16:05:42] <dcausse>	 this is the oauth subproject?
[16:05:45] <ebernhardson>	 (from runBlazegraph.sh)
[16:06:06] <ebernhardson>	 dcausse: it's i suppose in general about how jvm properties get into the blazegraph/oauth/jetty jvm
[16:06:43] <dcausse>	 jvm props have to be on the command line I fear
[16:07:46] <ebernhardson>	 dcausse: err, not like -Xmx, but -Dorg.wikidata.query.rdf.mwoauth.OAuxyProxyConfig.bannedUsernamesCsv=EBernhardson (WMF),John & Doe
[16:08:12] <ebernhardson>	 that can go through bash, but seems like it's asking for accidents
[16:08:23] <dcausse>	 yes
[16:09:06] <dcausse>	 I don't see a way to do that cleanly by keeping java system prop
[16:09:26] <ebernhardson>	 hmm, so it would be about bringing in some other parser for oauth config? hmm
[16:09:35] <ebernhardson>	 also seems unideal :)
[16:09:40] <ebernhardson>	 will have to ponder i suppose
[16:10:18] <dcausse>	 ideally you want to point java at a csv file to read
[16:10:42] <ebernhardson>	 I suppose that would be an easy enough fix for this part, can do that in a few lines. will do
[16:11:09] <dcausse>	 could even (if we need to) be reloaded regularly
[16:11:56] <dcausse>	 there's System.getProperties().load(new FileInputStream(System.getProperty("prop.file"))); but that seems ugly
[16:15:36] <ebernhardson>	 i'm a bit surprised there isn't a command line option that runs that when booting the jvm.  I guess i'm thinking of java properties a bit like php .ini files but it's not really the same
[16:15:56] <ebernhardson>	 (since the .ini is program and not engine specific)
[16:17:09] <dcausse>	 yes... not sure why they did not included this but still allowed that to be globally set at runtime
[16:18:14] <dcausse>	 btw I made the 10sec retry on 404 for the flink pipeline configurable, strangely it seems that on commons 10secs is not enough
[16:18:45] <dcausse>	 we still get 20ish events failing/day there and 0 on wikidata
[16:18:46] <ebernhardson>	 hmm, will that effect WCQS SLO?
[16:18:54] <dcausse>	 no
[16:19:06] <ebernhardson>	 ok, so just a few events here and there need the extra time
[16:19:10] <dcausse>	 yes
[16:20:08] <ebernhardson>	 curious. I would have expected wikidata to need omre time than commons if anything, wonder why. But i guess if the longer timeout works good enough :)
[16:22:22] <dcausse>	 yes curious too, rate of edits is supposedly a lot higher on wikidata
[16:22:56] <dcausse>	 looking at recent data they seem equal 5ish/sec (which is low for wikidata)(
[17:55:37] <inflatador>	 and back
[19:16:01] <inflatador>	 lunch, back in ~30-45
[19:53:48] <inflatador>	 back
[20:47:03] <ebernhardson>	 lunch
[21:52:00] <ebernhardson>	 back
[22:16:39] * ebernhardson looks into mjolnir..and realizes of course the inputs still come from oozie. Reasonable chance something juts fell out of sync
[22:41:56] <ebernhardson>	 86k Special:Search queries for one hour, better than 0.  Kicked off hourly jobs for nov 10 through now to backfill