[07:53:32] <gehel>	 ejoseph: I have to be at Oscar's school at 11am. Ping me when you arrive if you want to have our meeting earlier, otherwise I'll move it to this afternoon.
[08:17:39] <addshore>	 Hi all, wanted to let you know that my arch docs / diagrams have had some more review on our side now and I'd love another glance by one of you! https://c2d26c0110--wikidata-wikibase-architecture.netlify.app/systems/query/01-introduction
[08:18:19] <addshore>	 One thing that came up would be that it would be great to include a runtime view or 2 of the new streaming updater, but I dont feel like I could draw that right now
[08:26:48] <zpapierski>	 addshore: I like the changes, thanks!
[08:26:58] <zpapierski>	 how much detailed you want that view or 2?
[08:32:30] <addshore>	 hmm, so I I think I'd like to understand the difference between theupdaters, and what it does differently
[08:32:51] <addshore>	 ie, gets things from a stream, compares that to stuff it already knows somehow? outputs something to a stram
[08:33:03] <addshore>	 the middle bit I guess is the most interesting and the bit we dont know about
[08:33:18] <addshore>	 I had a quick glance through the code repos to see fi I could create it myself but didnt know where to start
[08:34:18] <dcausse>	 addshore: it's heavily async and a bit hard to describe as a runtime view/sequence diagram
[08:35:04] <dcausse>	 what I came up with originally is https://people.wikimedia.org/~dcausse/updater_v2r2.png 
[08:35:12] <addshore>	 oooooooooo
[08:35:46] <addshore>	 Is that still mostly accurate?
[08:35:54] <dcausse>	 addshore: yes
[08:35:59] <addshore>	 neat
[08:36:15] <addshore>	 I think maybe then that could live quire resonable as another building block breakdown view :)
[08:36:31] <addshore>	 *quite reasonably (needs another coffee)
[08:36:59] <addshore>	 Mind if I recreate that in draw.io?
[08:37:31] <dcausse>	 addshore: no please do, I have the dia source if you need
[08:37:45] <addshore>	 oooh yeah, lets see fi I can do some magic conversation etc :)
[08:37:54] <dcausse>	 ok :)
[08:38:08] <addshore>	 it also makes use of swift, nice, didnt know that!
[08:38:30] <dcausse>	 yes that's the cold storage for the flink state
[08:38:36] <addshore>	 what exactly does "rdf import" represent?
[08:38:48] <dcausse>	 the "consumer" part in your diagram
[08:38:53] <addshore>	 otcha
[08:38:55] <addshore>	 *gotcha
[08:39:18] <addshore>	 I'll probably rename some of these parts then inline with other parts of these docs so far
[08:39:36] <addshore>	 but obviously we can continue working on terminology if folks want etc
[08:41:16] <dcausse>	 addshore: please name things the way you want if it makes them clearer with the rest of the doc
[08:41:19] <dcausse>	 https://people.wikimedia.org/~dcausse/updater_v2.dia
[08:41:23] <addshore>	 thanks!
[09:45:19] <gehel>	 I have to go to Oscar's school. Back later
[09:48:36] <ejoseph>	 zpapierski: let me know when you are available
[09:48:54] <zpapierski>	 I'm available now, at least for the next 25min
[09:48:59] <ejoseph>	 ok cool
[09:49:07] <ejoseph>	 code with me?
[09:49:49] <ejoseph>	 and guys i might close early today i have a train to catch
[09:50:03] <zpapierski>	 yep, hit me with a link
[09:50:10] <zpapierski>	 a, wait
[09:50:26] <zpapierski>	 I haven't checked why it's broken for me now, lets go with zoom
[09:50:32] <ejoseph>	 I have a meeting with gehel in 10 minutes
[09:50:40] <zpapierski>	 ah, ok
[09:50:50] <zpapierski>	 really - didn't he just go to Oscar's school?
[09:51:00] <ejoseph>	 Oh
[09:51:12] <ejoseph>	 Let's start then
[12:33:28] <zpapierski>	 lunch
[13:01:17] <gehel>	 ejoseph: I'm back, ping me when available
[13:18:26] <ryankemper>	 forgot to mention in standup but I'm out on pto today
[13:18:41] <ryankemper>	 (will be pretty much off the grid till sunday)
[13:27:03] <gehel>	 ryankemper: enjoy!
[13:51:14] <ejoseph>	 dcausse: elastic has switched to 2.7.1 I wonder if we should do the same
[13:51:35] <ejoseph>	 mind giving a little more context ?
[13:51:52] <dcausse>	 ejoseph: this is about keeping dependencies up-to-date
[13:52:22] <dcausse>	 if elastic has switched from randomized.testing 2.7.0 to 2.7.1 then might want to do the same
[13:52:44] <ejoseph>	 ok I see
[13:53:51] <dcausse>	 while it seems to work for us, it's generally good to keep deps up-to-date so that you don't fall too far behind and discover few incompatibilities sooner than having huge set of problems at once
[13:54:14] <ejoseph>	 gehel: are you available 
[13:55:40] <gehel>	 ejoseph: https://meet.google.com/vpn-uqkz-obr
[13:56:26] <gehel>	 (yes, I'm available)
[14:14:17] <gehel>	 ejoseph: I see that you have created a string of CR (737026 is based on 737024). This is great and exactly what you should be doing! Addressing the comments on the parent CR will require rebasing the child CR.
[14:14:44] <gehel>	 I don't know how familiar you are with git, but this might get messy. Feel free to reach out to me (or zpapierski) if things look strange!
[14:17:04] <gehel>	 ejoseph: I had a look at those 2 CRs. It all looks good (except the comments that dcausse already mentioned). 
[14:17:15] <gehel>	 Congratulations on your first Java CR!
[14:19:52] <dcausse>	 ejoseph: I always use git rebase -i (interactive rebase) see https://git-scm.com/book/en/v2/Git-Tools-Rewriting-History
[14:20:48] <dcausse>	 the most important thing is to keep the Change-Id in the commit message intact so that gerrit can keep track of what you do
[14:24:28] <zpapierski>	 ejoseph: I'm back around if you need me
[15:02:22] <ebernhardson>	 \o
[15:03:36] <gehel>	 ejoseph: o/
[15:06:38] <ebernhardson>	 and don't `git commit --amend` when fixing a rebase conflict from `git rebase -i` :) Lose so much work that way...
[15:13:28] <dcausse>	 o
[15:13:32] <dcausse>	  /
[15:14:09] <dcausse>	 going out early, have a nice week-end!
[15:14:27] <gehel>	 dcausse: enjoy!
[15:14:39] <gehel>	 I'll be out a bit earlier as well, I'll miss the gaming unmeeting
[15:36:51] <zpapierski>	 I think we actually haven't set any  game this time ;)
[15:46:51] <ejoseph>	 zpapierski: are you available
[15:47:16] <zpapierski>	 yep
[15:47:55] <zpapierski>	 need help?
[15:57:21] <zpapierski>	 ok, need to hunt down some food before unmeeting
[16:36:04] <gehel>	 time to start the weekend! have fun!
[17:17:13] <zpapierski>	 same here, have fun y'all 
[17:17:58] <zpapierski>	 (in my head, y'all always sounds like spoken with Texas accent, don't know why)
[18:12:43] <Trey314159>	 FYI, it is the "pioneer anomaly" not the "voyager anomaly" that I was trying to recall during the unmeeting: https://en.wikipedia.org/wiki/Pioneer_anomaly —TL;DR: a small amount of heat loss created a miniscule deceleration that adds up over years and bajillions of miles.
[18:26:49] <ebernhardson>	 i wonder if it will matter...reading the streaming updater bootstrap process we need a revisions file that comes from entity_revision_map, but afaict we only build the revision map for wikidata. Might be as simple as running it
[19:17:59] <RhinosF1>	 Hi, someone just reported search is broken on CheckUser wiki in #wikimedia-tech
[19:18:11] <RhinosF1>	 19:15:36 <roy649_> I don't know if this is significant, but I'm getting "An error has occurred while searching: We could not complete your search due to a temporary problem. Please try again later." on some searches on https://checkuser.wikimedia.org/
[19:18:32] <RhinosF1>	 Could you have a look
[19:18:36] <RhinosF1>	 ebernhardson: ^
[19:22:16] <tn>	 (I can see some `index_not_found_exception: no such index` dramah for those searches, but can't replicate)
[19:22:59] <RhinosF1>	 It's apparently intermittent
[19:23:19] <RhinosF1>	 I wonder if it's broken partly and needs a rebuild
[19:35:14] <ebernhardson>	 hmm
[19:35:36] <ebernhardson>	 intermittent is odd, they only talk to a single cluster the index should exist or not
[19:35:42] <ebernhardson>	 reads at least
[19:40:47] <ebernhardson>	 there is some suspicious timeouts around the pool counter, starting ~7 hours age we've been putting out 2-300 pool counter timeouts per 10 minutes
[19:41:30] <ebernhardson>	 (that's really low compared to the overall traffic, and doesn't seem to be wiki-specific though)
[19:42:29] <tn>	 fwiw, it seems to be 1 in ~10 searches that error.. 
[19:45:50] <ebernhardson>	 looks like i have a few logged requests with that error, will try re-running a few times
[19:58:49] <ebernhardson>	 well, good news i can reproduce an intermittent problem. But bad news i have no clue why yet :P Will look
[20:17:26] <ebernhardson>	 something seems off in the cross-cluster communication between 9643 and the other clusters but it's not clear what. I can reproduce every time against elastic1044:9643, and it seems to work every time against elsatic1045:9643.  But there is no way for these settings to differe between instances they are part of the cluster state
[20:33:51] <bd808>	 ebernhardson: is there envoy in the mix there? Like is envoy locally on elastic1044 to bridge to the rest of the cluster with TLS?
[20:34:49] <ebernhardson>	 bd808: envoy only from mediawiki to the cluster, but these inter-cluster commms go over elastic's inter-node port and skip  envoy and such
[20:35:16] <bd808>	 *nod*
[20:35:34] <ebernhardson>	 i'm restarting psi cluster on 1043 (was reproducing error) to see if a fresh instance makes any difference
[20:35:43] <ebernhardson>	 and that does fix that instance...hmm
[20:36:19] <ebernhardson>	 ryankemper: while its a sad answer, feel like rolling a restart across psi? I dunno if we should invest much in this if we are rolling 6.8 in a month and then 7.10 a few after
[20:37:52] <ebernhardson>	 strictly guessing, each instance is maintaining it's own view on cross-cluster state and some of them have fallen out. Would like to know why, but not sure where to look
[20:59:04] <ebernhardson>	 logging is curious. There have been a low level of logging that looks like same issue, ~400 events a day since 9-14.  9-14 is when we switched primary traffic from codfw to eqiad, so the problem would have been introduced to the eqiad cluster some time prior to that, but no clear point when
[21:01:33] <RhinosF1>	 That's fun
[21:01:43] <RhinosF1>	 Like 6 weeks ago
[21:05:17] <ebernhardson>	 yea, not going to be finding great details on this one
[21:52:30] <ebernhardson>	 i've restarted instances i identified as failing in the psi cluster,  dunno if it means anything but only instances in 1030-1050 had issues, 1050-1067 were all fine
[21:55:26] <ebernhardson>	 curriously i can't reproduce the issue in any other direction,  each cluster has two remote clusters configured but only psi->chi is reproducing in my setup