[07:48:08] <gehel>	 my bad, Special and Advanced are the same
[11:00:31] <gehel>	 lunch
[11:32:44] <dcausse>	 lunch 2
[13:37:54] <ejoseph>	 dcausse would you like to continue by 3:30
[13:44:47] <dcausse>	 ejoseph: sure
[14:37:23] <dcausse>	 ejoseph: are you around?
[14:37:36] <ejoseph>	 Yes
[14:37:44] <ejoseph>	 Same link?
[14:39:02] <dcausse>	 sure
[15:52:35] <ebernhardson>	 \o
[15:52:45] <gehel>	 ejoseph: I moved our 1:1 tomorrow, I'll be at the doctor with Oscar. Let me know if the new time doesn't work for you
[15:57:51] <dcausse>	 o/
[16:03:41] <mpham>	 fyi, slack is still down for me (and others)
[16:08:19] <ejoseph>	 gehel it's ok
[16:08:30] <gehel>	 thanks!
[17:46:41] <mpham>	 back on slack again, but having some not so fun stiff neck cramps, so i'm going to take a break for a bit
[17:47:45] <dcausse>	 gehel: I'm having trouble with flink@codfw and I'm not yet clear why it's failing to deploy a new version of the app, could you switch wdqs traffic to eqiad?
[17:48:29] <gehel>	 ryankemper: around?
[17:49:26] <gehel>	 dcausse: I'm on it
[17:49:30] <dcausse>	 gehel: thanks!
[17:49:48] * gehel just needs to remember how we switch traffic
[17:53:09] <gehel>	 dcausse: we should be good, I depooled both internal and public on codfw
[17:53:18] <dcausse>	 thanks, investigating
[17:53:31] <gehel>	 scream if you need help, I'll go back to dinner
[17:53:54] <gehel>	 dcausse: do I also need to depool wcqs?
[17:54:16] <dcausse>	 gehel: no the app is functionning for wcqs
[17:54:26] <gehel>	 ack
[17:54:34] <dcausse>	 that's why I'm a bit puzzled
[17:55:44] <dcausse>	 not sure I like the exceptions I'm seeing... org.apache.flink.contrib.streaming.state.restore.RocksDBFullRestoreOperation.restoreKVStateData with java.io.EOFException, this does not sound great :/
[17:55:56] <ebernhardson>	 :S
[17:56:21] <dcausse>	 will try a previous a savepoint perhaps...
[17:58:25] <gehel>	 added a note on how to depool a cluster: https://wikitech.wikimedia.org/wiki/Wikidata_Query_Service/Runbook#Remediation
[17:59:14] * gehel goes back to dinner
[18:09:29] <gehel>	 I'm getting alerts on wcqs2003
[18:09:59] <gehel>	 Updater is failing
[18:10:07] <gehel>	 dcausse: is this related ?
[18:10:21] <dcausse>	 looking
[18:11:58] <dcausse>	 hm.. might be related to the new version
[18:13:32] <gehel>	 dcausse: ping me if you need me
[18:13:40] <dcausse>	 sure
[18:13:59] * gehel is trying to eat a bit of pie before being pinged
[18:24:38] <dcausse>	 patch coming for the problem with wcqs
[18:36:56] <dcausse>	 ebernhardson: if you have a couple minutes https://gerrit.wikimedia.org/r/c/wikidata/query/rdf/+/764864
[18:45:12] <ebernhardson>	 sure
[18:47:24] <ebernhardson>	 seems simple enough, +2'd
[18:48:10] <dcausse>	 thanks!
[18:59:59] <tarrow>	 Hi search people! I came to say hi because I got pinged by a wikidata maxlag alert. It looks to me like this is because all the 2xxx queryservice servers are lagging for the last 1.5 hrs
[19:01:02] <ryankemper>	 gehel: back now
[19:01:14] <ryankemper>	 tarrow: thanks, taking a look now
[19:01:18] <tarrow>	 cheers!
[19:01:24] <dcausse>	 tarrow: we have an incident in progress but we should have routed traffic to eqiad which are not affected
[19:01:54] <dcausse>	 codfw is affected by the problem, do you source maxlag from there as well?
[19:02:20] <tarrow>	 I beleive so: https://phabricator.wikimedia.org/T238751
[19:03:15] <dcausse>	 tarrow: is there a way for you to only source eqiad metrics while the incident is being addressed?
[19:04:22] <tarrow>	 that is a great question; I have no idea! I'm actually not "on the wikidata team" and it's after berlin office hours so let me have a dig around
[19:04:59] <dcausse>	 thanks!
[19:08:26] <addshore>	 dcausse: ca we depool codfw?
[19:08:32] <addshore>	 oh wait, no that wouldnt work
[19:08:37] <gehel>	 already depooled
[19:08:38] <dcausse>	 addshore: we did depool
[19:08:47] <addshore>	 yeah, tarrow already found the ticket https://phabricator.wikimedia.org/T238751
[19:09:12] <tarrow>	 I open https://phabricator.wikimedia.org/T302330 for this
[19:10:06] <addshore>	 https://gerrit.wikimedia.org/g/operations/puppet/+/b81506884faa046e4103412f1928ece934a157dc/modules/profile/manifests/mediawiki/maintenance/wikidata.pp#15 
[19:10:09] <addshore>	 we can remove the codfw bit
[19:19:11] <tarrow>	 Can any of you deploy https://gerrit.wikimedia.org/r/764875 for us?
[19:20:08] <tarrow>	 dcausse: ^^
[19:20:25] <ebernhardson>	 ryankemper: ^^
[19:22:13] <tarrow>	 We're keen to see it fixed because wikidata users are currently unable to make bot edits
[19:23:13] <ryankemper>	 tarrow: ebernhardson: thanks, deploying
[19:24:02] <tarrow>	 cheers!
[19:29:38] <addshore>	 but hillariously at the same time the wdqs updater started working again too? :P
[19:29:54] <tarrow>	 Did you have a ticket for the original incident on codfw just so I can leave a nice papertrail for the dayshift?
[19:32:36] <ryankemper>	 tarrow: I don't think there's a ticket for the original incident yet. I'm working on a lightweight incident report that can be linked to as well
[19:33:13] <gehel>	 ryankemper: should we cancel our pairing session since Brian is out and you're busy on the incident report?
[19:33:30] <ryankemper>	 gehel: yeah, let's do that
[19:33:35] <gehel>	 ack
[19:33:42] <gehel>	 ping me if you need me
[19:33:52] <tarrow>	 coolio! no problem; thanks for all the help; have a nice rest of the evening/day :)
[19:34:13] <gehel>	 tarrow: gutenacht
[19:34:18] <dcausse>	 rollbacked a previous version on a previous checkpoint and it worked...
[19:34:28] <gehel>	 dcausse: \o/
[19:35:04] <dcausse>	 still need to understand why this failed...
[19:35:19] <dcausse>	 going to deploy the fix for wcqs
[19:37:35] <dcausse>	 ryankemper: oh cool you triggered a build on the rdf repo
[19:38:04] <ryankemper>	 dcausse: yup, should be finished in ~10 mins
[19:38:22] <ryankemper>	 dcausse: so I gather the failures are related to event time getting set to null? or is that not the entire problem
[19:38:24] <dcausse>	 kk that should fix the updater issue on the wcqs@codfw updaters
[19:38:42] <dcausse>	 ryankemper:  it's related but not the main issue
[19:38:50] <ryankemper>	 ack
[19:39:14] <dcausse>	 main issue is flink unable to restart from a savepoint when I deployed a new version of the job
[19:40:08] <dcausse>	 tried multiple things without much luck and finally rolled back a previous version of the app on a old "checkpoint" (not savepoint)
[19:40:16] <dcausse>	 the cause is still unclear
[19:40:45] <dcausse>	 the 2 savepoints I took before the upgrade did not work
[19:42:27] <ryankemper>	 interesting
[19:51:44] <ryankemper>	 dcausse: rdf build done, I merged https://gerrit.wikimedia.org/r/c/wikidata/query/deploy/+/764883
[19:52:12] <dcausse>	 thanks!
[19:52:15] <ryankemper>	 dcausse: so is this deploy just to fix the event time null issue, and the other issues have already been addressed with the old checkpoint?
[19:52:48] <dcausse>	 ryankemper: yes, the old checkpoint is just a mitigation as the root cause is yet unclear
[19:53:03] <ryankemper>	 ack, understood
[19:53:06] <ryankemper>	 dcausse: deploying wcqs now
[19:53:43] <dcausse>	 kk checking the logs to confirm
[20:00:43] <ryankemper>	 tarrow: I created https://phabricator.wikimedia.org/T302340 if you still need a ticket to tie things to
[20:00:50] <ryankemper>	 dcausse: deploy finished on wcqs fleet
[20:01:25] <dcausse>	 ryankemper: seeing v103 on the machines not 104
[20:02:09] <dcausse>	 on wdqs2003 at least
[20:02:14] <dcausse>	 err
[20:02:18] <dcausse>	 wcqs2003 I mean
[20:03:11] <ryankemper>	 dcausse: bizarre, I ran `scap deploy -v --environment wcqs 'Deploy 0.3.104 to WCQS'`, sshing into the hosts to verify rn
[20:03:56] <dcausse>	 ryankemper: perhaps you forgot to fetch/rebase on the deployment host?
[20:04:10] * ryankemper facepalms
[20:04:21] <ryankemper>	 dcausse: that's exactly it, thanks
[20:04:42] <dcausse>	 there are some repos that does this automatically, it's less error prone imo
[20:08:40] <ryankemper>	 Could pretty trivially wrap the logic in a helper script so we're just running a "deploy_latest_qs w[d,c]qs" type command
[20:09:30] <ryankemper>	 dcausse: okay, I see `0.3.104` running now
[20:09:43] <dcausse>	 thanks!
[20:10:59] <dcausse>	 fix seems ok according to wcqs2003
[20:12:20] <ryankemper>	 dcausse: were we seeing errors in the logs previously that resolved? or what are you looking at to confirm
[20:12:39] <dcausse>	 both grafana and the logs
[20:12:57] <dcausse>	 https://grafana-rw.wikimedia.org/d/fdU5Zx-Mk/wdqs-streaming-updater?orgId=1&var-site=codfw&var-k8sds=codfw%20prometheus%2Fk8s&var-opsds=codfw%20prometheus%2Fops&var-cluster_name=wcqs&from=now-1h&to=now
[20:13:12] <dcausse>	 wcqs2003 resumed consumption
[20:13:30] <dcausse>	 will force start the updater on other nodes
[20:20:51] <gehel>	 inflatador: (for tomorrow) what's the status on T297907 ? Is everything done (especially VictorOps) ?
[20:20:51] <stashbot>	 T297907: SRE Onboarding - Brian King, Search Platform team - https://phabricator.wikimedia.org/T297907
[20:23:52] <gehel>	 ebernhardson: isn't T295734 done? I remember you saying that we have 2 instances of Cindy running...
[20:23:55] <stashbot>	 T295734: Bring up two copies of the CirrusSearch browser integration env in cloud - https://phabricator.wikimedia.org/T295734
[20:24:27] <ebernhardson>	 gehel: the distinction was the goal was a 7.10 instance, but we didn't have 7.10 plugins ready so i setup the current and a 6.8.  I guess i was leaving it around to upgrade the 6.5 instance to 7.10 after we switch
[20:24:43] <gehel>	 ebernhardson: make sense, thanks
[20:24:58] <dcausse>	 looks like we're out of the woods, going to eat something, will check graphs in a while... if we can afford keeping wdqs@codfw depooled that'd be great so that I can investigate deeper tomorrow
[20:26:07] <ebernhardson>	 dcausse: kk, thanks for getting it going again
[20:26:17] <gehel>	 dcausse: thanks ! Get some rest!
[20:28:41] <ryankemper>	 ^ echoing the above, thanks david
[20:28:55] <ryankemper>	 Keeping wdqs@codfw depooled sounds reasonable to me
[23:41:28] <ebernhardson>	 hmm, first stab at seeing if geodata was slow finds that the p99 latency elastic<->cirrus is ~1s.  But the total time spent in php only comes in at 400ms.  Something about these two ways of calculating percentiles don't agree :S
[23:41:51] <ebernhardson>	 Err, i mean the p99 for the length of a php request that includes a geodata request comes in at ~400ms
[23:43:34] <ebernhardson>	 i do wonder a bit about those tail latencies,  but unclear what we can do.  The p95 for elastic<->cirrus is reported as 80ms which is plenty good