[07:25:29] <dcausse>	 I started to download the dumps on wdqs1009 & wdqs2008
[07:26:14] <dcausse>	 I'll be bootstrapping flink with this new state
[07:28:30] <dcausse>	 regarding the data-reload cookbook I think we'll be setting offsets manually for these two machines this time
[07:42:38] <zpapierski>	 sure
[07:47:24] <zpapierski>	 dcausse: for offset transfer, I know that I need to add that to the data-transfer cookbook
[07:47:41] <zpapierski>	 for manual offset set based on timestamp, that's a new cookbook?
[07:48:01] <zpapierski>	 or data_reload?
[07:48:21] <dcausse>	 manual I meant run existing code we have 
[07:48:38] <dcausse>	 ideally it would have been part of the data-reload cookbook
[07:49:26] <dcausse>	 I think I'll copy/paste some of the spicerack module you wrote in a python script and do that manually from e.g. stat1004
[07:50:12] <zpapierski>	 I was about to suggest that :)
[07:50:54] <dcausse>	 but yes the next cookbook it'll be useful is data-transfer
[07:51:20] <dcausse>	 actually without your new module it'll be huge pain
[07:51:53] <zpapierski>	 I hacked some version of it
[07:53:58] <zpapierski>	 it's /home/zpapierski/kafka_cluster.py and requires /home/zpapierski/config.yaml
[07:54:21] <zpapierski>	 it doesn't take argument's, I just added a function call at the end
[07:55:33] <dcausse>	 cool, thanks!
[07:55:50] <zpapierski>	 you're going to use a manual timestamp set up - right?
[07:56:06] <zpapierski>	 I have tests for it, but I haven't tried that one manually
[07:56:42] <zpapierski>	 should be fine, it's just a part of the process that I did try manually
[07:58:25] <dcausse>	 yes I'll need this so this a good test
[07:58:54] <dcausse>	 both for the flink consumers and the blazegraph consumers
[08:00:20] <zpapierski>	 hmm, I wish I knew more about those cookbooks earlier, I'd cannibalize data-reload one for wcqs data reload
[08:01:58] <dcausse>	 yes but cookbooks have constraints, I doubt they can be used in wmcs and still require root
[08:02:56] <dcausse>	 but with wcqs moving to prod we'll definitely need new (or adapt) ones
[08:03:38] <zpapierski>	 by cannibalizing I meant copy all the logic there, extracting it from spicerack/cookbooks, but good point on the update
[08:04:35] <zpapierski>	 btw - is there a way in cookbooks that's there already to understand which updater is running? To know if we need to do offset transfer during data-transfer?
[08:05:16] <dcausse>	 hm.. maybe?
[08:05:56] <dcausse>	 well for the "transition" the target machine will still be running the old one
[08:06:04] <dcausse>	 so we might want to force that
[08:06:54] <zpapierski>	 target, yeah - but the source is what I'm interested here
[08:06:59] <dcausse>	 but we could assume that if the source is running the new one it means we want to copy offsets
[08:08:00] <dcausse>	 if the cookbook can inspect the puppetdb they it should be able to know
[08:09:50] <dcausse>	 zpapierski: if you have time and are interested in testing your module you could set offsets for the flink consumers?
[08:10:27] <zpapierski>	 sure thing, I'd need timestamps for topics, though
[08:10:48] <dcausse>	 yes
[08:10:51] <zpapierski>	 I'll also hack the newest version (this one is before the refactor
[08:13:38] <dcausse>	 P17389
[08:13:54] <dcausse>	 https://phabricator.wikimedia.org/P17389
[08:14:53] <zpapierski>	 ok, I'll take care of it
[08:15:54] <dcausse>	 thanks!
[08:37:42] <zpapierski>	 ok, dry run seems ok
[08:37:43] <zpapierski>	 I hope
[08:39:02] <zpapierski>	 huh, actually it works for eqiad and fails for codfw
[08:39:11] <zpapierski>	 weirdly I might add
[08:42:01] <zpapierski>	 hmm, offsets_for_times doesn' not return None on failure to find a message, it will return {topic_partition: None}
[08:42:58] <zpapierski>	 dcausse: is it possible that there are no messages on codfw topics?
[08:50:13] <zpapierski>	 no, there are messages
[08:50:30] <zpapierski>	 I'm not sure when will offsets_for_times return None
[08:55:47] <zpapierski>	 ah, greater or equal - codfw hasn't received any new messages probably long before that
[08:55:55] <zpapierski>	 I wonder what should we do in that situation
[08:56:14] <zpapierski>	 anyway - I can set the offset for eqiad
[08:59:19] <zpapierski>	 and it's done
[08:59:25] <zpapierski>	 I think :)
[09:11:48] <zpapierski>	 break
[09:12:01] <dcausse>	 zpapierski: sorry got distracted
[09:45:35] <dcausse>	 zpapierski: something's weird with the offsets
[09:46:33] <zpapierski>	 let me guess - off times 1000?
[09:47:15] <dcausse>	 commited offsets for wdqs_streaming_updater is 2349952093 for kafka-main@eqiad in eqiad.mediawiki.revision-create and this is a message from 2021-10-01T07:53:17Z
[09:47:20] <dcausse>	 ah perhaps checking
[09:47:52] <zpapierski>	 I used miliseconds timestamps, perhaps those have a resolution of seconds?
[09:48:30] <dcausse>	 offset_for_times returns 2341301388 for me
[09:48:52] <dcausse>	 for timestamp 2021-09-24T23:00:01
[09:49:11] <zpapierski>	 https://www.irccloud.com/pastebin/8YoYYGga/
[09:49:17] <dcausse>	 2021-10-01T07:53:17Z seems like today the time I stopped flink
[09:49:39] <zpapierski>	 seems ok?
[09:49:53] <zpapierski>	 ah, you say they weren't commited
[09:50:49] <dcausse>	 zpapierski: this paste if for kafka-main@eqiad and eqiad.mediawiki.revision-create ?
[09:51:19] <zpapierski>	 yes on the cluster and site, but it's a listing for all topics
[09:51:50] <dcausse>	 2341299974 seems correct it's a bit earlier apparently (2021-09-24T22:58:00Z) but probably on purpose
[09:52:03] <zpapierski>	 yeah
[09:52:08] <zpapierski>	 it's because if DELTA
[09:52:27] <dcausse>	 ah yes I remember
[09:52:29] <zpapierski>	 so apparently it didn't commit
[09:52:39] <dcausse>	 yes seems like it did not
[09:52:49] <zpapierski>	 I don't understand, auto commit is on
[09:53:07] <dcausse>	 auto commit is perhaps only when consuming?
[09:53:23] <zpapierski>	 not to read up on it
[09:53:55] <dcausse>	 in previous python scripts I explicitly called commit after calling seek
[09:54:13] <zpapierski>	 you might be right, from what I read
[09:54:24] <zpapierski>	 let me hack it real quick
[09:57:33] <gehel>	 Lunch
[09:57:57] <zpapierski>	 dcausse: how about now?
[09:58:00] <dcausse>	 looking
[10:00:44] <dcausse>	 zpapierski: sounds good now, thanks!
[10:00:55] <dcausse>	 zpapierski: did you run it on codfw brokers?
[10:00:57] <zpapierski>	 awesome, it turns out we don't need seek at all
[10:01:00] <zpapierski>	 I did, it failed
[10:01:13] <zpapierski>	 no offsets were found
[10:02:18] <dcausse>	 zpapierski: I mean eqiad.* topic on the codfw kafka brokers
[10:02:35] <zpapierski>	 oh, I can't do that
[10:02:49] <zpapierski>	 I mean I can, I can hack the script
[10:03:02] <dcausse>	 ah ok I see
[10:03:17] <zpapierski>	 but will this be a part of normal operations? The current spicerack module assumes a correlation between site and prefix
[10:03:56] <dcausse>	 not needed for the streaming-updater-consumer offsets indeed 
[10:04:26] <dcausse>	 but for flink running in codfw it needs to read (eqiad|codfw).* topics
[10:04:38] <dcausse>	 but we did not plan to have a cookbook for that yet
[10:04:47] <dcausse>	 it's only needed for bootstrapping
[10:05:03] <dcausse>	 and there might be better ways to do that within flink actuallly
[10:05:06] <zpapierski>	 in any case, I can hardcode and run it
[10:05:20] <dcausse>	 zpapierski: if you can that'd be awesome
[10:06:59] <zpapierski>	 done, I think - can you verify?
[10:09:07] <dcausse>	 sure
[10:10:39] <dcausse>	 zpapierski: sounds perfect thanks!
[10:11:00] <zpapierski>	 yw :)
[10:11:06] <dcausse>	 going to start the pipeline and the backfill
[10:11:26] <zpapierski>	 it's on :)
[10:11:57] <zpapierski>	 1.5 years of our work finally going to production, albeit a bit slowly :)
[10:14:03] <dcausse>	 :)
[10:26:34] <dcausse>	 done
[10:26:37] <dcausse>	 lunch
[10:39:49] <zpapierski>	 backfill is done? that was fast
[10:58:03] <zpapierski>	 ok, now really a break
[11:13:29] <zpapierski>	 5
[12:05:11] <Amir1>	 dcausse: hi, I don't know if you're aware but your jobs seem to be failing at a large scale rate https://phabricator.wikimedia.org/T292048#7394439
[12:14:17] <dcausse>	 Amir1: no? looking
[12:15:01] <Amir1>	 https://phabricator.wikimedia.org/T292048#7394443 it's creating half a million failed job per hour
[12:15:38] <dcausse>	 ouch
[12:16:47] <dcausse>	 thanks for the ping
[12:16:52] <Amir1>	 somehow it's at the same time we deployed something for wikidata but the timing doesn't match
[12:17:13] <Amir1>	 I mean the timing matches but it doesn't make sense 
[12:17:42] <Amir1>	 let me know if I can help on anything
[12:18:20] <dcausse>	 sure thanks!
[12:20:11] <dcausse>	 seems to be cloudelastic not prod, probably why nothing else screemed louder
[12:21:52] <Amir1>	 it increased the rate of logstash intake by 50%, that triggered an alarm :D
[12:22:27] <dcausse>	 ah ok :)
[13:02:35] <dcausse>	 gehel: would have some time to launch a cookbook?
[13:02:58] <gehel>	 sure
[13:03:01] <gehel>	 what do you need
[13:03:36] <dcausse>	 it's running the data-reload cookbook with some special options on wdqs2008 and wdqs1009
[13:04:13] <gehel>	 dcausse: sure, do you have the options?
[13:05:23] <dcausse>	 gehel: I don't know the usual options but the special ones I'd need are: --reuse-downloaded-dump --skolemize
[13:05:47] <dcausse>	 and perhaps --reload-data wikidata as we don't need to reload categories
[13:07:12] <gehel>	 do we have a phab task?
[13:07:23] <dcausse>	 yes: T288231
[13:07:24] <stashbot>	 T288231: Deploy the wdqs streaming updater to production - https://phabricator.wikimedia.org/T288231
[13:07:40] <gehel>	 sudo cookbook sre.wdqs.data-reload --reason "data reload for new streaming updater" --reuse-downloaded-dump --skolemize --reload-data wikidata --task-id T288231 wdqs1009.eqiad.wmnet
[13:08:15] <dcausse>	 no depool ?
[13:08:28] <gehel>	 not for wdqs1009, it's a test server, not behind LVS
[13:08:37] <gehel>	 I'll add --depool for 2008
[13:09:07] <dcausse>	 ok
[13:13:26] <gehel>	 dcausse: started for both
[13:13:33] <dcausse>	 gehel: thanks!
[13:14:07] <gehel>	 do you have an estimate of how long this will take?
[13:14:31] <dcausse>	 the usual time ~10days if we are lucky
[13:15:14] <dcausse>	 if you still have some time this one has to go as well https://gerrit.wikimedia.org/r/c/operations/puppet/+/721281
[13:16:11] <gehel>	 better doing it right now than forgetting to do it before the reload is completed!
[13:17:21] <gehel>	 done
[13:17:22] <dcausse>	 thanks!
[14:50:43] <gehel>	 time to go learn some signs!
[14:50:48] <gehel>	 Enjoy the weekend!
[14:51:52] <dcausse>	 enjoy!
[15:15:41] <ebernhardson>	 \o
[15:19:37] <dcausse>	 o/
[15:21:43] <ebernhardson>	 zpapierski: i was curious after looking over the mw-oauth-proxy code, what do we need to do to tie the session stores and caches together? We should expect that the first request to get an oauth token will hit one server, and the next request with the validation token will hit some other server
[15:21:56] <zpapierski>	 o/
[15:22:49] <zpapierski>	 hmm
[15:23:05] <zpapierski>	 this sounds like a need for a sticky session or distributed cache
[15:23:22] <ebernhardson>	 well, yes :) But that should be super generic web stuff we just plug something in?
[15:23:41] <zpapierski>	 I'd think so
[15:23:54] <ebernhardson>	 i guess, i was hoping you would know what :P I know nothing of java webdev
[15:24:16] <zpapierski>	 not really, it isn't specific to Java, at least to my knowledge
[15:24:46] <ebernhardson>	 session store should be super specific to java ?
[15:24:56] <ebernhardson>	 unless we are talking about writing a new session store with a KV api
[15:24:56] <zpapierski>	 I'mI'm not sure I understand
[15:25:19] <zpapierski>	 oh, ok, now I do
[15:26:01] <zpapierski>	 if we do a sticky session with LVS, this will still be problematic because users will be tied to a single server and if they hit another, they will reauthenticate
[15:26:10] <ebernhardson>	 i mean i can google and choose something random, but it wont be an intelligent decision :P
[15:26:36] <zpapierski>	 I'm thinking JWT
[15:27:02] <zpapierski>	 we can solve the authentication part with sticky session for that part (I hope) and then JWT doesn't require a session store
[15:27:02] <ebernhardson>	 yea i don't think LVS sticky sessions are enough, ideally no hiccups when a server restarts or whatever
[15:27:48] <ebernhardson>	 hmm, ok i can see what can be done about swapping session store to JWT.  For the cache indeed anthing generic, i'll have to check if generic apps are allowed to use the cluster memcache/redis instances
[15:28:11] <zpapierski>	 you could, potentially, but I'd rather simplify the code
[15:28:16] <dcausse>	 we have https://www.mediawiki.org/wiki/Kask
[15:28:41] <ebernhardson>	 zpapierski: don't you still have to hold the requestToken in a cache regardless?
[15:29:03] <ebernhardson>	 zpapierski: i suppose i'd have to check the oauth spec, but you generate a token, give it to the user, then the user returns. I think we still need to know at that point we created the initial token and what it was?
[15:29:27] <zpapierski>	 you mean service <-> mediawiki auth?
[15:29:54] <ebernhardson>	 service should never talk to mw directly? app generates a token, user visits mediawiki, mediawiki redirects back to app with verification token
[15:30:08] <ebernhardson>	 in the current code at least you lookup that initial token in the cache
[15:30:14] <ebernhardson>	 so i'm assuming we have to know that token exists and we created it
[15:31:05] <ebernhardson>	 oh, does this actually talk to mw in the background? Haven't loked into what ServiceBuilder is doing. I don't think when i did oauth1 years ago there was any pingback
[15:32:06] <zpapierski>	 I need to confirm it, but I think it doesn't need to outside of the initial authentication
[15:33:25] <zpapierski>	 we could replace checkLogin (that checks if access token is present) with JWT verification
[15:33:48] <zpapierski>	 I'm not sure about requestTokens though yet
[17:54:42] <zpapierski>	 ebernhardson: anything I can help with regarding that oauth stuff?
[17:58:49] <zpapierski>	 also, there's a ticket for that JWT stuff - T290299
[17:58:50] <stashbot>	 T290299: Replace token store in MW OAuth WCQS proxy with JWT  - https://phabricator.wikimedia.org/T290299
[18:02:50] <ebernhardson>	 zpapierski: hmm, not sure how much. Checking the oauth protocol we do need to keep the initial requestToken somewhere. In discernatron that's using php's session mechanism
[18:03:23] <zpapierski>	 do we need to replicate it, or is it fine that each service has its own?
[18:03:44] <ebernhardson>	 has to be replicated, i don't think we can expect two requests from a user to always go to the same host
[18:04:03] <ebernhardson>	 we can set it up to sometimes do that, but it would still fail sometimes
[18:04:07] <zpapierski>	 what if we sticky up auth calls?
[18:04:42] <ebernhardson>	 sticky is a best case, it still fails on server restart, pool/depool, etc.
[18:04:49] <zpapierski>	 cost of infrequent failure doesn't seem terrible - you'll restart the authentication session 
[18:05:11] <zpapierski>	 and this is mostly invisible to user, at least web one
[18:05:35] <zpapierski>	 and on restart you loose the session we have now anyway, so it's not worse than it is now
[18:06:24] <ebernhardson>	 hmm, will it start another round of redirects or just give them a forbidden?
[18:06:50] <zpapierski>	 I think the former
[18:07:08] <zpapierski>	 it's the same as user session isn't present
[18:07:24] <zpapierski>	 (at least it's how it is right now)
[18:07:41] <ebernhardson>	 hmm, reading it /oauth_veify returns forbidden if the token isn't in the cache
[18:07:49] <zpapierski>	 yep
[18:09:16] <zpapierski>	 this forces checkLogin, which starts the whole redirect dance
[18:09:45] <zpapierski>	 since you're already logged in and have a cookie, it doesn't even show you a login page
[18:10:22] <ebernhardson>	 hmm, ok then if it just sends them around in redirects i suppose thats not so bad
[18:11:05] <ebernhardson>	 so then all we need is a jwt token signed with an expiration date of the sessions. And then i guess something to refresh that occasionally?
[18:11:36] <zpapierski>	 it should refresh itself automatically, if we implement this the same way - just another redirect loop when expired
[18:12:01] <ebernhardson>	 it means we have no way to expire sessions forcefully, do we care?
[18:12:26] <ebernhardson>	 probably not?
[18:13:02] <zpapierski>	 not really, I guess
[18:13:14] <zpapierski>	 I mean, if the need be, we can block users
[18:13:34] <zpapierski>	 but apart from that, it's all or nothing
[18:15:31] <ebernhardson>	 so remaining question, how do we sticky auth sessions?
[18:16:18] <ebernhardson>	 or do we have to sticky all incoming request routing?
[18:16:29] <zpapierski>	 this I don't know :(
[18:17:09] <zpapierski>	 SRE's probably should though
[18:17:43] <ebernhardson>	 making all requests sticky seems problematic, easily get unbalanced loading.  But reading the nginx config /check_login subrequest can be called from basically any incoming request
[18:19:07] <zpapierski>	 does it need to be sticky though?
[18:19:28] <zpapierski>	 if you check_login with JWT, any instance can authenticate
[18:19:33] <ebernhardson>	 check_login creates the token, the returning oauth_verify req has to go to whatever called check_login
[18:19:57] <zpapierski>	 ah, sorry I was thinking about check_auth
[18:21:00] <ebernhardson>	 probably easier to write a kask wrapper and avoid changing much of this, just replace the two Cache's with shared remote cache's
[18:21:22] <zpapierski>	 which will be called on each call? how performant it is?
[18:21:37] <ebernhardson>	 it's a small api sitting in front of dedicated cassandra cluster
[18:21:48] <ebernhardson>	 i dunno what the latency expected would be
[18:22:04] <ebernhardson>	 i suppose envoy is proxying it for mw, there ought to be some stats in prometheus somewhere
[18:22:19] <zpapierski>	 are you sure about check_login, though?
[18:22:28] <zpapierski>	 check_app is actually the one being called each time
[18:22:50] <zpapierski>	 check_login is only called on 403
[18:23:15] <ebernhardson>	 Anything that fails auth will 403, nginx is configured to subrequest that, so any incoming request that fails auth (all first time requests) go to check_login
[18:23:32] <zpapierski>	 any call can trigger it, but maybe we can set it up so that only 403 trigger sticky session?
[18:24:52] <zpapierski>	 also, we might just go for a very short sticky sessions - auth calls are very close together
[18:24:52] <ebernhardson>	 hmm, i guess need to ask sre. From looking arround i think lvs calls it persistence
[18:25:08] <zpapierski>	 true
[18:25:21] <zpapierski>	 and I think it can be configured for a time period
[18:25:28] <zpapierski>	 10s would be literally enough
[18:26:12] <zpapierski>	 sorry to push for this so much, I was really hoping to make this proxy simpler and less prone to failure, and I'm afraid of all I/O
[18:27:12] <zpapierski>	 hmm, also you can configure sticky session via headers apparently, so maybe we cn leverage nginx proxy_set_header for that
[18:27:56] <ebernhardson>	 via http headers? I don't think lvs sees those, it's all https
[18:28:35] <zpapierski>	 sorry then, random googling
[18:29:17] <zpapierski>	 in any case, short sticky session would probably be enough?
[18:31:31] <zpapierski>	 need to take care of the younger generation and probably go to sleep afterwards, I hope there is a solution within LVS realm for this