[02:16:42] <ryankemper>	 wrt the streaming updater: all of codfw internal is done, which leaves the rest of codfw public wdqs and all of eqiad left
[05:57:57] <zpapierski>	 awesome :)
[07:09:46] <zpapierski>	 gehel: we could trigger next transfers
[07:10:07] <zpapierski>	 I still need to finish my breakfast, but if you have time, we can sync up
[07:10:17] <zpapierski>	 with dcausse as well, once he's not away
[07:13:01] <gehel>	 dcausse: ping us when around!
[07:13:27] <gehel>	 it looks like T288231 has not been updated with completed transfers
[07:13:28] <stashbot>	 T288231: Deploy the wdqs streaming updater to production - https://phabricator.wikimedia.org/T288231
[07:15:05] <zpapierski>	 it has been now
[07:15:35] <gehel>	 you were faster than me!
[07:16:01] <zpapierski>	 :)
[07:16:25] <zpapierski>	 wdqs1009 isn't yet ready, so codfw is the only option
[07:17:04] <zpapierski>	 https://grafana.wikimedia.org/d/fdU5Zx-Mk/wdqs-streaming-updater?orgId=1&var-site=codfw&var-k8sds=codfw%20prometheus%2Fk8s&var-opsds=codfw%20prometheus%2Fops
[07:17:05] <gehel>	 and it's probably going to need 5 or 6 h to be ready
[07:17:12] <zpapierski>	 more or less, yeah
[07:17:47] <zpapierski>	 we estimated around 5PM UTC
[07:29:22] <dcausse>	 gehel: I'm around
[07:31:49] <dcausse>	 regarding data-transfer at a glance I agree with has been said, looks like codfw can continue, eqiad is still stuck on wdqs1009 catching up
[07:34:55] <gehel>	 I'll start the transfer from 2008 -> 2001
[07:35:20] <gehel>	 ryankemper: I'm reusing your existing tmux session so you can easily take over later today
[07:35:35] <zpapierski>	 I'm here as well
[07:35:54] <gehel>	 do we need to discuss anything else before starting this transfer?
[07:35:58] <zpapierski>	 yep
[07:36:12] <zpapierski>	 https://etherpad.wikimedia.org/p/streaming_updater_cutover
[07:36:40] <zpapierski>	 we switched order a bit, since doing the puppet during the cookbook run broke the process
[07:37:08] <zpapierski>	 the idea now is to stop the updater from starting, merge& apply the patch and start the cookbook
[07:37:26] <gehel>	 ok, will do
[07:37:32] <zpapierski>	 it means we'll stoping an update process for a pooled instance, but it's only for few seconds before depooling
[07:38:34] <gehel>	 rebasing https://gerrit.wikimedia.org/r/c/operations/puppet/+/730796
[07:40:46] <gehel>	 patch merged, running puppet-agent
[07:41:20] <gehel>	 cookbook started
[07:41:45] <zpapierski>	 ok, it should take about 6h, so we can do another one later on
[07:43:24] <zpapierski>	 we estimated that codfw should be done by Tuesday, later on we can fire up eqiad tranfer as well :)
[07:44:30] <dcausse>	 looking at previous run I think it takes 1h30, 1h15 for the transfer and 15min to catch-up
[07:44:43] <zpapierski>	 really? that's waay shorter than we though
[07:44:55] <zpapierski>	 in that case, we can finish codfw today
[07:45:09] <zpapierski>	 I'm guessing eqiad will take more time, though
[07:46:29] <dcausse>	 hard to tell, eqiad machines are older so perhaps
[07:46:43] <zpapierski>	 that and there are more of them
[07:46:52] <dcausse>	 ah yes sure
[08:05:51] <zpapierski>	 I need to say Gnome is growing super quickly on me, though it required some extensions first
[08:08:05] <dcausse>	 it's seamless I rarely feel I use a desktop env but might that I'm used to it
[08:08:35] <zpapierski>	 I suppose that's probably the best thing you can say about DE, it doesn't get in your way
[08:09:53] <zpapierski>	 anyway, with per screen scaling (especially fractional scaling) this is all finally comfortable again
[08:10:42] <dcausse>	 yes... these are the "hard features"
[09:06:15] <gehel>	 transfer completed for wdqs2001
[09:06:39] <zpapierski>	 but not catchup, right?
[09:07:21] <gehel>	 It does not look like it, but I thought that the cookbook would be waiting for catchup before repooling
[09:07:27] <zpapierski>	 yep, both are catchup up right now, according to graphs
[09:07:36] <zpapierski>	 so did I, it doesn't?
[09:08:17] <gehel>	 https://github.com/wikimedia/operations-cookbooks/blob/master/cookbooks/sre/wdqs/data-transfer.py#L183
[09:08:27] <dcausse>	 perhaps there's a tolerance
[09:08:43] <gehel>	 does not look catched up to me: https://grafana.wikimedia.org/d/000000489/wikidata-query-service?viewPanel=8&orgId=1&refresh=1m&from=now-24h&to=now&var-cluster_name=wdqs
[09:08:52] <zpapierski>	 it's not
[09:09:13] <zpapierski>	 ok, so the script is suppose to wait, from what I see
[09:09:16] <dcausse>	 it does not wait in fact
[09:09:19] <zpapierski>	 pool only happens after the wait
[09:09:25] <gehel>	 depooling 2001 until it catches up
[09:09:45] <gehel>	 and 2008 as well
[09:09:47] <zpapierski>	 yep
[09:09:57] <dcausse>	 it just throws an exception if the lag is too high to let the operator repool manually I guess
[09:10:26] <gehel>	 well, 2008 is already at lag < 10'
[09:10:41] <zpapierski>	 as it should be, process took 1.5h
[09:10:43] <dcausse>	 gehel: it's already back to normal not sure it's worthwhile to depool
[09:10:44] <gehel>	 and 2001 as well
[09:10:51] <gehel>	 damn, that's fast! repooling all that
[09:11:01] <zpapierski>	 whoa
[09:11:12] <zpapierski>	 nice work, well, us
[09:11:45] <gehel>	 yeah! that's pretty cool!
[09:12:14] <gehel>	 let's start 2008->2002
[09:12:24] <zpapierski>	 agreed
[09:14:26] <zpapierski>	 is there an easy way to force backend, or do I did to tunnel?
[09:15:03] <gehel>	 you need to tunnel
[09:15:48] <gehel>	 data transfer started for wdqs2008 -> wdqs2002
[09:15:56] <dcausse>	 thanks!
[09:33:31] <gehel>	 early lunch
[09:33:46] <gehel>	 ejoseph: I'll be back around 2pm CEST
[09:45:09] <zpapierski>	 taking a break as well
[10:33:57] <dcausse>	 lunch
[11:44:18] <gehel>	 wdqs2002 transfer completed
[11:44:40] <zpapierski>	 including catchup
[11:44:45] <zpapierski>	 we can start another one
[11:44:56] <gehel>	 time for 2003!
[11:45:43] <zpapierski>	 it turns out we will actually do the codfw transfer before eqiad :)
[11:45:49] <zpapierski>	 so much for plans
[12:27:36] <gehel>	 ejoseph: are you back? how's the laptop?
[12:33:10] <ejoseph>	 They changed it
[12:33:22] <ejoseph>	 I’m still on my way back home
[12:33:24] <gehel>	 cool! So brand new one!
[12:34:12] <zpapierski>	 with 32GB of RAM, which I think we all agree should be a bare minimum for any developer :P
[12:34:20] <zpapierski>	 (yes, I'm very subtle)
[12:46:06] <zpapierski>	 lunch break
[13:20:11] <gehel>	 transfer completed for 2002
[13:20:35] <gehel>	 s/2002/2003/
[13:21:18] <gehel>	 and wdqs1009 is catched up
[13:21:27] <dcausse>	 cool!
[13:22:06] <gehel>	 getting ready for 2004
[13:24:55] <gehel>	 dcausse, zpapierski: are you feeling brave enough to start a transfer in eqiad as well?
[13:25:27] <dcausse>	 gehel: starting the internal cluster should be low risk
[13:25:46] <dcausse>	 it's in this order in the patch chain anyways
[13:33:14] <gehel>	 starting transfer wdqs1009 -> wdqs1003
[13:33:23] <dcausse>	 thanks!
[13:33:57] <gehel>	 slightly annoying: 1009 isn't behind LVS, but 1003 is, that's a case that the cookbok does not support: depooling 1003 manually (and let's not forget to re-pool)
[13:34:48] <dcausse>	 ah we should use 1003 as a source for next ones probably
[13:43:37] <gehel>	 yeah, that would be simpler
[13:54:35] <zpapierski>	 too late to answer, but definitely brave enough
[13:54:59] <zpapierski>	 I see our estimate on wdqs1009 was a bit too conservative
[14:06:41] <zpapierski>	 gehel: I did this - https://gerrit.wikimedia.org/r/c/wikimedia/irc/ircservserv-config/+/731106 - what are the next steps for irc configuration?
[14:07:31] <gehel>	 done!
[14:07:34] <zpapierski>	 ok, very little steps then :)
[14:07:42] <zpapierski>	 how is it applied?
[14:07:56] <zpapierski>	 s/little/few
[14:08:11] <gehel>	 There might be a command to send to ircservserv-wm to get the config change applied
[14:09:38] <gehel>	 !isspull
[14:09:45] <zpapierski>	 ok, from what I see, only channel founders can do this
[14:09:48] <gehel>	 !issync
[14:09:49] <ircservserv-wm>	 Syncing #wikimedia-search (requested by gehel)
[14:09:50] <ircservserv-wm>	 No updates for #wikimedia-search
[14:10:06] <zpapierski>	 huh
[14:10:23] <zpapierski>	 no changes, maybe it takes ime
[14:10:24] <gehel>	 !isspull
[14:10:27] <zpapierski>	 s/ime/time
[14:11:06] <majavah>	 read the instructions please https://meta.wikimedia.org/wiki/IRC/Bots/ircservserv#Making_changes
[14:11:12] <zpapierski>	 this !ispull needs to be executed on ops channel
[14:12:18] * gehel is reading the instructions and still isn't super clear on what's missing
[14:13:03] <zpapierski>	 gehel: ispull goes into -ops channel
[14:13:18] <gehel>	 Oh, is that in the doc?
[14:13:21] <zpapierski>	 yep
[14:13:21] <majavah>	 and you would need to be on this array https://github.com/wikimedia/wikimedia-irc-ircservserv-config/blob/8c553ef393c093fe001464427ff4df6916cba010/config.toml#L9 to do it
[14:13:22] * gehel needs to learn to read
[14:13:22] <zpapierski>	 !isspull: Pull configuration updates that have been merged into Git. Should tell you which channels potentially need syncing. This command should be run in #wikimedia-ops.
[14:14:14] <gehel>	 Oh, ok, I misread that doc, I am only elligible to be trusted, but not actually trusted
[14:14:34] <gehel>	 majavah: any chance you could pull that config for us?
[14:14:39] <majavah>	 {{done}}
[14:14:43] <gehel>	 thanks!
[14:14:43] <zpapierski>	 ah, I misread that part as well
[14:14:50] <gehel>	 !issync
[14:14:51] <ircservserv-wm>	 Syncing #wikimedia-search (requested by gehel)
[14:14:52] <ircservserv-wm>	 Set /cs flags #wikimedia-search ejoseph +AVfiortv
[14:15:30] <majavah>	 -NickServ- ejoseph is not registered.
[14:15:35] <zpapierski>	 huh?
[14:15:43] <gehel>	 yeah, we'll get him to register his nick
[14:15:47] <zpapierski>	 ejoseph: did you register you nick with NickServ?
[14:15:48] <gehel>	 he's still onboarding
[14:16:04] <zpapierski>	 if not, you should do so as quickly as possible, for various reasons
[14:16:26] <gehel>	 ejoseph: and you should probably ping one of us for an introduction to the joys of IRC
[14:16:59] <zpapierski>	 do we issue a stern warning when people apply about IRC? I think it should be in the job description :)
[14:17:05] <gehel>	 quick errand to get some food for dinner
[14:17:27] <ejoseph>	 I did not @zpapierski
[14:18:07] <zpapierski>	 there is a point about it on the checklist, but if you still need an assist, I'm happy to help 
[14:18:13] <ejoseph>	 I am home now
[14:18:30] <ejoseph>	 I am trying to setup
[14:18:32] <zpapierski>	 with your new and shiny mac I presume :)
[14:21:21] <bd808>	 zpapierski: it should be in the standard Wikimedia benefits footer :)
[14:21:36] <bd808>	 it == irc
[14:21:52] <zpapierski>	 what is that they say about one man's treasure ;) ?
[14:22:22] <bd808>	 https://xkcd.com/1782/
[14:22:43] <zpapierski>	 one of my favs :)
[14:46:14] * gehel did not think that getting a COVID test for a 6 year old would be that complicated
[14:48:40] <dcausse>	 going offline, have a nice week-end!
[14:49:42] <gehel>	 dcausse: enjoy!
[15:00:32] <ebernhardson>	 \o
[15:00:37] <zpapierski>	 o/
[15:00:59] <ebernhardson>	 zpapierski: oauth proxy works locally, ship it? :P
[15:01:41] <zpapierski>	 I'm doing the review (sorry, sat down a bit to late to gerrit today)
[15:02:02] <ebernhardson>	 tis ok, i'm sure there's a mountain of problems still
[15:03:31] <zpapierski>	 I'm so happy to see that session store gone
[15:09:10] <gehel>	 2004 and 1003 completed transfers, waiting for catchup
[15:13:13] <zpapierski>	 nice, it's going really well
[15:13:39] <zpapierski>	 if we'd automate it, we could migrate eqiad over the weekend ;)
[15:14:47] <gehel>	 keep that idea for next time!
[15:15:01] <gehel>	 But yes, it would be nice to have a fully automated system
[15:17:33] <gehel>	 re-pooling 1003
[15:18:08] <gehel>	 and getting ready for the next batch: 1008 and 2007
[15:19:54] <gehel>	 will use 1003 as a source instead of 1009 to work around the depool limitations of the cookbook
[15:24:07] <gehel>	 data transfer started for 1008 and 2007
[15:34:32] <gehel>	 ryankemper: I'll let you take over from there
[15:35:00] <ryankemper>	 gehel: thanks!
[15:35:01] <gehel>	 ejoseph: we are in our unmeeting if you want to join: https://meet.google.com/xgq-wvik-dkp
[15:40:17] <ejoseph>	 my internet sucks
[15:40:29] <ejoseph>	 i cant load simple pages
[15:40:39] <zpapierski>	 :(
[15:50:12] <zpapierski>	 ebernhardson: apart from the test patch and maven stuff I'm refering to gehel, LGTM
[15:50:24] <zpapierski>	 test I don't yet fully understand, but it is marked WIP
[15:52:41] <ebernhardson>	 zpapierski: it's not any kind of automated testing, it's just a script to start kask+cassandra+jetty
[23:29:37] <ryankemper>	 It's absolutely insane how fast the new streaming updater catches up on lag
[23:29:39] <ryankemper>	 very exciting