[02:59:03] <mpham>	 I'm looking at https://grafana.wikimedia.org/d/000000489/wikidata-query-service?viewPanel=8&orgId=1&refresh=1m&from=now-30d&to=now and noticing that prior to Streaming Updater release, most servers are <1min lag when not spiking, but in the past couple of days, they're all uniformly just over 1m lag. Is that expected?
[06:39:30] <zpapierski>	 mpham: it's expected - while current pipeline has much higher throughput than the old one, which means less (by far) spikes, it is also a longer one, somewhat by design. To better reconcile the differences between entities and sync few different changes topics (revision-create, delete-page,etc) a partial ordering is being peformed, which requires some time window (which is now set to 1m).  
[07:04:56] <gehel>	 Also the lag is sampled every minute, so the best precision of that metric is 1 minute. A degradation of 1 minute is within the error margin of the metric.
[07:05:11] <gehel>	 Yes, I know I should be on vacation
[07:08:55] <zpapierski>	 really?  I thought it doesn't really matter what the sampling rate here is, time() will return the timestamp at the time of the sampling?
[07:09:02] <zpapierski>	 (calculation is time() - blazegraph_lastupdated)
[08:05:46] <zpapierski>	 ok, now I have a percentage share in traffic for each instance 
[08:06:05] <zpapierski>	 I don't think we're going to get anything better from preexisting values
[10:06:23] <zpapierski>	 dcausse: mind if I create a short meet after Wikidata sync to discuss what metric are we interested in for SLO? I'm still verifying the my math on dashboards (I think I'm missing something in my traffic percentage calculation), but I think we know enough to make an informed decision on what do we want to commit to
[10:07:11] <dcausse>	 zpapierski: please do
[10:12:36] <zpapierski>	 hmm, actually he's around before retro, even better
[10:16:33] <dcausse>	 perfect!
[10:16:39] <dcausse>	 lunch
[11:01:00] <zpapierski>	 break
[14:30:52] <zpapierski>	 mpham: around to meet?
[14:31:01] <mpham>	 yeah, one sec
[14:49:01] <cbogen_>	 anyone else see this? https://phabricator.wikimedia.org/T294025
[14:56:09] <mpham>	 zpapierski: I think this is the same issue I brought up earlier in the chat. Sounds like maybe we ought to explain what;s happening more generally so people understand
[14:56:16] <zpapierski>	 yep, will do
[14:57:56] <ebernhardson>	 \o
[15:00:24] <cbogen_>	 sorry, should have scrolled up. Yes, I think some communication around this to the community is a good idea - if something was degraded in favor of the improvements, we should communicate that choice
[15:05:58] <Trey314159>	 ebernhardson: retro?
[15:36:23] <ebernhardson>	 zpapierski: what do i need to do to deploy mw-oauth-proxy to wcqs? trigger a build in jenkins, but then what?
[15:36:54] * ebernhardson could probably find it, but asking too :P
[15:36:55] <zpapierski>	 standard blazegraph deployment is when it happens
[15:37:18] <ebernhardson>	 zpapierski: hmm, is there any danger in doing a full blazegraph deployment then, will that force wdqs too?
[15:38:06] <zpapierski>	 not sure, I never used scap for wcqs-beta
[15:38:58] <ebernhardson>	 heh, actually i never updated scap to sync to wcqs :P
[15:39:09] <zpapierski>	 I guess you can do a group deployment? as in create wcqs group, alongside wdqs, wdqs-internal, etc?
[15:39:13] <ebernhardson>	 ok, i'll do that. It can be a separate scap group
[16:10:37] <zpapierski>	 cbogen_: I answered in the ticket - https://phabricator.wikimedia.org/T294025#7448648
[16:10:56] <zpapierski>	 I'm tempted to close it now as weel, since we now this isn't a bug
[16:11:43] <zpapierski>	 s/weel/well
[17:20:22] <ebernhardson>	 aww, i don't have permission to submit in wikidata/query/deploy :) Anyone want to send these two: https://gerrit.wikimedia.org/r/c/wikidata/query/deploy/+/732742/1
[17:38:08] <ryankemper>	 ebernhardson: done
[17:38:14] <ebernhardson>	 thx
[17:38:45] <ryankemper>	 we should get you merge rights as well
[17:38:51] <ryankemper>	 i assume that's somewhere in the gerrit repo settings
[17:39:13] <ebernhardson>	 yea, looks like guillaume or david are in wikidata-query-admins and would need to add me
[17:40:26] * ebernhardson somehow expects this to fail anyways, i suspect this scap deploy is still too tied to wdqs. Will find out :)
[17:40:55] <ryankemper>	 heh yeah I had the same "concern" (re deploy possibly not working)
[17:41:14] <ryankemper>	 I'll do a deploy today so we can see (/ we need to anyway, haven't deployed since completing the streaming updater cutover)
[17:41:33] <ebernhardson>	 yea, i'm expecting this to fail today but hopefully can figure it out for next week :)
[17:42:54] <ebernhardson>	 for wcqs i wasn't going to do the full deploy procedure, although i guess we could. I was going to scap sync only wcqs 
[17:43:11] <ebernhardson>	 i suppose best to do a proper deploy, will get awkward if we have different versions running in different places
[17:43:58] <cbogen_>	 zpapierski: thanks for the response, it's good! i say we wait a day or two and then close it
[17:44:10] <ryankemper>	 ebernhardson: do you remember how we add to `wikidata-query-admins`? I'm blanking
[17:46:32] <ebernhardson>	 ryankemper: i mean david and guillaume are in wikidata-query-owners (s/admins/owners/, oops). https://gerrit.wikimedia.org/r/admin/repos/wikidata/query/deploy,access says i need to be in wikidata-query-deploy which is owned by wikidata-query-owners
[17:46:39] <ebernhardson>	 so i think either of them can add me in the gerrit ui somewhere
[17:48:20] <ryankemper>	 I'm in `wikidata-query-deploy` (along with david/guillaume) but I don't see how to see who's in `wikidata-query-owners`specifically
[17:48:38] <ryankemper>	 https://usercontent.irccloud-cdn.com/file/anNwuDM6/Screen%20Shot%202021-10-21%20at%2010.48.30%20AM.png
[17:49:16] <majavah>	 https://gerrit.wikimedia.org/r/admin/groups/fa916121a583488e2a983d14a3ede1c455782dbe,members
[17:50:28] <ebernhardson>	 yup, mostly re-browsed to group search, click through, and then there is a `members` link in the sidebar
[17:51:33] <ryankemper>	 ah gotcha (& thanks majavah), well hopefully dcausse can add you (and me) to https://gerrit.wikimedia.org/r/admin/groups/fa916121a583488e2a983d14a3ede1c455782dbe,members thru the UI tomorrow
[17:54:32] <ryankemper>	 ebernhardson: what was the scap command you were envisioning btw? would it be a `scap sync-world`?
[17:54:57] <ryankemper>	 the only command i'm aware of are `scap sync-file` and `scap sync-world` (i'd bet there's more tho)
[17:55:00] <ryankemper>	 commands*
[17:56:28] <ebernhardson>	 it's scap deploy with cli options i believe, lemme check
[17:58:44] <ebernhardson>	 i guess `scap deploy -l 'wcqs*'`  should work, but i'm sure i've seen a single-group deploy option before. Still looking :)
[18:02:42] <ebernhardson>	 hmm, interestingly reading through scap code there doesn't seem to be a way to choose the server groups, those are strictly from config. We would indeed need to use the limit-hosts option
[18:13:23] <ryankemper>	 Okay I like the idea of rolling the deploy for just wcqs first before doing a full one
[18:13:50] <ryankemper>	 And also if wcqs deploy is broken in a way that isn't trivially fixable today I can still deploy wdqs w/ `-l wdqs*`
[18:18:45] <ebernhardson>	 yea sounds reasonable
[18:30:25] <mpham>	 Trey314159: do you want to be invited to the ElasticSearch planning/kickoff? I couldn't remember how involved you are in this process  but didn't want to exclude you either (I think everyone else should probably be there)
[18:31:45] <Trey314159>	 mpham, yeah, please include me. I usually do a quick round of testing of analyzers to see if there are any unexpected changes. We've caught a few minor things that way—and discovered they'd added Ukrainian one year!
[18:32:12] <mpham>	 ok will do! 
[19:10:20] <ebernhardson>	 yea scap isn't happy, the problem is it's trying to restart the service but the service name varies
[20:03:07] <ebernhardson>	 ryankemper: if you can merge this one it might work this time: https://gerrit.wikimedia.org/r/c/wikidata/query/deploy/+/732772
[20:47:01] <ryankemper>	 ebernhardson: merged
[20:48:00] <ebernhardson>	 ryankemper: thanks!
[20:55:31] <ebernhardson>	 hmm, nope. dry-run deploying the wcqs environment tried to deploy to canary...hmm
[21:01:44] <ebernhardson>	 huh, and then ignoring that and specifying the config file directly it fails to connect to wcqs2003: Load key "/etc/keyholder.d/deploy_service.pub": invalid format  
[21:02:00] <ebernhardson>	 so, more to figure out :)
[21:18:01] <ryankemper>	 Weird that it'd be invalid format
[21:19:06] <ryankemper>	 The contents of the pubkey start with `ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvbV8H7Vzy...`, i.e. it looks pretty standard
[21:20:08] <ryankemper>	 It's possible that format was invalidated and there's a new format now, but wouldn't someone else have gotten bit by that already if the `deploy_service` key is shared across services? hmm
[21:26:57] <ebernhardson>	 Testing a bit, i think it might be something happening on the deploy host side, perhaps intentional
[21:27:55] <ebernhardson>	 doing a normal `scap deploy --dry-run -l 'wcqs*'` looks to talk to wcqs ok, but adding a `-c scap/environments/wcqs/scap.cfg` causes the keyholder error
[21:32:39] <ryankemper>	 ebernhardson: looking back at your earlier message...what was the command that led to it trying to deploy to the canary?
[21:33:47] <ryankemper>	 `scap/scap.cfg` has `server_groups: canary,default`, which is absent in `scap/environments/wcqs/scap.cfg`, so i'm wondering if the environment scap.cfg functions more as an override meaning that it's still getting `server_groups` set by `scap/scap.cfg`
[21:33:59] <ryankemper>	 Or if the command you ran just doesn't involve `scap/environments/wcqs/scap.cfg` and that's why
[21:33:59] <ebernhardson>	 ryankemper: hmm, sec lemme reconnect
[21:34:29] <ebernhardson>	 ryankemper: scap deploy --dry-run --no-log-message -v --environment wcqs
[21:34:54] <ryankemper>	 Okay so I imagine that it always reads `scap/scap.cfg` and just uses the environment as an override
[21:34:57] <ebernhardson>	 ryankemper: from the docs i had the impression only vars.yaml is merged between environments, otherwise files in the environment override the root
[21:36:00] <ebernhardson>	 i think now on closer review though, it's also doing merging inside scap.cfg, so i can probably drop most of whats in the wcqs config and only list the differences
[21:36:35] <ebernhardson>	 (the merging would also be where canary comes from, server_groups is canary,default in top level scap.cfg, wcqs i left it blank to take the system default of `default`
[21:36:58] <ryankemper>	 Sounds reasonable...so we'll want to make sure we override `server_groups` in the wcqs env so that we omit the canary
[21:39:55] <ebernhardson>	 ryankemper: so i think https://gerrit.wikimedia.org/r/c/wikidata/query/deploy/+/732797 will do it, hopefully :)
[21:40:52] <ryankemper>	 ebernhardson: LGTM, I'll let you do the honors
[21:42:06] <ebernhardson>	 looks happy in dry run, letting it go for real
[21:44:01] <ebernhardson>	 well, kinda sorta. It's complaining that i forgot to set the new required config values
[21:46:11] <ryankemper>	 new required config values?
[21:47:02] <ebernhardson>	 it needs a random secret value to use as the JWT secret key, and it needs a url for kask
[21:47:40] <ebernhardson>	 i suppose the secret needs to be deployed from puppet, has to be same on each and private
[21:54:54] <ryankemper>	 okay, if you go ahead and throw a secret into a private phab I can get started on getting it into /srv/private as well as the `operations/puppet` logic to stick the secret where it needs to go
[21:55:22] <ryankemper>	 does it just need to be written to a file on disk somewhere?
[21:55:49] <ebernhardson>	 ryankemper: it doesn't have to be anything in particular, probably some reasonable length and representable in mostly ascii since it goes into a defaults file and sourced into a shell script
[21:56:14] <ryankemper>	 okay sounds simple enough
[21:56:18] <ryankemper>	 yeah i'll keep it ascii armorable :P
[21:56:43] <ryankemper>	 or whatever the appropriate technical term is for "not a bunch of binary gobbledygook"
[21:57:14] <ebernhardson>	 i mean, /dev/urandom isn't a horrible place to get a secret. But it might need to be encoded first :)
[21:57:45] <ebernhardson>	 i'm working out the little bits for exact property names and where they go, shouldn't take too long
[22:00:46] <ryankemper>	 they'll never figure out that the password is `{WӭwvwS?dBXֈ9jҔyz/m1u"]XKzdx;wk|`
[22:22:27] <ebernhardson>	 so i think https://gerrit.wikimedia.org/r/c/wikidata/query/deploy/+/732800 updates the startup script as necessary, and then https://gerrit.wikimedia.org/r/c/operations/puppet/+/732801 has puppet provide them, needs a secret added