[00:11:19] <ebernhardson>	 I put a banner in place on wcqs-beta.wmflabs.org, I imagine someone will ask to rewrite that copy though so let me know :)
[00:11:45] <ebernhardson>	 it says "Please try the new WCQS Beta deployment, now with more capacity and live updates. The old beta instance in wmflabs will be decomissioned March 1, 2022." with WCQS Beta as a link
[00:12:34] <ebernhardson>	 (it's also a total hack job, and wont get i18n translations like a normal banner would if we went through the full process)
[00:27:36] * ebernhardson realizes while answering someones question the feature sets used in mjolnir in prod don't exist anywhere except the prod instances.
[00:53:58] <ebernhardson>	 i guess they mostly exist in featuresets.py, but for some reason i suspect there are custom hax
[08:27:09] <gehel>	 ebernhardson: the banner looks good to me. Thanks!
[09:03:04] <gehel>	 ejoseph: around? https://meet.google.com/ukb-kgxq-gvq
[10:02:52] <zpapierski>	 adding the "Patches for Review" section to standup notes is a great idea
[10:04:17] <zpapierski>	 volans: mind looking at https://gerrit.wikimedia.org/r/c/operations/cookbooks/+/753426 when convenient?
[10:05:02] <gehel>	 zpapierski: around? available for a chat?
[10:05:09] <gehel>	 https://meet.google.com/sxf-uyhu-cyp
[10:05:13] <zpapierski>	 yep, just let me get my headphones
[10:05:23] <volans>	 zpapierski: sure, no prob
[10:05:32] <zpapierski>	 volans: thanks!
[10:07:48] <volans>	 zpapierski: just to clarify, you don't "need" my review for cookbooks, it's totally ok to self-review/merge them within your team. I'm always happy to review if you're looking for suggestions or have questions on how to do things or if there are other ways to do them
[10:08:39] <volans>	 (by self-review I meant review by members of the same team without me/j.ohn being involved)
[10:15:41] <volans>	 zpapierski: {done}
[10:18:24] <zpapierski>	 volans: thx! I'm not able to +2 there, actually...
[10:18:37] <gehel>	 I can do the +2, let me read it first
[10:38:53] <gehel>	 lunch + errand, I might be back slightly later than usual (covid vaccin for Oscar)
[10:39:55] <gehel>	 zpapierski: I +2 the cookbook
[10:40:29] <gehel>	 The comment from volans about the class API should be addressed separately and might be a good exercise for inflatador to get more familiar with our cookbooks
[10:41:16] <zpapierski>	 gehel: thx
[11:32:27] <dcausse>	 lunch
[11:38:55] <Amir1>	 Hi, We got email for DegradedArray event on /dev/md/0:elastic2035 
[11:39:06] <Amir1>	 let me know if you want the email
[11:46:19] <volans>	 Amir1: that's T298853 AFAICT
[11:46:19] <stashbot>	 T298853: Degraded RAID on elastic2035 - https://phabricator.wikimedia.org/T298853
[11:50:57] <Amir1>	 awesome
[11:51:09] <Amir1>	 one less root@ email to worry about
[11:51:39] <zpapierski>	 lunch
[13:33:21] <ottomata>	 dcausse:  i'm not sure why , but eventgate-main didn't have stream config for rdf-streaming-updater.reconcile
[13:33:31] <ottomata>	 i'm roll restarting it, that seems to be fixing it
[13:33:38] <dcausse>	 :|
[13:33:44] <ottomata>	 i'm not sure why though, it is in stream config correctly afaict and has been since you deployed it
[13:33:53] <ottomata>	 eventgate-main requests stream config on startup
[13:34:05] <dcausse>	 ah
[13:34:15] <ottomata>	 ...also, btw, do you think we should move the rdf-streaming-updater stuff into schemas/event/primary 
[13:34:21] <dcausse>	 so we've made a mistake
[13:34:23] <ottomata>	 it is a 'production' service
[13:34:25] <ottomata>	 dcausse: oh?
[13:34:56] <ottomata>	 dcausse:  i'm not sure what mistake you've made, from what I can tell you did everything right.
[13:35:02] <dcausse>	 I thought it was: (1) schema -> (2) eventgate  (restart) and (3) mw-config
[13:35:15] <ottomata>	 oh, but hasn't mw config for this  been there?
[13:35:20] <ottomata>	 thought i looked at git blame
[13:35:20] <ottomata>	 OH
[13:35:21] <ottomata>	 deploy
[13:36:20] <dcausse>	 I thought somehow that after deploying a mw-config patch then some stuff will be starting to produce data to it (canary  events) and then eventgate will be in a bad shape without the schema reloaded
[13:36:55] <dcausse>	 did not occur to me that eventgate would pull data out of meta
[13:37:32] <dcausse>	 hm... thinking about this, it seems they need to happen almost at the same time
[13:37:43] <dcausse>	 what does trigger canary events to be sent?
[13:38:12] <ottomata>	 just updated eventgate docs to indicate what happens with stream config for each eventgate
[13:38:13] <ottomata>	 https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate#EventGate_clusters
[13:38:26] <ottomata>	 eventgate-analytics-external is the only one that requests stream config dynamically
[13:38:34] <dcausse>	 ok
[13:38:47] <ottomata>	 dcausse:  declaration of stream will cause canary events to be sent to it
[13:38:58] <ottomata>	 the canary producer just reads all stream configs and produces
[13:39:41] <dcausse>	 ok so deploys to mw-config (stream config) + eventgate restart must occur at the "same" time
[13:40:32] <ottomata>	 yeah, doesn't have to be exactly same time
[13:40:37] <ottomata>	 but shortly after
[13:40:49] <dcausse>	 yes I get it now
[13:41:05] <ottomata>	 it isn't ideal, its just done that way to avoid runtime coupling
[13:41:14] <dcausse>	 sure
[13:41:29] <dcausse>	 and yes it's a prod like service, is this for primary vs secondary ?
[13:41:46] <ottomata>	 maybe we could make it request streams it doesn't have from mw api, buuuut then if someone asks for an invalid stream name, it will just ask every time
[13:41:52] <ottomata>	 we do have some caching in eventgate-analytics-external
[13:42:14] <dcausse>	 ottomata: I don't think it's a big deal once you know what to do
[13:42:15] <ottomata>	 erg. i dunno, i think the way it is is good, it is not expected that streams change often for anything but eventgate-analytics-external
[13:42:17] <ottomata>	 okay
[13:59:30] <gehel>	 inflatador, ryankemper: could you take care of the decommissioning step for elastic2035 (T298853) to ensure that we don't keep alerting people for nothing? See Amir1_ message earlier today.
[13:59:30] <stashbot>	 T298853: Degraded RAID on elastic2035 - https://phabricator.wikimedia.org/T298853
[14:03:38] <inflatador>	 gehel  ACK , will take a look
[14:05:14] <inflatador>	 Also, Happy Groundhog Day! https://www.farmersalmanac.com/when-groundhog-day-winter-forecast
[14:58:14] <ottomata>	 dcausse:  what MW state does streaming updater ask get from MW API currently?
[14:58:23] <ottomata>	 wikidata revision content?  commons revision content?
[14:58:24] <ottomata>	 is that it?
[14:58:38] <ottomata>	 do you ask for a specific format from the API?
[14:58:45] <ottomata>	 (or, point me to code that is dong this, I will read)
[15:01:27] <ottomata>	 oh I think I found it, WIkibaseRepository?
[15:02:06] <dcausse>	 ottomata: yes its main MW dependency is /wiki/Special:EntityData
[15:02:45] <dcausse>	 it's not under "api" but it really falls under the MW (wikibase) APIs
[15:03:50] <dcausse>	 example call is: https://commons.wikimedia.org/wiki/Special:EntityData/M114759078.ttl?flavor=dump&revision=625508650
[15:12:43] <ottomata>	 perfect
[15:12:55] <ottomata>	 okay that's a wikibase specific thing then
[15:12:57] <ottomata>	 cool
[15:42:26] <gehel>	 inflatador: there is no one for my refactoring workshop tonight, let's use that time to continue our earlier conversation (invite sent)
[15:46:54] <inflatador>	 gehel ACK, accepted
[16:02:55] <mpham>	 ebernhardson: Thanks for putting the banner in place. If you don't mind changing the copy to: "WCQS beta 2 is now available at https://commons-query.wikimedia.org/ and has live updates and better service reliability (release notes). This WCQS beta 1 instance in wmflabs will be decommissioned March 1, 2022, and all traffic will be automatically directed to WCQS beta 2 following that date."
[16:03:02] <mpham>	 release notes link: https://commons.wikimedia.org/wiki/Commons:SPARQL_query_service/WCQSbeta2-release-notes
[16:39:28] <inflatador>	 andreaw remind me to ask you about your bass sometime! My son just started playing
[17:00:26] <zpapierski>	 inflatador: we are holding an office hours now, wanna join (we forgot to add you to this one,sorry!)  - https://meet.google.com/vgj-bbeb-uyi?authuser=0?
[17:01:30] <inflatador>	 zpapierski gotta run errands but should be back in time for "unmeeting"
[17:02:01] <zpapierski>	 no problem! Trey314159 added you to the next one
[17:32:50] <Trey314159>	 unmeeting!
[17:57:49] <zpapierski>	 and I missed that it was moved :(
[18:22:11] <ebernhardson>	 answering random question from earlier, Webb has a 2kw solar array
[19:01:24] <inflatador>	 lunch, back in ~30-45
[19:43:54] <inflatador>	 back
[19:58:50] * ebernhardson would love to see an auto-formatter (as an open source tool that can be used in the CI pipeline to verify) for java instead of errors like 'Wrong import order' (with no hint of the correct order). I just push the magic button in idea and hope it passes next time :S
[20:13:50] * ebernhardson notes that almost every change to cpjobqueue config since april was someone increasing concurrency somewhere
[20:14:28] <ebernhardson>	 what i'm not finding is a good metric about per-queue concurrency, or even how data gets from cpjobqueue to prometheus :S has to be somewhere...
[20:21:29] * ebernhardson apparently asked per-job concurrency graphs in 2020 and the answer is T255684, still pending
[20:50:33] <ebernhardson>	 mpham: banner updated
[20:51:18] <ebernhardson>	 lunch
[20:58:07] <inflatador>	 out for the day, see y'all tomorrow
[21:30:00] <ebernhardson>	 baclk
[21:30:42] <ebernhardson>	 so these job queue graphs, i guess we can derive concurrency from the job completion rate and 50th percentile latency? Something doesn't feel right about it but maybe...
[21:41:57] <ryankemper>	 ebernhardson: I imagine that should get us to a reasonable approximation
[21:42:36] <ryankemper>	 I should catch back up on reading your (excellent) writeups on that phab ticket tho, sec
[21:44:00] <ryankemper>	 ebernhardson: btw what are the graphs you're referring to here?
[21:44:07] <ryankemper>	 > We suspect though that ElasticaWrite jobs are already consuming 300 concurrent runners and the graphs don't really show any different activity before/after saneitizer deployment.
[21:44:24] <ryankemper>	 I imagine that's a jobrunner specific graph or something along those lines?
[21:46:08] <ryankemper>	 Ah must be https://grafana.wikimedia.org/d/LSeAShkGz/jobqueue?orgId=1
[21:46:46] <ebernhardson>	 ryankemper: there is a graph, but i asked the same thing a few years ago and this was response: https://phabricator.wikimedia.org/T266762#6625756
[21:47:25] <ryankemper>	 oof, I see
[21:47:26] <ebernhardson>	 oh i totally missed first question :) Yea it's JobQueue Job graph
[21:48:57] <mpham>	 ebernhardson: thanks!
[21:50:30] <ebernhardson>	 I'm dubious on whats really going on here, i feel like we aren't using 300 concurrent runners but can't find proof yet
[21:53:58] <ryankemper>	 just echoing what you were saying in your phab comment, there's no noticeable spike in job insertions during the period in which saneitizer was activated / subsequently disabled
[21:54:10] <ryankemper>	 here's the cirrus side of things for eqiad: https://grafana.wikimedia.org/d/000000455/elasticsearch-percentiles?viewPanel=85&orgId=1&from=1643182091936&to=1643848004131&var-cirrus_group=eqiad&var-cluster=elasticsearch&var-exported_cluster=production-search&var-smoothing=1
[21:54:21] <ryankemper>	 and the corresponding time range for the `JobQueue Job` dashboard: https://grafana.wikimedia.org/d/CbmStnlGk/jobqueue-job?orgId=1&from=1643182091936&to=1643848004131
[21:56:04] <ebernhardson>	 if we use a simple derivation, p50 is pretty constant at 250-300ms, job retirement rate peaks around 200.  That would imply significantly less than 300 concurrent requests
[21:56:13] <ebernhardson>	 300 concurrent requests at 300ms should retirn 900/s ?
[21:56:25] <ebernhardson>	 *retire
[21:57:27] <ebernhardson>	 if we suspect that cpjobqueue isn't giving the desired concurrency, i suppose i'd be less worried about bumping it up
[21:57:27] <ryankemper>	 Math checks out (minus a sig fig)
[21:57:56] <ryankemper>	 I guess technically that napkin math could lie to use if the p99 or p01 massively shifted after the intervention...but there's no way
[21:58:30] <ebernhardson>	 yea i'm not super thrilled with deriving instead of logging concurrency directly. There is a Set() object per-job that they could report the length of and have real direct concurrency. Not clear why they prefer to derive it
[22:03:45] <ryankemper>	 ebernhardson: btw where do we actually set the elastica jobrunner concurrency parameter (or whatever it's properly called)? my first guess would be `mediawiki-config` but my (likely insufficient) greps aren't finding it
[22:07:10] <ebernhardson>	 ryankemper: operations/deployment-charts repository in helmfile.d/services/changeprop-jobqueue/values.yaml
[22:08:49] <ebernhardson>	 I don't think of that one naturally either, but searching for ElasticaWrite turns it up since i had it cloned already
[23:18:04] * ebernhardson wonders what the rate limiting in cpjobqueue does, never saw that before