[05:24:20] <ryankemper>	 Neat WDQS-powered browser extension I came across when looking at https://phabricator.wikimedia.org/T362570: https://www.wikidata.org/wiki/Wikidata:Entity_Explosion
[10:04:53] <dcausse>	 added notes to the staff meeting slides, please double check and fix/add anything I would have missed (cc pfischer, gehel, dr0ptp4kt)
[10:05:53] <dcausse>	 hm... seems like the slides publicly editable so not sharing the link here
[10:08:08] <dcausse>	 lunch
[12:24:15] <gehel>	 pfischer / dcausse: do we have a number for the updates / second on SUP?
[12:31:21] <dcausse>	 gehel: at the top of https://grafana-rw.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=eqiad%20prometheus%2Fk8s&var-namespace=cirrus-streaming-updater&var-helm_release=producer&var-flink_job_name=cirrus_streaming_updater_producer_eqiad&var-operator_name=All
[12:32:05] <dcausse>	 consume rate total is the number of events ingested, produce rate total is the number of updates we will apply to elastic
[12:33:04] <dcausse>	 minus the updates triggered by the saneitizer
[13:08:46] <pfischer>	 gehel: additionally, if you are just interested in updates sent to ES, our dedicated SUP dashboard has a metric for that: 
[13:08:46] <pfischer>	 https://grafana.wikimedia.org/d/jKqki4MSk/cirrus-streaming-updater?orgId=1&var-k8sds=eqiad%20prometheus%2Fk8s&var-opsds=eqiad%20prometheus%2Fops&var-site=eqiad&var-app=flink-app-consumer-cloudelastic&var-app=flink-app-consumer-search&var-operator_name=rerender_fetch_failure_sink:_Writer&var-fetch_operator_name=fetch&var-fetch_operator_name=rerender_fetch&var-quartile=0.5&var-fetch_by=authority&var-top_k=1&from=now-24h&to=
[13:08:47] <pfischer>	 now&viewPanel=50
[13:10:22] <pfischer>	 But, as dcausse pointed out, this includes updates from more sources than just page_change(-inflicted)
[13:35:22] <inflatador>	 ryankemper nice extension!
[13:35:26] <inflatador>	 <o/
[13:35:35] <dcausse>	 o/
[17:11:01] <dcausse>	 dinner
[19:12:57] <pfischer>	 dcausse: thank you for the review and added tests, I’ll adapt the implementation tomorrow
[20:01:59] <ottomata>	 inflatador: o/
[20:02:15] <inflatador>	 ottomata I'm gonna try `helmfile destroy/apply` as opposed to `rollout restart` (already tried that and it failed)
[20:02:43] <ottomata>	 oh okay yes.
[20:02:50] <ottomata>	 sorry yeah, i think rollout restart is just going to delete pod like I did
[20:04:03] <inflatador>	 OK, destroy/apply finished, let's see what happens now
[20:05:13] <ottomata>	 better so far!
[20:05:16] <inflatador>	 we have task managers at least
[20:07:11] <ottomata>	 failed.  looks like a json error?
[20:07:17] <ottomata>	 at least i have a cause now!
[20:08:30] <inflatador>	 back to having no task mgrs
[20:08:58] <inflatador>	 Ah, what are the next steps? Roll back a code change somewhere?
[20:10:38] <ottomata>	 no we haven't deployed anyting in a long time
[20:10:40] <ottomata>	 i'm not sure
[20:10:44] <ottomata>	 trying to keep https://phabricator.wikimedia.org/T368667 updated
[20:14:19] <inflatador>	 We might have to wait for g-modena to come in tomorrow?
[20:14:38] <inflatador>	 The other question is why DPE SRE didn't get these alerts, at least I haven't found them yet. I can probably fix that
[20:15:53] <inflatador>	 dcausse do the errors listed above look familiar? Just curious if we have had similar issues with SUP
[20:16:46] <ottomata>	 i see them going to sre-observability@wikimedia.org
[20:16:46] <ottomata>	 cc:	data-engineering-alerts@wikimedia.org
[20:17:28] <inflatador>	 that 2nd email addr looks right. Not sure if the list addrs are considered sensitive data though
[20:18:09] <ottomata>	 wait.
[20:18:12] <ottomata>	 i think i know.
[20:18:14] <inflatador>	 oh, I did get the alert btw
[20:18:20] <inflatador>	 started about an hr ago
[20:19:13] <ottomata>	 this is my bad
[20:19:14] <ottomata>	 https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1047995
[20:19:16] <ottomata>	 needs deployed....
[20:19:20] <ottomata>	 the train rolled out to this wiki today
[20:19:24] <ottomata>	 where this problem is happening!
[20:19:29] <ottomata>	 on it...
[20:20:38] <ottomata>	 I did not follow through
[20:20:38] <ottomata>	 https://phabricator.wikimedia.org/T367923#9913264
[20:21:18] <ottomata>	 applying
[20:21:31] <inflatador>	 NP, glad you have the solution!
[20:23:33] <ottomata>	 inflatador: um, staging did not deploy for me
[20:23:40] <ottomata>	 helmfile was happy
[20:23:52] <ottomata>	 Error creating: pods "flink-app-main-6844786bb8-w2btg" is forbidden: violates PodSecurity
[20:23:55] <ottomata>	 in kubectl get events
[20:24:18] <ottomata>	 assuming its weird staging stuff, trying codfw...
[20:24:45] <inflatador>	 I know we've been adding PodSecurity contexts lately, but I don't think they're obligatory?
[20:25:17] <inflatador>	 The git log on deployment-charts probably has some examples if we do end up needing to do that
[20:26:46] <ottomata>	 failed in codfw, but also my fault.
[20:26:51] <ottomata>	 need schemas in image
[20:26:55] <ottomata>	 releasing new image now...
[20:27:19] <ottomata>	 https://gitlab.wikimedia.org/repos/data-engineering/mediawiki-event-enrichment/-/pipelines/60820
[20:43:24] <inflatador>	 how's it looking?
[20:43:51] <ottomata>	 better
[20:43:52] <ottomata>	 Job status changed from CREATED to RUNNING
[20:44:48] <ottomata>	 gotta catch up from lag now
[20:53:02] <inflatador>	 ACK, ping me if you need anything else
[21:00:47] <inflatador>	 ottomata FYI if you need a user that has perms to `helmfile destroy` but you don't wanna use root, you. can do `kube_env ${APP_NAME}-deploy $ENV` . Note the "deploy"
[21:01:25] <inflatador>	 sorry, should be `${NAMESPACE}-deploy`
[21:23:02] <ottomata>	 really!?
[21:23:11] <ottomata>	 cool, inflatador could you add that to the k8s wikitech page?
[21:23:20] <ottomata>	 inflatador: all looks better now!
[21:28:32] <inflatador>	 ottomata added to https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments . Glad everything looks good!