[11:35:48] <elukey>	 hi folks!
[11:36:22] <elukey>	 there seems to be something weird happening on kafka-logging100[4,5], some ISRs are failing
[11:37:07] <elukey>	 the errors are all like
[11:37:08] <elukey>	 "org.apache.kafka.common.errors.TopicAuthorizationException: Not authorized to access topics: [Topic authorization failed."
[11:37:27] <elukey>	 that seem related to ACLs, but not sure if you added/removed anything recently
[13:05:49] <herron>	 elukey: https://phabricator.wikimedia.org/T334733#8823904 comes to mind
[13:31:10] <elukey>	 herron: ah wow that's definitely a possible culprit!
[13:31:26] <elukey>	 I was wondering why we had only that rule in kafka logging, and the time matches perfectly
[13:31:42] <herron>	 I just added a comment on that task, yeah timing really is perfect
[13:32:01] <elukey>	 ottomata: o// around?
[13:32:42] <elukey>	 I think we can just
[13:32:48] <elukey>	 kafka acls  --delete --allow-principal User:ANONYMOUS --cluster --operation IdempotentWrite
[13:32:54] <elukey>	 and test if it solves the issue
[13:33:03] <elukey>	 logging is the only one not using ACLs I think
[13:33:11] <elukey>	 so probably adding one triggers some defaults
[13:33:41] <herron>	 ok, yeah makes sense sgtm
[13:33:56] <elukey>	 the alternative is to come up with ACLs (maybe matching the ones on kafka-main)
[13:34:00] <elukey>	 but it could be risky
[13:34:17] <herron>	 I was having a look around to see if other clusters had ACL entries that lined up with the error from kafka-authorizer.log like this
[13:34:19] <herron>	 User:CN=kafka-logging1005.eqiad.wmnet is Denied Operation = ClusterAction from host = 10.64.135.13 on resource = Cluster:kafka-cluster
[13:34:30] <herron>	 but didn't see anything for that operation offhand
[13:34:40] <herron>	 yeah revert seems best for now
[13:35:14] <elukey>	 herron: ok to proceed?
[13:35:18] <herron>	 +1
[13:36:42] <elukey>	 done
[13:36:45] <elukey>	 let's see
[13:36:57] <herron>	 annnd the errors immediately stopped
[13:37:30] <herron>	 yeah replicas catching up already
[13:38:48] <herron>	 good catch elukey thank you, now we know to tail the kafka journal for a bit after acl change
[13:39:18] <elukey>	 np! Very weird state of ACLs across clusters though, I didn't know we had some for main for example
[13:40:24] <herron>	 same I'm wondering which of the ACLs in main is addressing this issue there
[14:28:13] <ottomata>	 elukey:  o/
[14:28:35] <ottomata>	 OHH NOOO
[14:28:42] <elukey>	 yeah :(
[14:28:57] <ottomata>	 i'm sorry i didn't even check that possiblity!  gahhhh
[14:29:12] <ottomata>	 its been so long since we had to change ACLs that i thought how could it hurt adding one for anon
[14:29:28] <ottomata>	 but of course adding one for anon meant it started thinking about anons!
[14:29:52] <elukey>	 I think so yes, I wasn't aware of that behavior either :(
[14:30:17] <ottomata>	 we should remoe it from logging codfw too
[14:30:20] <elukey>	 we can probably think about a meaningful set of ACLs to add for a cluster, kafka main could be used as baseline (didn't know it had ACLs)
[14:30:30] <elukey>	 ah right yes, doing it
[14:30:42] <ottomata>	 i mean, if we had somethign like https://phabricator.wikimedia.org/T276088 config management for kafka stufff
[14:31:25] <elukey>	 wow we don't have the problem in logging-codfw
[14:32:04] <elukey>	 I am wondering if it triggers when one creates topics or similar
[14:33:00] <ottomata>	 like if the anon perms get applied?  yea dunno
[14:33:20] <elukey>	 I created test-elukey now, let's see
[14:33:31] <ottomata>	 k will you delete acl when you ready?
[14:33:33] <ottomata>	 or should I?
[14:33:57] <elukey>	 I will no problem, but I can't repro the issue
[14:34:09] <elukey>	 mmm maybe I need to produce to ti
[14:34:55] <ottomata>	 weird
[14:35:31] <elukey>	 nope, all good
[14:43:16] <ottomata>	 strange
[14:46:14] <elukey>	 before removing it I'd love to repro the issue, not sure how to trigger it though
[15:18:45] <elukey>	 herron: lemme know what you prefer, shall we simply remove from logging-codfw too and figure out what ACLs to rollout next?
[15:22:24] <herron>	 elukey: sgtm, strange indeed that it isn't happening in codfw
[15:22:36] <herron>	 but good I think to bring back to known good state
[15:23:04] <elukey>	 done!
[15:23:09] <herron>	 ty ty
[18:10:20] <rzl>	 herron: hey, do these grr preview dashboards expire automatically? or do I need to be conscientious about cleaning up after myself with the delete links?
[18:46:51] <herron>	 there is a flag to expire them, but its nothing much to worry about they can be cleaned up via the grafana ui too
[19:13:20] <rzl>	 ah okay, cool
[20:00:24] <rzl>	 hrm, I was wrong about that recording rule -- it turns out we were dividing a percentage by a percentage, so the existing math is correct -- it's just that one of the percentages is named "ratio," a little misleadingly
[20:01:10] <rzl>	 so I do want to get them both to unit scale but (a) we'll have to figure out what to do about the naming, and (b) there's no rush to change it, since it isn't wrong