[09:48:42] <elukey>	 The service ops team needs to reboot our codfw ores redis primary, going to failover to the standby (will require an ores roll restart)
[11:08:04] <elukey>	 aaand now failing back
[12:33:56] <wikibugs>	 10Lift-Wing, 10Machine-Learning-Team: Add network policies to the ML k8s clusters - https://phabricator.wikimedia.org/T289834 (10elukey) List of ports used by various containers (got via nsenter):  * istio ingress gateway: ` Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address...
[13:48:41] <chrisalbon>	 morning all
[13:49:17] <elukey>	 good morning :)
[15:10:50] <elukey>	 first error in helm3 deployments
[15:11:02] <elukey>	  User "revscoring-editquality-deploy" cannot get resource "inferenceservices" in API group "serving.kubeflow.org" in the namespace "revscoring-editquality"
[15:11:15] <elukey>	 that seems cryptic but it is a step forward, it needs some extra permissions
[15:17:49] <chrisalbon>	 the march of pain continues
[16:33:32] * elukey afk!
[18:19:12] <wikibugs>	 10Lift-Wing, 10Machine-Learning-Team, 10ORES, 10artificial-intelligence, and 2 others: Developing the `algo-accountability` repository - https://phabricator.wikimedia.org/T290746 (10Isaac) > Just wanted to re-respond to your post about protected classes with a strong measure of agreement. Yay! The beauty o...