[14:40:31] <fnielsen>	 I am trying to use Wikimedia Chat at https://chat.wmcloud.org. I have been trying to create an account but am missing the email verification step. I do not receive and email and I have check the spam filter. Does anyone knows how to register?
[14:55:47] <fnielsen>	 I am using an invite link that is available on this page: https://en.wikipedia.org/wiki/Wikipedia:Wikipedia_Mattermost
[15:15:45] <bd808>	 fnielsen: Amir1 is the only admin of that project today per https://openstack-browser.toolforge.org/project/chat. Maybe he can help?
[15:16:06] <Amir1>	 oh hi
[15:16:14] <Amir1>	 yeah, let me produce one
[15:16:57] <Amir1>	 hmm, so it's a mailserver issue
[15:17:04] <Amir1>	 let me check
[15:22:41] <Amir1>	 I was about to say, please try again, I restarted exim4 there but the person has left the channel 
[15:26:25] <bd808>	 Amir1: you could leave them a note at https://en.wikipedia.org/wiki/User:Fnielsen
[15:26:48] <Amir1>	 I will
[15:26:50] <Amir1>	 thanks
[15:27:05] <bd808>	 thank you for prompt irc tech support :)
[15:27:39] <balloons>	 !log tools.sdzerobot set deployment and cpu limit to 5
[15:27:43] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.sdzerobot/SAL
[15:28:50] <balloons>	 !log tools.sdzerobot set deployment and cpu limit to 5 T334913
[15:28:52] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.sdzerobot/SAL
[15:28:52] <stashbot>	 T334913: k8s quota increase request for sdzerobot tool - https://phabricator.wikimedia.org/T334913
[19:57:12] <wm-bot>	 !log tools.wd-image-positions <lucaswerkmeister> deployed c6cf151896 (update dependencies, mainly Flask+Werkzeug 2.3)
[19:57:15] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wd-image-positions/SAL
[20:25:49] <wm-bb>	 <lucaswerkmeister> pod wd-image-positions-7f449d6fb6-xm9l9 seems to be stuck terminating
[20:25:51] <wm-bb>	 <lucaswerkmeister> kubectl get events has:
[20:25:53] <wm-bb>	 <lucaswerkmeister> error killing pod: failed to "KillContainer" for "webservice" with KillContainerError: "rpc error: code = Unknown desc = Error response from daemon: cannot stop container: bfad3389eef89d46f2fe69faec89dc6e276bcf825b93cce1ba9bc9f42f8aa0f4: tried to kill container, but did not receive an exit event"
[20:33:14] <wm-bb>	 <lucaswerkmeister> looks like I can ssh into the worker and see the container in docker ps – would things break terribly if I tried to docker kill it?
[20:35:03] <bd808>	 @lucaswerkmeister: there's one way to find out... but I think k8s would deal with that
[20:35:16] <wm-bb>	 <lucaswerkmeister> https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Kubernetes says “It is ok to kill pods on individual nodes” ^^
[20:35:24] <wm-bb>	 <lucaswerkmeister> and IIUC a pod = a docker container
[20:35:41] <bd808>	 mostly, yes. And for that doc almost certainly
[20:35:42] <wm-bb>	 <lucaswerkmeister> (assuming docker is used as the container runtime at all)
[20:35:46] <wm-bb>	 <lucaswerkmeister> I’ll try it ^^
[20:37:04] <wm-bb>	 <lucaswerkmeister> meh. Error response from daemon: Cannot kill container: bfad3389eef8: tried to kill container, but did not receive an exit event
[20:37:28] <wm-bb>	 <lucaswerkmeister> I suppose I just did the same thing that k8s tried, really
[20:39:08] <wm-bb>	 <lucaswerkmeister> it’s probably fine to just leave it alone for now? I don’t think it’ll do much harm except make my `kubectl get pods` output slightly more confusing
[20:39:58] <wm-bb>	 <lucaswerkmeister> ah, in htop I can see that all five uwsgi processes are blocked on disk
[20:40:11] <wm-bb>	 <lucaswerkmeister> so let’s wait a bit and see if NFS recovers maybe