[07:22:46] Is there any alternative SSH port for the Cloud bastion? (annoying airport wifi blocking port 21) [07:26:08] SSH is port 22, no? FTP is port 21. [07:29:39] lol [07:29:47] yep ssh is 22 [08:07:54] !log tools reboot tools-sgebastion-10 (T316544) [08:07:57] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [08:07:57] T316544: Upgrade cloudsw1-c8-eqiad and cloudsw1-d5-eqiad to Junos 20+ - https://phabricator.wikimedia.org/T316544 [08:08:04] !log tools reboot tools-mail-03 (T316544) [08:08:06] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [11:45:27] !log toolsbeta release toolforge-weld 0.2.0 and toolforge-webservice 0.98 [11:45:30] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [11:52:21] !log tools release toolforge-weld 0.2.0 and toolforge-webservice 0.98 [11:52:23] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [12:22:09] !log tools.heritage Run check_emailable_users.py -category:"Images_from_Wiki_Loves_Earth_2023" -delta:23040 -notify to catch up with the first 16 days of WLE 2023 [12:22:11] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.heritage/SAL [13:41:51] bd808: think you can lend me a hand with an email issue? [13:44:21] Skynet: there are quite a few people on this channel who probably can.. https://dontasktoask.com/ [13:45:31] Great, I used the docs to setup PHPMailer to send emails through the given endpoints. They don't seem to be sending. Do you think you could help me figure out what is going on the iabot tool? [13:45:56] Following https://wikitech.wikimedia.org/wiki/Help:Toolforge/Email#Mail_from_Tools [13:47:22] do you have an example message-id I can grep for in the logs? [13:47:30] and show your code please [13:47:41] No. Not even aware of the message ID honestly. [13:48:30] https://github.com/internetarchive/internetarchivebot/blob/master/app/src/Core/Email/PHPMailer.php is the code handling the PHPMailer library. [13:49:27] https://github.com/internetarchive/internetarchivebot/blob/master/app/src/html/Includes/actionfunctions.php#L44 is the part of the code making the call to the Email library [13:50:21] why is joe the default editor on beta mwdeploy? [13:50:59] figuring out how to exit the commit summary screen was a whole project [13:51:26] The configuration directing the endpoint is stored under /data/project/iabot/master/app/src/deadlink.config.local.inc.php, a non-public file [13:52:19] could we just use vim or nano like everyone else? [13:52:20] Skynet: did you try this already? https://github.com/PHPMailer/PHPMailer/wiki/Troubleshooting#read-the-smtp-transcript [13:53:18] Managed to miss that. I will give that a try. :-) [14:45:17] !log toolsbeta deploy builds-api (T336225) [14:45:19] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [14:45:20] T336225: [tbs] Create an API with a single /build//logs endpoint to retrieve the build logs - https://phabricator.wikimedia.org/T336225 [15:06:28] Toolforge down again? [15:09:19] no [15:09:20] no [15:14:37] ? cannot access tools.wmflabs.org/superyetkin/index.html (re @wmtelegram_bot: no) [15:17:58] !log tools.superyetkin Kicked webservice. was not responsive to requests. [15:18:00] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.superyetkin/SAL [15:18:08] hmm, something borked for me too [15:18:22] two tools suddenly stopped responding to HTTP requests (TCP connects but no data sent back) [15:18:26] restart seems to have fixed it [15:18:42] @Yetkin: https://superyetkin.toolforge.org/ is working now. Please update your bookmarks/muscle memory to use modern URLs for your tools. [15:20:34] I've seen some goofiness after past NFS outage related Kubernetes cluster restarts too where it appears that the Kubernetes ingress layer stops passing traffic to some (but not all) webservice backends. Generally a `webservice restart` seems to fix that. Sometimes it has taken a full `webservice stop; webservice start` to reset everything. [15:20:55] ok, thanks [18:05:15] And when even doing a webservice stop and webservice start doesn't work? 🤔 [18:05:16] https://wikiconcursos.toolforge.org stopped working yesterday and even restarting, stoping and starting it doesn't returns anything other than an 503 [18:09:15] @ederporto: I'm sorry. :( It looks like the last attempt to restart it is hung now too. ContainerCreating state should not last for 15 minutes. I'll try to see if I can spot a reason. [18:10:48] Looks like https://train-blockers.toolforge.org/ needs to be kicked. [18:11:03] bd808: ^ [18:13:04] !log tools.train-blockers Hard restart after IRC notification of outage [18:13:06] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.train-blockers/SAL [18:13:15] I think it's working now dancy [18:13:22] Looks good. Thanks! [18:14:52] Please, let me know if you need me to do something on my part, thank you for looking into it :) (re @wmtelegram_bot: @ederporto: I'm sorry. :( It looks like the last attempt to restart it is hung now too. ContainerCreating state should n...) [18:15:48] !log tools.wikiconcursos Hard stop/start after reports on IRC that maintainer was having trouble getting the webservice running [18:15:49] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikiconcursos/SAL [18:16:29] @ederporto: I don't know that I did anything differently than you had, but https://wikiconcursos.toolforge.org/ seems to be alive now. [18:17:23] I did `webservice stop`, then `kubectl get all` several times to see all everything other than the cronjob removed, and finally `webservice start` [18:19:47] We have a saying in Portuguese that translates roughly to "Blacksmith's house, wooden skewer", meaning basically that someone is able to do things, but not for themselves, rsrs, maybe was that. (re @wmtelegram_bot: @ederporto: I don't know that I did anything differently than you had, but https://wikiconcursos.toolforge.org/ seems to...) [18:24:03] En casa del herrero, cuchillo de palo @ederporto :-) [22:55:26] hi, I can't delete my pod spacemedia-68c9f99ff-9pfkf. When i run the kubectl delete pod command from bastion, it just hangs and the pos remains in a "terminating" state [22:55:36] hi, I can't delete my pod spacemedia-68c9f99ff-9pfkf. When i run the kubectl delete pod command from bastion, it just hangs and the pod remains in a "terminating" state [23:00:55] it's on tools-k8s-worker-69 : https://k8s-status.toolforge.org/namespaces/tool-spacemedia/pods/spacemedia-68c9f99ff-9pfkf/ [23:05:20] * bd808 looks at @don-vip's pod issue [23:06:03] load average of 102 on tools-k8s-worker-69 might be part of the problem o_0 [23:06:51] cpu actually doesn't look too bad. there must be IO wait issues driving the load up. [23:11:34] 33 processes in "D" (uninterruptible sleep) state after 28 hours of uptime. :/ [23:17:51] !log tools kubectl cordon tools-k8s-worker-69 [23:17:54] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [23:18:13] !log tools kubectl drain --ignore-daemonsets --delete-emptydir-data --force tools-k8s-worker-69 [23:18:14] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [23:22:31] !log tools Force reboot tools-k8s-worker-69 via Horizon [23:22:31] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [23:24:44] ah, it worked, my pod is no more [23:24:54] !log tools kubectl uncordon tools-k8s-worker-69 [23:24:56] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [23:25:16] @don-vip I hit it with the biggest hammers I could find ;) [23:27:29] :) got a new pod on the same worker. I'll see if it lasts longer [23:27:40] thanks a lot, good night! :)