[00:28:32] My attempts to access login.toolforge.org keep timing out [00:32:34] GrayStorm: there was an epic debugging session on ssh attempt timeouts here yesterday and the ultimate reason was the user typoing the hostname (.com instead of .org). Assuming that's not your issue, https://wikitech-static.wikimedia.org/wiki/Reporting_a_connectivity_issue has some ideas on how you can collect information on the network between you and us. [00:33:19] That page is written from the point of view of debugging HTTPS access to the wikis, but things are very similar to debug SSH access to a toolforge bastion. [00:36:04] You might also get some inspiration from the logs of the last person who was having trouble and the debugging steps they took -- https://wm-bot.wmcloud.org/browser/index.php?start=02/14/2024&end=02/15/2024&display=%23wikimedia-cloud [00:37:21] and now I see they parted. meh [01:22:09] bd808 it was interesting, not sure I would call it "epic". [01:22:22] It's not epic until tcpdump enters the room :-) [01:53:53] hah epic [03:02:57] bd808: thanks for volunteering, I added you [03:03:34] probably better than me trying to rewrite it in rust ;) [04:23:42] tools.apersonbot@tools-sgebastion-10:~$ toolforge jobs load jobs.yaml [04:23:42] /usr/bin/toolforge-jobs:15: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html [04:23:42] from pkg_resources import load_entry_point [04:23:44] o.O [04:25:17] ah, https://phabricator.wikimedia.org/T357387 [06:51:11] If someone can resize and rescue the db instance in https://phabricator.wikimedia.org/T355138 I'd be greatful, sorry for the trouble. [09:45:31] !log tools.admin restarted webservice as it was giving 500 errors, seems back online [09:45:34] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.admin/SAL [10:38:23] Random, unimportant question: What is the most efficient way to make a tool works as a redirect to another tool? I made it with 3 lines of PHP, but I wonder if there is a simplier (nginx?) method. [10:41:07] it is possible to do that on the toolforge ingress layer but it requires manually creating the required kubernetes ingress object. https://wikitech.wikimedia.org/wiki/User:BryanDavis/Kubernetes#Make_a_tool_redirect_to_another_tool_WITHOUT_running_a_webservice [10:43:42] Interesting... I'll have a look on that. Thanks! =D [11:03:21] note that the 3 lines of php/python/etc. are preferred (any custom k8s setting you might set, might break unexpectedly on any upgrade/etc.) [11:03:32] (though might be slightly less efficient) [11:13:28] !log tools.hay restarted the webservice as it was hung [11:13:31] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.hay/SAL [11:13:52] !log tools.admin restarted hay tool webservice and now it's back working :/ [11:13:54] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.admin/SAL [11:16:39] !log taavi@tools-sgebastion-11 tools.wikibugs toolforge jobs restart redis2irc [11:16:41] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikibugs/SAL [12:10:42] taavi: did the wikibugs restart work? I don’t see any news from it in -dev or -operations [12:11:21] seemingly it disconnected but never came back [12:12:30] ok now it's back it seems [12:12:42] hm, ok [12:12:44] !log taavi@tools-sgebastion-11 tools.wikibugs toolforge jobs restart redis2irc [12:12:48] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikibugs/SAL [13:32:21] !log superpes@tools-sgebastion-10 tools.stewardbots Restarted StewardBot not feeding on IRC and SULWatcher which had quit from IRC [13:32:24] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stewardbots/SAL