[01:01:02] I'm helping debug mdwiki tool, I don't know that much about the tool, but it seems like the core issue is that it tried to make a curl connection to https://mdwiki.org which times out breaking everything. I can curl this fine from the toolforge login server. however when i try to do it from within `webservice --backend=kubernetes shell` the curl request fails [It gives "Immediate connect fail for [01:01:08] 2602:fc47:1:3::a: Network is unreachable", and then just hangs forever trying to connect to 108.170.35.3] [01:01:35] I'm not sure what to make of all this. Is there some sort of firewall this is falling afoul of? Any suggestions for additional things i can try to debug this with [01:02:01] from within the webservice shell, other websites can be curled [01:35:41] hmm, i did a tracepath and it does seem to get out of the wikimedia network ( getting all the way to 14: eth.104.ar3.colo.156.phx0.phoenixnap.com (23.235.242.53) 52.747ms ) so i guess not toolforge fault, but still confusing [03:36:16] [i figured out the issue i was talking about. turns out said script was blocked for making too much traffic] [07:25:56] !log dcaro@tools-bastion-13 tools.fourohfour deploying cache_ldap_connection branch [07:25:59] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.fourohfour/SAL [12:58:18] !log bsadowski1@tools-bastion-13 tools.stewardbots Restarted StewardBot/SULWatcher because of a connection loss [12:58:21] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stewardbots/SAL [20:02:22] Rook (not urgent): just me or did the paws-public link in the UI for PAWS disappear recently? i couldn't find any relevant tickets [20:13:56] I'm in the hospital at the moment, so can't check right away. But it might not have been updated in dns after a cluster switch. I think one of those happened two weeks ago. I'll probably check later, though no promises. I may forget still coming off anesthesia [20:14:49] But it is probably just the dbs proxy setting in horizon for public was updated. andrewbogott could you double check that if you get a chance? [20:17:39] oooh sorry for bothering then -- no urgency like I said! just to be clear: public links are working, I just can't see that nice little shortcut in the PAWS UI where you can automatically open the public version of your notebook while working on it (so a lot easier to find the public URL for sharing) [20:25:11] seems like labpawspublic extension is not working. My guess is some api changed, let me tinker a bit to see if I can figure it out [20:27:17] thanks chicocvenancio! [20:30:19] chicocvenancio, please ping me if this turns out to be something that I need to fix [20:32:34] andrewbogott: how is kubectl access to these k8s clusters done? this is a frontend extension, so its not exactly blocking but access would make debugging a bit easier [20:34:51] The cluster is deployed in a batch using tofu and magnum (and I think ansible). I'm not 100% sure if you can log in and poke around but you /probably/ can [20:34:53] * andrewbogott digs a bit [20:38:34] Yeah, I don't see a direct way to execute kubectl commands in any graceful way [20:39:29] Is the public link a jupyter feature or something we've added on? [20:39:36] no problem we can figure that out later, I think I see some issues with the extension install already. Not sure I'll have access to deploy, but I might be able to get a new version built [20:39:51] yeah, its https://github.com/toolforge/labpawspublic [20:40:03] ah, I see [20:40:23] OK -- I'm stepping away for a bit but let me know if/when you need me to deploy new code. In theory I know how to do that :) [20:50:02] Chicovenancio you should be able to run deploy.sh from the bastion on the main branch, the tofu part should only generate a live config file that you can use. The ansible should be a noop but you can also skip it if you feed deploy.sh the tofu option [20:50:40] Live config = kube config [21:00:01] Thanks rook! [21:33:20] tofu for local kubeconfig, neat solution