[11:23:49] !log tools.growth-community-configuration Updated Growth community config repos (main=c2208e2, example=82c99a9) [11:23:52] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.growth-community-configuration/SAL [19:08:59] I have a bot running on WMCS, using Kubernetes with the Magnum images. I haven't changed the code, and it seems recently it's stopped working, and I see lots of the following in the kubectl logs on the pod: [19:08:59] /usr/local/lib/ruby/2.7.0/net/http.rb:960:in `initialize': Failed to open TCP connection to en.wikipedia.org:443 (getaddrinfo: Try again) (Faraday::ConnectionFailed) [19:09:56] I think I had the bot randomly stop working once before and rebooting the instances in Horizon fixed it, though I can't remember if it was the same issue (the pods being unable to connect to the outside world) [19:10:26] I could reboot it to test the theory but wanted to ask in here before I do that, to avoid losing any diagnostic value [19:11:20] (the project in Horizon is called `procbot` and the node's instance name is `procbot-k8s-lgbvzzr4gb6x-node-0`) [19:43:52] Restart didn't help, though tbf I can't even ping en.wikipedia.org from a regular (non-k8s) instance: [19:43:56] https://www.irccloud.com/pastebin/L7A1Spxr/ [19:50:44] that's odd.. regular HTTPS traffic still works, and it's just the pings that don't, right? [19:52:52] andrewbogott: Rook: any guesses on the network issues above? I don't seem to be able to figure out how to get a shell on a Magnum node [19:53:33] Which node are you trying to get access to? [19:53:59] the one proc mentioned, so procbot-k8s-lgbvzzr4gb6x-node-0.procbot [19:54:38] Oh I see, probably need to go in through a debug pod. Let me add myself and look [19:55:07] for the bastion (normal non-k8s instance), yeah, regular HTTPS traffic works and pings don't [19:56:18] for the k8s nodes... not sure. The error suggests it can't open a TCP connection with en.wikipedia.org:443 - I'd think this means it's having more connectivity issues than my bastion, but not sure [19:56:32] I see the issue with ping traffic, but it doesn't explain why you would be getting a complete failure on a k8s node [19:57:06] did you check if you can access anything that's not a wiki? [19:58:08] `kubectl debug node/procbot-k8s-lgbvzzr4gb6x-node-0 -it --image=ubuntu` should get you in. Then you might want `chroot /host/` after you're in the container [19:58:13] I didn't, my particular image doesn't talk to anything else. I could spin up a quick pod if I can think of an image for an app that just tries to talk to some random site [19:58:39] If you want the control node change `procbot-k8s-lgbvzzr4gb6x-node-0` to `procbot-k8s-lgbvzzr4gb6x-master-0` [19:59:14] I can ping out from the worker node [19:59:48] where do I run that from? [20:00:32] proc's user on the procbot bastion (or anywhere inside the network with the kube.config setup) [20:34:16] I just accidentally rebooted a cloudvirt. sorry [20:39:22] ah, and here come the root@ emails [20:41:42] that's my clue to step away for the weekend, I think. as far as I can tell proc's issue is some sort of a network issue within the magnum-managed kubernetes control plane. lots of the infra things in the kube-system namespace are failing to talk to the kubernetes api server. I don't seem to be able to figure out why that's happening, hopefully [20:41:42] someone with more magnum foo can find it [20:48:37] Oh it didn't have the log fix https://www.irccloud.com/pastebin/OLnMLaTj/ [20:49:04] proc: looks like it was the log issue https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Magnum#Known_issues I've added it and restarted the control node. You shouldn't have that problem any more with that cluster [21:45:01] I'm using PAWS, i try to do Do you want to select scripts setting sections? ([y]es, [a]ll, [N]one, i choose a, but then it says permission denied and folder "corn" is now locked out from me, it contains my user config n' crap [21:45:59] Guest2: what's your username? [21:46:06] is tomafahe [21:46:08] my bot is tomafahe bot [21:46:19] And it looks like some of your irc comment got cut out. What are you running? [21:46:47] i'm running pywikibot [21:46:54] i try to setup [21:46:59] is got to final step [21:47:05] says permission denied [21:47:11] now i'm locked out of user_config.py [21:47:19] how are you running pwb? Could you post a paste? [21:48:32] what paste [21:49:36] Could you paste the exact command you're running? [21:49:55] I'm not seeing a user_config.py in your pywikibot directory [21:49:59] python pwb.py generate_user_files [21:50:03] i'm running paws [21:50:14] Yeah, I'm looking in your container [21:50:18] also [21:50:19] Your default user directory is "/srv/paws" [21:50:19] user-config.py already exists in the target directory "/srv/paws". [21:50:20] Would you like to change the directory? ([Y]es, [n]o): y [21:50:21] it says this [21:50:35] i chose corn it ran until that [21:53:45] ? [21:54:05] Is corn the directory that you wanted to switch it to? [21:54:10] yeah [21:54:13] it asked me to switch it [21:57:12] Looks like it is an issue with letting pwb create the directory for you. Try removing the directory, then adding it back manually `cd ~/pywikibot; mkdir corn` then repeating the generate_user_files from before [21:57:41] When pywikibot is creating the dir itself the permissions are strange `drw---S---. 2 tools.paws tools.paws 4096 Nov 17 21:36 corn` [21:58:15] ok thx [21:58:19] i go [22:03:18] problem [22:03:20] @PAWS:~/pywikibot$ pwb login [22:03:21] WARNING: API error mwoauth-invalid-authorization-invalid-user: The authorization headers in your request are for a user that does not exist here [22:03:21] Traceback (most recent call last): [22:03:22]   File "/srv/paws/bin/pwb", line 8, in [22:03:22]     sys.exit(run()) [22:03:23]   File "/srv/paws/lib/python3.10/site-packages/pywikibot/scripts/wrapper.py", line 508, in run [22:03:23]     if not execute(): [22:03:24]   File "/srv/paws/lib/python3.10/site-packages/pywikibot/scripts/wrapper.py", line 484, in execute [22:03:24]     run_python_file(filename, script_args, module) [22:03:25]   File "/srv/paws/lib/python3.10/site-packages/pywikibot/scripts/wrapper.py", line 147, in run_python_file [22:03:25]     exec(compile(source, filename, 'exec', dont_inherit=True), [22:03:26]   File "/srv/paws/lib/python3.10/site-packages/pywikibot/scripts/login.py", line 177, in [22:03:26]     main() [22:03:27]   File "/srv/paws/lib/python3.10/site-packages/pywikibot/scripts/login.py", line 140, in main [22:03:27]     for arg in pywikibot.handle_args(args): [22:03:28]   File "/srv/paws/lib/python3.10/site-packages/pywikibot/bot.py", line 996, in handle_args [22:03:28]     pywikibot.Site() [22:03:29]   File "/srv/paws/lib/python3.10/site-packages/pywikibot/__init__.py", line 265, in Site [22:07:07] hey? [22:07:12] i got new problem [22:10:46] nvm [22:10:48] fixed it