[00:00:51] <roy649>	 I'm trying to build a python venv for my new tool account (dyk-tools).
[00:01:01] <roy649>	 I'm getting:
[00:01:13] <roy649>	 The virtual environment was not created successfully because ensurepip is not
[00:01:13] <roy649>	 available.  On Debian/Ubuntu systems, you need to install the python3-venv
[00:01:13] <roy649>	 package using the following command.
[00:01:13] <roy649>	     apt-get install python3-venv
[00:01:13] <roy649>	 You may need to use sudo with that command.  After installing the python3-venv
[00:01:13] <roy649>	 package, recreate your virtual environment.
[00:01:14] <roy649>	 Failing command: ['/data/project/dyk-tools/www/python/venv/bin/python3', '-Im', 'ensurepip', '--upgrade', '--default-pip']
[00:05:59] <roy649>	 any ideas what's wrong?
[00:09:05] <roy649>	 hmm, I might have just had the wrong container type
[00:09:08] <roy649>	 nevermind :-)
[00:09:10] <wm-bb>	 <lucaswerkmeister> roy649: are you creating the venv in a `webservice shell`?
[00:09:17] <wm-bb>	 <lucaswerkmeister> heh
[00:09:31] <roy649>	 Yeah, I did "webservice --backend=kubernetes shell"
[00:09:42] <roy649>	 I thought it would default to python3.7, but apparently not
[00:09:54] <roy649>	 when I explicitly ask for python3.7, it works :-)
[00:10:30] <wm-bb>	 <lucaswerkmeister> if you don't have any files telling webservice the default type, I would guess it probably falls back to php7.4 or something like that
[00:11:01] <wm-bb>	 <lucaswerkmeister> (or one of the other types that serves public_html by default)
[00:11:12] <roy649>	 ah
[00:11:15] <wm-bb>	 <lucaswerkmeister> good that it works with the explicit type 👍
[00:11:19] <roy649>	 I see my other tool has a service.manifest.
[00:11:24] <roy649>	 I guess that's what I'm missing.
[00:11:31] <wm-bb>	 <lucaswerkmeister> yeah, sounds like it
[00:13:34] <wm-bb>	 <lucaswerkmeister> though I think .manifest is the file you're not supposed to edit manually, and service.template is the one intended for users
[00:13:55] <roy649>	 yeah, I do see it's got a comment telling you not to edit it.
[00:14:03] <wm-bb>	 <lucaswerkmeister> (can be in the home dir, or also in www/python/src/ so you can check it into source control)
[01:22:37] <andrewbogott>	 !log admin restarting designate-sink on cloudservices100[45], possible example of T316614
[01:22:41] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[01:22:42] <stashbot>	 T316614: designate-sink lock ups (was: cloudcontrol1005/nova instance creation test is CRITICAL) - https://phabricator.wikimedia.org/T316614
[05:58:20] <Rook>	 !log paws Upgrade IRkernel fd62fefdcd42ced29f30be56f82f14ed765fe780 T318275
[05:58:23] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Paws/SAL
[05:58:23] <stashbot>	 T318275: upgrade IRkernel - https://phabricator.wikimedia.org/T318275
[11:35:36] <arturo>	 !log tools aborrero@tools-k8s-control-1:~$ sudo -i kubectl -n jobs-emailer rollout restart deployment/jobs-emailer (T317998)
[11:35:39] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[11:35:39] <stashbot>	 T317998: toolforge-jobs emails not working - https://phabricator.wikimedia.org/T317998
[12:13:00] <Rook>	 !log paws T318279 Tidy R config files c220d165ea779946fa7a3ff98cf806f023e46109
[12:13:03] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Paws/SAL
[12:13:03] <stashbot>	 T318279: cleanup R files - https://phabricator.wikimedia.org/T318279
[14:01:03] <dcaro>	 !log admin test
[14:01:49] <dcaro>	 !log admin test2
[14:01:52] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[14:02:14] <arturo>	 not sure what happened with the bots
[14:02:23] <arturo>	 checked the network, but it seems just fine
[14:02:49] <dcaro>	 it seems they are back online and working 🤷
[14:43:36] <diskdance[m]>	 https://phabricator.wikimedia.org/T319155
[14:43:36] <diskdance[m]>	 Hello, I actually cannot agree with this declination. Please let me explain
[14:43:40] <diskdance[m]>	 Probably I didn't use the correct term here - my tool checks the availability of w.wiki by examining whether the browser can receive a request from the remote domain, no matter what the response is and how it is served. That is, it counts as a "failure" only you get network errors when connecting to w.wiki (this happens when connections to w.wiki are censored or interfered, detecting which is one of my tool's objective), while
[14:43:40] <diskdance[m]>	 other states including 404 are considered successful 
[16:24:11] <wm-bot>	 !log tools.stashbot <lucaswerkmeister> restarted
[16:24:13] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stashbot/SAL
[16:32:28] <danilo>	 does someone know why bots are geting down? wmopbot logs doesn't show any clue about what is happening
[16:34:39] <AmandaNP>	 Stewardsbot keeps dipping too
[16:36:44] <danilo>	 wm-bot is not getting down like other bots, probably the issue is related to something wm-bot does not use
[16:37:39] <wm-bb>	 <bozzy> Ouch I hope I have not destroyed the Striker internal consistence, when I tried to create a Diffusion repo, but it created a GitLab repo, and then I tried to fix deleting the GitLab repo
[16:37:41] <wm-bb>	 <bozzy> https://phabricator.wikimedia.org/T320438
[16:38:26] <Reedy>	 FWIW, no one else can view your attachment images
[16:38:37] <wm-bb>	 <bozzy> :O
[16:38:45] <Reedy>	 It's phab weirdness from a little while ago
[16:38:56] <Reedy>	 If you click through, click edit, and make them viewable to anyone, should fix it
[16:39:11] <wm-bot>	 !log tools.stewardbots <deltaquad> restart StewardBot
[16:39:13] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stewardbots/SAL
[16:39:29] <wm-bb>	 <bozzy> Yep yep, try now
[16:39:39] <wm-bb>	 <bozzy> In my private Phabricator it does not happen
[16:40:10] <taavi>	 hmm.. wm-bot is on its own vps project I think, so this sounds like a toolforge specific issue?
[16:40:35] <taavi>	 looking but no promises yet
[16:40:35] <Reedy>	 I'm guessing that bug is just an i18n message that wasn't updated properly
[16:40:44] <wm-bb>	 <bozzy> Probably
[16:40:46] <AmandaNP>	 could be, stewardsbot is on tools
[16:47:49] <taavi>	 !log tools clean up labstore1006/7 mounts from k8s control nodes T320425
[16:48:14] <taavi>	 let's see if that helps. in theory it should stabilize the k8s network components
[16:51:48] <taavi>	 !log tools clean up labstore1006/7 mounts from k8s control nodes T320425
[16:51:51] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[16:51:51] <stashbot>	 T320425: [cloudvps] Find and cleanup any mounts to labstore1006/1007 - https://phabricator.wikimedia.org/T320425
[16:52:01] <taavi>	 ping me if you see any more disconnections later today please?
[16:54:57] <taavi>	 danilo: is wmopbot manually down or is it still having troubles?
[17:01:58] <Rook>	 !log paws Bump pywikibot to 7.7.1 24d0e2223175024bd11465decbf0a1087c1cc2c3 T320432
[17:02:01] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Paws/SAL
[17:02:01] <stashbot>	 T320432: New upstream stable release for Pywikibot 7.7.1 - https://phabricator.wikimedia.org/T320432
[17:19:24] <danilo>	 taavi: wmopbot auto restarted, thank you
[17:57:19] <MacFan4000>	 taavi:^ seems like disconnections are still hapening
[18:37:39] <wm-bot>	 !log tools.stewardbots <anticomposite> ./stewardbots/SULWatcher/manage.sh restart # All bots disconnected
[18:37:40] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stewardbots/SAL
[18:57:18] <danilo>	 I am not finding anything in wmopbot logs that explain why it is getting down, I only found in "kubectl describe pod <podname>" Container: Last State: Terminated (Reason: Error; Exit code: 1), so it is not being killed, it seems something that makes the process hang till be disconnected by IRC ping timeout
[19:02:16] <taavi>	 sigh. looks like NFS being annoying as usual.
[19:30:07] <taavi>	 !log tools rebooting all k8s worker nodes to clean up labstore1006/7 remains
[19:30:08] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[19:56:21] <Sariboo>	 taavi: another wmopbot quit ^^
[20:01:18] <wm-bot>	 !log tools.stewardbots <anticomposite> Restart all bots
[20:01:20] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stewardbots/SAL
[20:03:38] <AntiComposite>	 Error from server: error when creating "/data/project/stewardbots/stewardbots/StewardBot/k8s-deployment.yaml": admission webhook "registry-admission.tools.wmflabs.org" denied the request: The following container images did not match any of the allowed registries ([['docker-registry.tools.wmflabs.org']]): [Kind=apps/v1, Kind=Deployment, Namespace=tool-stewardbots Name=stewardbot Image=docker-registry.tools.wmflabs.org/toolf
[20:03:38] <AntiComposite>	 orge-python39-sssd-base:latest]
[20:03:43] <AntiComposite>	 well that's new
[20:04:27] <TheresNoTime>	 T320446
[20:04:28] <stashbot>	 T320446: The following container images did not match any of the allowed registries ([['docker-registry.tools.wmflabs.org']]) - https://phabricator.wikimedia.org/T320446
[20:04:35] <TheresNoTime>	 AntiComposite: I've *just* seen that too!
[20:05:46] <AntiComposite>	 ubn?
[20:06:13] <TheresNoTime>	 I'm getting toolforge job failures across my tools now too :/
[20:08:13] <danilo>	 and wmopbot pods were killed and deleted, different to what happened earlier
[20:08:20] <AntiComposite>	 yeah, wasn't a timeout, was (Remote host closed the connection)
[20:25:30] <JJMC89>	 taavi: k8s issues ^
[20:25:53] <taavi>	 JJMC89: aware, being looked at
[20:26:48] <DENelson83>	 Seems all toolforge services are down.
[20:27:16] <Sariboo>	 DENelson83: yup, T320446
[20:27:20] <taavi>	 yes, looking into it
[20:27:35] <DENelson83>	 What is "T320446"?  No link.
[20:27:44] <TheresNoTime>	 https://phabricator.wikimedia.org/T320446
[20:28:42] <DENelson83>	 Okay.  I will monitor that for more info.
[20:29:08] <wm-bb>	 <lucaswerkmeister> is something up with toolforge?
[20:29:10] <wm-bb>	 <lucaswerkmeister> $ kubectl rollout restart deployment lexeme-forms
[20:29:12] <wm-bb>	 <lucaswerkmeister> error: failed to patch: admission webhook "registry-admission.tools.wmflabs.org" denied the request: The following container images did not match any of the allowed registries ([['docker-registry.tools.wmflabs.org']]): [Kind=apps/v1, Kind=Deployment, Namespace=tool-lexeme-forms Name=lexeme-forms Image=docker-registry.tools.wmflabs.org/toolforge-python39-sssd-web:latest]
[20:29:20] <wm-bb>	 <lucaswerkmeister> (and the tool is currently down, uWSGI killed itself following a SIGINT/SIGQUIT that I have no explanation for)
[20:29:23] <RhinosF1>	 Read a few messages up
[20:29:28] <wm-bb>	 <lucaswerkmeister> (the log doesn’t say which signal it was, helpfully)
[20:29:30] <wm-bb>	 <lucaswerkmeister> `kubectl get events` shows several copies of that “denied the request” message, k8s has been trying to recreate the container for about half an hour it seems
[20:29:32] <wm-bb>	 <lucaswerkmeister> s/container/pod/
[20:29:34] <wm-bb>	 <lucaswerkmeister> I’ll try recreating the deployment from scratch
[20:29:36] <wm-bb>	 <lucaswerkmeister> ugh, and now there’s a pod that’s terminating, so apparently it managed to create one after all?
[20:29:47] <arturo>	 the bridgebot is catching up
[20:29:53] <taavi>	 I believe everything is starting to recover now
[20:29:53] <RhinosF1>	 Oh
[20:30:01] <DENelson83>	 Oh... Looks like it's working again.
[20:30:14] <DENelson83>	 Only like ten minutes of downtime.
[20:30:33] <wm-bb>	 <lucaswerkmeister> ah. did I miss something over here
[20:30:35] <wm-bb>	 <lucaswerkmeister> I suppose I should’ve expected that :|
[20:30:37] <wm-bb>	 <lucaswerkmeister> ok, sorry for the noise :(
[20:30:50] <wm-bb>	 <Samuel> Your message only just came to IRC @lucaswerkmeister
[20:30:58] <wm-bb>	 <Samuel> But yes there was an issue
[20:31:07] <wm-bb>	 <lucaswerkmeister> I’m looking at the IRC logs now
[20:31:31] <taavi>	 sorry folks :/ will follow up on the phabricator task when it's not so late
[20:36:35] <urbanecm>	 no worries, thanks for putting it under control!
[20:37:38] <urbanecm>	 TheresNoTime or others: can someone (try to) restart wikibugs (https://www.mediawiki.org/wiki/Wikibugs)?
[20:37:55] <TheresNoTime>	 urbanecm: lookin'
[20:38:38] <taavi>	 on the bright side, I hope the issues with irc bots disconnecting has been solved by now
[20:40:50] <TheresNoTime>	 urbanecm: done :)
[20:41:03] <urbanecm>	 thank you kindly
[20:41:47] <urbanecm>	 TheresNoTime: can i bother you to restart stashbot as well? docs: https://wikitech.wikimedia.org/wiki/Tool:Stashbot#Maintenance
[20:42:01] <TheresNoTime>	 sure :)
[20:42:17] <TheresNoTime>	 whats the cli SAL log command again?
[20:42:24] <TheresNoTime>	 thought it was like `logmsg` or something
[20:42:37] <taavi>	 dologmsg
[20:42:52] <TheresNoTime>	 thanks :)
[20:43:29] <wm-bot>	 !log tools.wikibugs <root> restart wikibugs
[20:43:32] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikibugs/SAL
[20:43:58] <urbanecm>	 seems it came back
[20:44:11] <wm-bot>	 !log tools.stashbot <root> restart stashbot
[20:44:12] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stashbot/SAL
[20:58:20] <urbanecm>	 and...bots are down again :/
[21:04:22] <TheresNoTime>	 hmm
[21:05:05] <wm-bb>	 <jeremy_b> bridge is doubling IRC on telegram
[21:08:50] <wm-bot>	 !log tools.wikibugs <root> restart wikibugs
[21:08:52] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikibugs/SAL
[21:09:03] <wm-bot>	 !log tools.stashbot <root> restart stashbot
[21:09:04] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stashbot/SAL
[21:26:43] <wm-bot>	 !log tools.bridgebot <lucaswerkmeister> Double IRC messages to other bridges
[21:26:45] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.bridgebot/SAL
[23:03:07] <wm-bot>	 !log tools.stewardbots <anticomposite> ./stewardbots/StewardBot/manage.sh restart # Ping timeout not noticed by the bot
[23:03:08] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stewardbots/SAL
[23:04:13] <wm-bot>	 !log tools.bridgebot <lucaswerkmeister> Double IRC messages to other bridges
[23:04:14] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.bridgebot/SAL
[23:04:24] <wm-bot>	 !log tools.stewardbots <anticomposite> ./SULWatcher/manage.sh restert # Ping timeout, all bots disconnected
[23:04:25] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stewardbots/SAL
[23:05:45] <AntiComposite>	 always helps when stashbot doesn't go down