[07:12:52] @abartov likely to require a namespace as well (I guess is the same as python virtualenvs) [08:25:56] !log tools.phabsearchemail create toolforge-jobs venv for T305290 [08:25:58] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.phabsearchemail/SAL [08:25:59] T305290: Upgrade 'phabsearchemail' to toolforge-jobs from stretch gridengine - https://phabricator.wikimedia.org/T305290 [08:27:05] !log tools.phabsearchemail delete *.(err|out) [08:27:06] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.phabsearchemail/SAL [08:31:10] how do you send mail from toolforge-jobs [08:31:19] my current cron connects to localhost [08:31:22] but that fails [08:31:35] OSError: [Errno 99] Cannot assign requested address [08:32:30] arturo: ^ [08:33:01] RhinosF1: send emails as part of the application running inside the job? or email notifications about the job success/failure? [08:33:19] arturo: as part of the application [08:33:32] * RhinosF1 is currently doing smtp = smtplib.SMTP("localhost") [08:33:40] mmmm [08:34:49] yeah, that lib should work. Perhaps try different server [08:34:51] I use this [08:34:52] https://gerrit.wikimedia.org/r/plugins/gitiles/cloud/toolforge/jobs-framework-emailer/+/refs/heads/main/emailer/send.py#25 [08:35:31] with server being `mail.tools.wmflabs.org` and TCP port `25` [08:35:49] wm-bridgebot is mirroring everything to Telegram twice again btw [08:35:57] happened a few weeks ago too [08:37:07] I'll write this message twice on IRC, but you will see it 4 times on Telegram [08:37:08] I'll write this message twice on IRC, but you will see it 4 times on Telegram [08:37:14] arturo: thank you! that works! [08:37:24] RhinosF1: ๐ŸŽ‰ [08:39:04] !log tools.phabsearchemail toolforge-jobs run pseweekly --command "pyvenv39/bin/python3 /data/project/phabsearchemail/phabsearchemail/script.py /data/project/phabsearchemail/weekly.csv" --image tf-python39 --schedule '0 14 * * 1' [08:39:07] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.phabsearchemail/SAL [08:39:22] !log toolforge-jobs run pseweekly --command "pyvenv39/bin/python3 /data/project/phabsearchemail/phabsearchemail/script.py /data/project/phabsearchemail/weekly.csv" --image tf-python39 --schedule '0 14 * * 1' (T305290) [08:39:22] RhinosF1: Unknown project "toolforge-jobs" [08:39:22] T305290: Upgrade 'phabsearchemail' to toolforge-jobs from stretch gridengine - https://phabricator.wikimedia.org/T305290 [08:39:33] !log tools.phabsearchemail toolforge-jobs run pseweekly --command "pyvenv39/bin/python3 /data/project/phabsearchemail/phabsearchemail/script.py /data/project/phabsearchemail/weekly.csv" --image tf-python39 --schedule '0 14 * * 1' (T305290) [08:39:35] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.phabsearchemail/SAL [08:40:54] !log tools.phabsearchemail toolforge-jobs run psemonthly --command "pyvenv39/bin/python3 /data/project/phabsearchemail/phabsearchemail/script.py /data/project/phabsearchemail/monthly.csv" --image tf-python39 --schedule '0 9 1 * *' (T305290) [08:40:56] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.phabsearchemail/SAL [08:42:08] @Lucas who fixed it before? can I help? [08:43:35] !log tools.phabsearchemail stop/start webservice to clear that very old bug about restart not working [08:43:35] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.phabsearchemail/SAL [08:46:24] !log tools.phabsearchemail crontab -are (T305290) [08:46:26] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.phabsearchemail/SAL [08:46:27] T305290: Upgrade 'phabsearchemail' to toolforge-jobs from stretch gridengine - https://phabricator.wikimedia.org/T305290 [08:47:12] \o/ gridengine migrated off [08:54:09] I think it was bd808 / @bd808 who fixed the double-mirroring issue last time (re @wmtelegram_bot: @Lucas who fixed it before? can I help?) [08:54:22] don't remember how (am on phone) but it might be in a relevant SAL [08:54:28] RhinosF1: ๐ŸŽ‰ [08:57:38] arturo: thanks for the help! [09:01:59] you are welcome [09:03:36] looks like he just updated it @lucaswerkmeister (https://sal.toolforge.org/log/YIYxsn8Ba_6PSCT9kDHT) [09:28:24] !log tools deployed toolforge-jobs-framework-cli v6 into aptly and installed it on buster bastions [09:28:26] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [09:28:29] !log toolsbeta deployed toolforge-jobs-framework-cli v6 into aptly and installed it on buster bastions [09:28:30] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [11:51:46] Hello, I need assistance with the Grid Engine migration that I have been told to perform on my tool account. [12:08:39] Blahma: feel free to write here your question, or use cloud@lists.wikimedia.org [12:10:50] arturo: Thank you. The migration documentation feels overwhelming. I only have a set of jsub commands in my crontab file, such as: jsub -cwd -l h_vmem=512M myscript.sh Is it enough if I add -release buster to each jsub? [12:12:00] Blahma: likely yes, but with the release change there are also changes to the version of the runtimes [12:12:44] arturo Thank you. I'm trying them out now with that argument and so far it seems everything works. [12:12:59] Blahma: cool [12:13:08] I'm sorry to hear the migration documentation feels overwhelming :-( [12:13:20] do you have suggestions on how we could do better? [12:16:13] I indeed had a difficult time figuring out what is actually required from me. I mostly use these tools to periodically generate some output from the database replicas for fellow Wikipedians. I can understand how the maintainers are enthusiastic about a new platform etc. and I appreciate their ongoing care for us developers, but all that matters to [12:16:13] me as a maintainer of these little tools in my spare time is "what do I need to change where to keep it running". [12:16:56] I see [12:17:12] cc komla [12:17:14] So I would welcome such brief instructions like "If you only have cron jobs, try adding -relase buster and check if they still work afterwards, and that should be it." [12:18:19] Blahma: by the way, while you are at it, migrating to kubernetes should also help future-proof your tool to don't require further changes in the near term future [12:18:58] I tried to follow the manual and turn my "jsub" into "toolforge-jobs", but upon saving, I got the automatic message that "some crontab entries have been modified to grid submissions" and the result of that was a mess, a mixture of jsub and toolforge-jobs. [12:19:12] -_- [12:19:17] I'm sorry... [12:19:49] for the record, this is how to do it https://wikitech.wikimedia.org/wiki/Help:Toolforge/Jobs_framework#Creating_scheduled_jobs_(cron_jobs) [12:19:53] That's okay, I am sure it's not your fault, I rather feel sorry for other low-key developers who will not find that much patience as I and whose tools might stop working. [12:20:47] I appreciate the feedback. The process and workflows you are now "suffering" has been mostly designed by me hehe [12:20:51] I see. So I should abandon crontab and schedule my jobs with this command instead. [12:20:58] yes [12:21:15] And is that the future-proof solution you suggest? [12:22:21] yes! [12:22:43] because we would like to shutdown grid engine in the future, see https://techblog.wikimedia.org/2022/03/14/toolforge-and-grid-engine/ [12:23:45] Perfect. I think I'll play with it at some later time, now that I now how to do it. At this moment, I'll see whether I have managed to get rid off those email notifications by the extra argument and check whether my tools still work as supposed after that change. [12:23:49] Thank you for your assistance and for the ongoing maintenance of this great service! [12:24:57] ๐Ÿ‘ [12:24:58] thanks! [12:31:08] !log paws moving all VMs from paws-puppetmaster-01 -> paws-puppetmaster-2 [12:31:10] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Paws/SAL [13:00:55] !log quarry delete quarry-db-01 [13:00:57] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Quarry/SAL [16:05:36] !log tools.bridgebot Update to matterbridge v1.25.0; restart to stop duplicate telegram message mirroring issue [16:05:37] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.bridgebot/SAL [16:09:50] looks better now, thanks :) [16:56:22] !log tools build & push docker image docker-registry.tools.wmflabs.org/toolforge-jobs-framework-api:latest from https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api (cbcfc47) - cookbook ran by arturo@nostromo [16:56:24] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [16:58:31] !log toolsbeta deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api (cbcfc47) - cookbook ran by arturo@nostromo [16:58:32] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [17:05:12] !log tools deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api (cbcfc47) - cookbook ran by arturo@nostromo [17:05:14] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [18:25:21] Hello, I have some very old tools on https://toolsadmin.wikimedia.org/ that I created almost a decade ago mostly for the purposes of experimentation. I got an email from DevRel at Toolforge explaining that the environment my tool is operating on is slated to be removed. This is all well and good, but I'd actually like to delete the tool(s) [18:25:22] themselves. [18:26:52] I was able to log into https://toolsadmin.wikimedia.org/ and navigate to the tools in question (primarily https://toolsadmin.wikimedia.org/tools/id/fc-importer). However, I see no option for _deleting_ the tool anywhere in the UI -- only "manage maintainers", "create repository", "create project", and "add toolinfo". [18:29:17] I see now that https://phabricator.wikimedia.org/project/view/3975/ has the procedure, which seems...cumbersome. But very well. [18:34:51] if youโ€™re still able to SSH into Toolforge, you could also stop the webservice of the tool (if itโ€™s really not needed anymore), and that would probably disable any further emails about it running on deprecated infra [19:09:26] wm-bb I minted a new SSH keypair and confirmed that it's attached to my account in https://toolsadmin.wikimedia.org/profile/settings/ssh-keys/. What is the Toolforge SSH connection URI? [19:11:13] NVM I seem to have guess right with ssh -i [...] residentmario@primary.bastion.wmcloud.org [19:12:34] !log tools.lexeme-forms deployed 197baf2940 (l10n updates) [19:12:37] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.lexeme-forms/SAL [19:20:38] Hmm, I was able to navigate to to my Toolforge directory (/data/project/fc-importer) and I'm getting "toolsws.tool.InvalidToolException: Tool username should begin with tools." when running any nontrivial webservice command ({status, stop}) [19:23:54] ResidentMario: yes, you need to do `become toolname` before you can use webservice command [19:24:53] Nice, tyty [19:33:12] Right, I've turned the webserver off, cleaned out logs, and followed the process to file a Phabricator ticket for deletion. Hopefully that's that. Thanks for the help ya'll!