[10:19:37] !log superpes@tools-sgebastion-10 tools.stewardbots Restarted StewardBot not feeding on IRC and SULWatcher which had quit from IRC [10:19:41] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stewardbots/SAL [11:07:50] !log lucaswerkmeister@tools-sgebastion-10 tools.bridgebot Double IRC messages to other bridges [11:07:53] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.bridgebot/SAL [12:06:32] !log lucaswerkmeister@tools-sgebastion-10 tools.wd-shex-infer update config.yaml: increase job limits.memory from 6G to 8G, should be possible now (T357209) [12:06:36] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wd-shex-infer/SAL [16:00:15] I read your answers with logs dcaro, "the service.template has to be in the tool home, not the git repository (it's a bit of a chicken-and-egg issue if it's in the repository, we are working on a long-term solution for that) ;  yep, I meant in the remote git repo (not in the tool home), like in https://gitlab..../service.template (the repo you [16:00:16] build from), not under $HOME/*" [16:00:21] But it seems to be the opposite [16:00:45] Or my english really sucks. Cause I have the "service.template" at the root: https://gitlab.wikimedia.org/toolforge-repos/hitaden [16:01:09] And `toolforge build start https://gitlab.wikimedia.org/toolforge-repos/hitaden && toolforge webservice buildservice stop && toolforge webservice buildservice start` was failling with the mount none thing [16:01:44] now I just created a `service.template` with the same content in $HOME, and it works... [16:02:26] What did you mean with "not under $HOME/*" ? [16:02:50] And if I am not here anymore, you could still answer, I will read the logs again anyway [16:04:39] (of course, $HOME is the one when I `become` the tool) [16:43:44] Lofhi: The service.template needs to exist on the filesystem of the server where you are running the `toolforge ...` commands for it to be found by the command runner. Keeping the file in version control is a good idea, but unless that version controlled file is checked out to the $HOME of the tool on a toolforge bastion it will not be applied. [16:48:01] Alright, I see bd808. I read somewhere that you would like to automate building when pushing on gitlab.wikimedia.org. How one would update `service.template` without directly editing the file by SSHing? [16:48:20] Cause I am sure that I tried `toolforge webservice buildservice start --mount none` and it was failling too [16:48:32] We do not have a solution for that use case yet (an may never have it) [16:48:44] Alright [16:49:00] And when using it as a command? `--mount none` [16:50:11] Can you rephrase that question Lofhi? I'm not sure I understand what you are asking. [16:50:31] ehh, let me try again [16:51:38] So, I just deleted `service.template` from $HOME, and now running `toolforge build start https://gitlab.wikimedia.org/toolforge-repos/hitaden && toolforge webservice buildservice stop && toolforge webservice buildservice start`, waiting the result, but it was failling before [16:52:24] erratum for the last part: toolforge webservice buildservice start --mount none* [16:53:10] Sorry for the multi-lines message: [16:53:15] [step-results] 2024-03-02T16:52:52.711143256Z Built image tools-harbor.wmcloud.org/tool-hitaden/tool-hitaden:latest@sha256:66cf4e3ee92281b6456ba67d95c2bd755b267fb718e6aa2a412413b129868072 [16:53:15] Stopping webservice [16:53:16] WARNING: No explict backend provided. [16:53:16]   Using default of 'kubernetes' [16:53:17]   For help refer to [16:53:42] And now it works... Okay. [16:54:01] I swear, yesterday I got multiples times an error message about needing to set `--mount` [16:54:08] Even when using it :-( [16:54:29] sorry for the trouble [16:55:14] Guess I will stay with the `service.template` in $HOME, this file doesn't really need to be changed a lot. [16:56:39] The version control isn't a bad idea, it just is not sufficient to track the file inside of the repo you are using for build service image configuration. [16:57:04] If you have time, I asked yesterday this question, but didn't get any answers, maybe I need to specify it? "is it possible to call somehow Harbor within GitLab CI/CD pipeline to use heroku-builder:22? " [16:58:20] bd808 yea, but I am not sure about how updating it properly, so I guess it will stay here... [16:58:50] I think you are asking if there is a way to skip `toolforge build start ..` by using CI. Today our answer for that is no, but folks are thinking about how to make it happen. [17:00:44] You successfuly translated my question thanks for the answer :-) [17:01:08] So, to be clear, I NEED to run the commands after pushing changes? [17:01:15] that is correct [17:01:20] Alright alright [17:04:35] Lofhi: T334587 is maybe the task you would like to watch for progress on automated builds [17:04:36] T334587: [builds-api] Add triggering support - https://phabricator.wikimedia.org/T334587 [17:04:58] Meh, don't see anyway to automatically update the service.template, and creating a scheduled job for this is a bit ridiculous. Also the service can't update it alone since it needs to be updated before running the service [17:05:11] And then T341065 for automated deploy [17:05:11] T341065: [builds-api] Automatically deploy the webservice when the image is built - https://phabricator.wikimedia.org/T341065 [17:05:18] thanks [17:05:36] !log taavi@tools-sgebastion-11 tools.wikibugs toolforge jobs restart redis2irc [17:05:39] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikibugs/SAL [17:05:59] Lofhi: what needs to change regularly in your service.template? I haven't noticed that need yet myself. [17:06:17] I think I am just overengineering it, don't mind! [17:06:25] taavi: didn't it restart itself? I saw the bot rejoin [17:07:45] bd808: no, I certainly had to manually restart it a few minutes ago. it seems like it does rejoin automatically after a disconnect, but somehow doesn't manage to connect to redis and so it only sits in this channel without saying anything. [17:08:34] ah. fun times. I am hoping to deploy my pile of refactors today and put the bouncer in place [17:08:59] not sure that will fix anything, but it will make it easier for me to reason about next steps [17:10:50] Getting rid of redis for queuing is on my mind there. I'm thinking about letting the bouncer multiplex the different feed bots instead of them queuing messages in redis for as single irc bot [17:11:09] Do you have any meaningful ressources about Cloud Native Buildpacks except https://buildpacks.io/docs/ ? Since Toolforge is pushing to use it, but it's quite complexe for people discovering the concept and there are no documentation links on Wikitech. [17:12:13] Lofhi: https://devcenter.heroku.com/articles/buildpacks would be a good place to start since we are using the Heroku flavor of things today [17:13:18] Not everything that works at Heroku will work here however. We don't have a system for you to pick the stack or add new packs to the stack at the moment [17:14:11] So in classic containers, there is like Docker and Podman and there is also another alternative OCI-compliant, the Cloud Native Buildpacks pushed by Neroku? [17:14:13] Heroku* [17:15:05] Cause even when Googling-DuckDuckGoogling, I really hardly find ressources about it, except on the link I quoted and yea, the Heroku documentation one [17:15:40] Not that I don't like it, but the turn is violent [17:16:59] The general idea that became the buildpacks.io standard was born at Heroku. [17:18:56] Most of the deep details should not matter to a tool maintainer. By that I only mean that you shouldn't need to understand the entire system to make use of it for your tool's image. [17:19:16] Is there any public documentation backing the choices made by the WMF with the Build Service using Buildpacks? Not to criticize it, I just want more information about it and the why-s [17:19:54] "you shouldn't need to understand the entire system", I'm too curious! [17:20:59] Lofhi: https://wikitech.wikimedia.org/wiki/Wikimedia_Cloud_Services_team/EnhancementProposals/Toolforge_push_to_deploy and https://wikitech.wikimedia.org/wiki/Wikimedia_Cloud_Services_team/EnhancementProposals/Toolforge_Buildpack_Implementation may have some of what you are looking for [17:21:39] Noice [17:23:10] If you really want to get in the wayback machine: https://meta.wikimedia.org/wiki/Community_Tech/Tool_Labs_support/Tool_Labs_vision [17:24:39] Yay, this will satisfy my curiosity, also it'll be more simple to talk about it to others développers from frwiki if I want to motivate them to migrate their house-projects on Toolforge, even for bots. [17:25:00] Some old bots are not even using a container, so... A buildpack... [17:25:17] They'll look at me with big eyes. [17:26:06] :nod: For some of us these are old ideas, but many tools are mostly in a "this worked in 2007 on toolserver" state :) [17:27:39] way back in 2016 I was hoping we could just adopt an existing FOSS platform as a service (T136265) [17:27:40] T136265: Develop evaluation criteria for comparing Platform as a Service (PaaS) solutions - https://phabricator.wikimedia.org/T136265 [17:28:11] Example: we have a bot clearing the more obvious vandalisms... Three years ago I think, it was shutdown and we got no answers for some days from the maintainer. I tried to run their Perl bot, but never successfully set the environnement needed. [17:28:37] Also, I had the idea to move the development of wiki JavaScript gadgets on GitLab, but not sure about how and if interfaces administrators will agree [17:29:17] But GitLab could bring a lot, since a lot of gadgets will need to be updated with JQuery 4 and Codex, and the deprecating of OOUI [17:29:36] So yeah, curiosity is needed [17:29:44] I'm interested in helping folks figure out how to make better use of gitlab for managing gadgets [17:30:16] https://wikitech.wikimedia.org/wiki/User:BryanDavis/Developing_community_norms_for_critical_bots_and_tools is my essay about the general problem of tools that only one person knows how to keep running [17:32:02] I tried to think about it a bit, but the CI/CD pipeline is again the problem. The first blocking point I see is: even if we only accept merging from interface administrators on GitLab, we still need to import the changes on frwiki by example, meaning we need a bot with the interface administrator role ; it's a significant risk vector, and I'm not [17:32:03] sure I can get others to come to GitLab if they end up having to copy/paste everything by hand. [17:32:34] Also, licensing problem (we need to reference correctly the changes) [17:32:50] Will read your essay, thanks [17:33:37] I would like to see the gitlab equivalent of https://github.com/wikimedia-gadgets/deploy-action created [17:34:09] Oh, it seems to be what I said, but by assuming the risk :D [17:34:27] This can't be done with the... GitLab Runners thing, if I remember correctly the name? [17:35:15] It would be way safer that the idea I had [17:35:54] I don't want another accident with global JS loaded with malicious code [17:36:00] There are runners that live inside of Cloud VPS. They at least have IP addresses that are allowed to edit the wikis (most public clouds are IP range blocked) [17:36:35] "that are allowed" this is stable, or just "not blocked today until" [17:36:37] ? [17:37:43] If they get blocked so will 50% of edits to wikidata and about 30% of edits globally, so probably stable -- https://wmcs-edits.wmflabs.org/#wmcs-edits [17:38:12] same IP range as all toolforge hosted bots [17:38:43] Page is broken of Firefox :-( [17:38:57] Ah, refreshed, it works now [17:39:26] This is "enoughly" stable I think [17:41:57] * bd808 wanders away for a bit [17:43:01] Ok, the equivalent of Github Action is just... a CI/CD component [17:43:14] Adding it to the TODO list I will never end [17:43:22] https://docs.gitlab.com/ee/ci/components/ [17:44:06] Anyway, I would need to built something equivalent if I want to push interfaces administrators on GitLab [17:47:35] Alright, I have enough things to read for the week, and this project is still a simple idea, I have time to think about it, thanks for the answers bd808 have a nice week [18:37:21] !log bd808@tools-sgebastion-11 tools.wikibugs Comment out git pull logic in pull.php while landing and deploying refactor. [18:37:25] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikibugs/SAL [18:52:12] * bd808 will eat lunch and then roll out the big wikibugs refactor [19:20:09] !log bd808@tools-sgebastion-11 tools.wikibugs Deployed znc and service (T357729) [19:20:14] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikibugs/SAL [19:26:36] !log bd808@tools-sgebastion-11 tools.wikibugs Jobs stopped pending code base switch [19:26:39] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikibugs/SAL [19:34:25] !log tools.wikibugs Building new venv for updated codebase [19:34:28] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikibugs/SAL [20:00:29] !log bd808@tools-sgebastion-11 tools.wikibugs Debugging things. I wish this was going more smoothly. [20:00:33] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikibugs/SAL [20:32:05] ugh. the wikibugs irc bot is being very mysteriously broken. I can run it from my laptop, but when I try to run it from toolforge it never gets connected to libra.chat. Something funky is happening that I don't understand yet. [20:42:08] !log tools.wikibugs Rolled back to pre-refactor code and restarted all jobs [20:42:11] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikibugs/SAL [20:59:41] !log bd808@tools-sgebastion-10 tools.wikibugs Rebuilt venv for older codebase and restarted all jobs yet again [20:59:44] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikibugs/SAL [21:02:36] ok, old wikibugs code still works. this is actually a good thing. I can be a bit slower in figuring out what went wrong. [21:06:28] Hi everyone [21:07:20] @bd808 "I'm having a hard time debugging the python irc bot with or without the znc in the mix." -> I vaguely remember that there is a script to dump everything that goes onto the 'service bus' in redis. That at least separates any irc changes from the other changes [21:09:16] valhallasw: the busted thing is the irc client itself. It connects to libra.chat fine on my laptop, but never gets connected from the k8s cluster. I can't spot the difference in the config & startup yet. [21:10:01] But I should put back a script for peeking at the redis queue too. That will very likely be helpful at some future point of being stuck. [21:10:22] Hummmm [21:11:06] I just made a wikibugs-testing tool so I can fart around without feeling like the world is burning [21:13:17] looks like https://github.com/gawel/irc3/blob/master/irc3/plugins/log.py would be able to log all IRC level I/O, but it sounds like it might be one level lower already [21:13:53] that might be helpful actually. [21:14:39] not being familiar with the irc3 lib or anything involving asyncio is "fun" [21:17:17] Heh. Yeah, anything to do with IRC bots is complex quickly, and although I think asyncio probably is a net positive here, it sure has a steep learning curve [21:25:40] I'm pretty practiced with the python `irc` library and bot building. I'm not sure I see the need for async complexity yet, but I'm trying to give it a chance [21:30:52] I think the main reason for asyncio was needing to integrate redis with an irc bot, but the decision was made a long time ago. I'm pretty certain that "building something with this new asyncio stuff" was also a driver :D [21:33:14] updating package versions broke some of it. When the irc bot did poll redis it blew up with a "TypeError: a coroutine was expected, got None" that is certainly fallout from https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/commit/7eb17950e897d25262d5ca06b557ac008a8ba045 [21:36:11] I can't say I remember well enough exactly how the python event model works after >5 years of C# and JS/TS [21:38:54] That bit happened at https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/blob/main/src/wikibugs2/irc.py?ref_type=heads#L133 -- pretty sure there is just more syntax change needed there, just not sure what yet. First I'll try to figure out the irc server connection dilemma. :)