[00:02:09] 00:01 UTC and I just saw my first dumb April 1st joke site 😭 [00:02:43] * bd808 is not a fan of the long day of "who is lying on the internet" [00:03:51] I heard someone is pranking people by pretending a compression library is being used to backdoor a lot of stuff [00:07:33] too soon @yuvipanda, too soon ;) [00:14:49] Telegram doesn't allow the upside down smiley :( (re @yuvipanda: I heard someone is pranking people by pretending a compression library is being used to backdoor a lot of stuff) [01:13:29] Now that Redis has been un-open-sourced, what's a good future proof alternative to celery that I can use ? (I'm planning to use some kind of worker in a few more projects) [01:13:44] For projects on Toolforge that is [01:26:57] I think Elasticsearch got stuck on the last version with apache license at toolforge, so I see the same outcome with Redis (re @sohom_datta: Now that Redis has been un-open-sourced, what's a good future proof alternative to celery that I can use ? (I'm planning to use ...) [01:43:20] @sohom_datta you can watch https://phabricator.wikimedia.org/T360596 and https://phabricator.wikimedia.org/T360378. The redis container task is likely to move forward sooner than the bigger open question one unless I lose interest after moving the wikibugs work queue off redis. [02:12:45] !log anticomposite@tools-sgebastion-10 tools.stewardbots SULWatcher/manage.sh restart # SULWatchers disconnected [02:12:48] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stewardbots/SAL [02:17:45] !log anticomposite@tools-sgebastion-10 tools.stewardbots ./stewardbots/StewardBot/manage.sh restart # RC reader not reading RC [02:17:47] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stewardbots/SAL [10:54:57] Hey there! A newbie question: In my tool account I see both "pyvenv" and "venv" directories. Do they serve the same purpose and I can safely delete the "pyvenv" one? [11:01:03] what’s the name of the tool? [11:04:32] smallem [11:06:55] I was managing the files there after the migration to Kubernetes. Given that there were a lot of trial and error I followed different kinds of instructions and created different stuff until I was able to get all my scripts back to work. Now I wanted to know what I need from those stuff and what I don't. [11:08:54] GPT tells me that pyvenv is just an obsolete version of venv so I thought to ask. [11:09:14] please ignore GPT’s advice [11:09:30] (in this case in particular, but imho also in general, really) [11:10:08] “pyvenv is just an obsolete version of venv” is a somewhat accurate statement (AFAIK) when talking about *libraries*, but that has nothing to do with the *directories* in your tool account [11:11:48] I’m looking at the tool now… the webservice uses php7.4, so that’s definitely not using any kind of python venv [11:11:59] and then there are five toolforge jobs / cronjobs [11:12:50] Hi, it seems that wiki bugs has stopped reporting gerrit changes (and maybe tasks as well) since around 10 this morning (BST) [11:13:44] !log lucaswerkmeister@tools-sgebastion-10 tools.wikibugs toolforge jobs restart irc [11:13:47] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikibugs/SAL [11:14:45] `smallem-transclusion.sh` and `smallem-transclusion-la.sh` use `pyvenv/`, according to the commands they’re running [11:15:19] and `smallem-wp.sh` and `smallem-wq.sh` use `pwbvenv/` according to their shell scripts [11:15:43] I don’t immediately see anything that uses `venv/` [11:16:27] (paladox: better now?) [11:16:38] oh, yes. Thanks! [11:16:42] yay [11:16:50] In GPT's defense, my question had no context whatsoever so that might have been a factor. [11:16:51] [11:16:53] I asked it a very open, simple question to get the most information: [11:16:54] [11:16:56] What is pwbvenv, pyvenv, pywikibot-core and venv? [11:16:57] [11:16:59] And then I thought pwbvenv and pywikibot-core are needed by smallem-wp/sw so that left only those two. [11:17:53] Now you mention the other two scripts and you're right so I assume everything is needed (since I don't wanna go and delete venv since it might be useful in the future). [11:19:16] I’m still looking for things that might use `venv/`, haven’t found any so far [11:20:11] You also mention the webservice which is the only thing I haven't dealt with since the migration happened and reminds me that I might need to ask for some more help in the future when/if I want to get my webservice running again (assuming it's not - haven't checked since it's a less important part of my job, just a presentation page of its jobs basically). [11:21:14] something’s definitely running at https://smallem.toolforge.org/ [11:21:20] anyway, I don’t see anything that uses `venv/` [11:21:38] I’d suggest renaming it to something else, to reduce the number of confusing venvs in your tool directory ^^ [11:21:43] like `venv-probably-unused` or whatever [11:21:50] and if nothing breaks, remove it after a month or two [11:21:59] Oh, it is running! Thank you very much for the help! [11:22:25] Yes, I've done that thing with two scripts just today (backup-....) [11:22:59] I deleted them and I'm waiting to see if nothing will break so I can also delete their backups. [11:23:53] Thank you one more time! :)) [11:24:18] np, good luck :) [11:41:55] Whenever I use a Toolforge command, such as "toolforge jobs list" I get a deprecation warning about pkg_resources. [11:41:56] [11:41:57] Do I need to do something about it? [11:48:23] probably not [11:48:34] *looks for phabricator tasks* [11:48:34] !log bsadowski1@tools-sgebastion-10 tools.stewardbots Restarted StewardBot/SULWatcher because it had a connection loss [11:48:37] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stewardbots/SAL [11:49:02] hm, from https://phabricator.wikimedia.org/T357387 it sounds like it might be specific to your tool after all [11:49:32] yeah, you also have a `PYTHONPATH=` in your bash profile [11:50:27] you can probably still ignore the warning for now and wait for `/data/project/shared/pywikibot/stable/` to be updated so the warning disappears [11:51:14] I wonder if I need that in my bash profile though. [11:56:09] good question, I don’t know [12:00:30] It was suggested to me quite some years ago by Bryan Davis when I just got my Toolforge account and didn't know much about the whole thing. I'm guessing I'll just "oh well" it and leave it like it is currently. Thank you! :) [12:48:21] !log quarry quarry moved to k8s T349032 [12:48:25] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Quarry/SAL [12:48:25] T349032: Deploy magnum cluster for quarry - https://phabricator.wikimedia.org/T349032 [14:57:46] @Klein: Setting PYTHONPATH to point to the pywikibot/stable directories was part of the recommended pywikibot configuration on the grid engine. In the current Toolforge era it is a legacy configuration. See https://wikitech.wikimedia.org/wiki/Help:Toolforge/Pywikibot for links to more current pywikibot practices. [14:59:39] I was suspecting that. Well, I can safely remove it then I suppose. Thank you! :)) [15:00:04] I don't know what the communities plans are for the `/data/project/shared/pywikibot` directories, but I don't think they are readily usable in the current era. No bastion or container image comes pre-configured to work with that git clone as far as I know. [15:00:16] *community's [15:26:17] Is there a way to keep some files on a tool account synchronised with some local files on my machine? [15:26:17] [15:26:18] The scenario I'm picturing is that once I SSH into the server, every change I save, either on the server or on the local copy (of those files), gets replicated in the other copy? [15:28:16] I use Syncthing to do the same for two machines and I assume theoretically I can do the same with a server and my local machine. [15:28:17] [15:28:18] Is that possible? Any other simpler ways? [15:33:57] @Klein: the ideal solution in my mind is a git repo on gitlab.wikimedia.org that you can push to and pull from from any internet connected device. https://wikitech.wikimedia.org/wiki/Help:Toolforge/Version_Control_in_Toolforge [15:36:26] the sort of by-directional rsync solution you are describing can be tricky when there are more than tow copies. Since we ideally would like to see every tool have >1 maintainers something like git can be a bit more robust to multiple copies and reconciling edit conflicts when they arise. [15:36:36] *more than two [15:51:21] Yes, I read around and those were the only two proposed solutions: Either a git repo or a rsync script. Rsync looked more aligned with the bi-directional aspect I had in mind where you don't have a master-slave type organisation. Syncthing looks even easier than that. Would it be possible to theoretically use it on the tool account? Or would there be privilege problems with that approach? [15:54:26] Just asking because that's what I'm more familiar with currently. I would need to "start from scratch" with the gitlab approach. [15:55:04] @Klein: I haven't used Syncthing, but it seems to require running a dedicated service on each node in its network. It also looks like a single tenant solution. It is not obvious at a quick glance how it could be made to work for everyone on a bastion. [16:02:08] Yes, that was the original question: Can tool accounts make use of such technology individually if they so wish? But I'm understanding that might bring "standardisation problems" in the general sense. I'll start reading more about the gitlab approach. If I decide to go through with it when I feel confident enough, maybe I ask for help here again. Thanks! [16:10:47] @Klein: the general problem with Syncthing as I understand it currently is that you would need a dedicated process running in Toolforge just for your tool and that process is not webservice. This means you would need CPU and RAM to run the code and to expose its data port on a public IP address so your local client could attach to it. [16:11:23] The CPU and RAM would need to come from the Kubernetes cluster; we can't afford to have long running processes for each tool on a shared bastion. And then you would need custom egress to expose a non-http service to the internet. [16:12:05] It probably could all be made to work with some thought and effort, but it will not be trivial if I'm understanding the system correctly. [16:12:38] Learning to use git would provide other advantages in the long run [16:36:11] I understand. I thought maybe "I'd get lucky" and a Syncthing infrastructure was already in place since it was an open source project but I do understand the lack of resources for such a thing. Thank you for the thorough explanation! Appreciated. I'm already reading about gitlab. [16:57:08] !log library-upgrader stop upgrader cron on upgrader-06 to prepare for move to libup-runner08 in T361488 [16:57:11] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Library-upgrader/SAL [16:57:11] T361488: Upgrade LibUp worker to Bookworm - https://phabricator.wikimedia.org/T361488 [17:12:18] !log packaging - puppet is broken on builder-envoy-03 because 'profile::docker::builder::imageupdate_git_key' cant't be found but it is in labs/private since 2023 - what changed? [17:12:21] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Packaging/SAL [17:14:20] !log packaging - added 'profile::docker::builder::imageupdate_git_key: "12345" [17:14:21] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Packaging/SAL [17:16:01] !log packaging - added https://horizon.wikimedia.org/project/puppet/ in project hiera in Horizon as a test - it did fix the puppet runs on builder-envoy-03 - which had been mailing us daily since February 2nd [17:16:02] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Packaging/SAL [17:23:54] !log wikistats - deleting instance bwplanet [17:23:57] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikistats/SAL [17:25:19] !log devtools - attempting to fix puppet on instance etherpad-bookworm but SSL provider cfssl doesn't appear to work in cloud [17:25:20] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Devtools/SAL [17:27:12] !log devtools - added profile::pki::client::ensure: present to instance hiera for etherpad-bookworm - fixing broken puppet run [17:27:13] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Devtools/SAL [18:12:01] * bd808 has a proof of concept of replacing Redis in wikibugs [18:13:02] I'm messing about with https://en.wikipedia.org/wiki/Server-sent_events as a replacement [18:13:06] !log h2o@tools-sgebastion-10 tools.stewardbots ./stewardbots/StewardBot/manage.sh restart # RC reader not reading RC [18:13:09] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stewardbots/SAL [20:41:56] what is the maximum allowed quota on bastions? [20:48:45] quota for what? [20:50:17] There are limits on the number of CPS and threads per maintainer per bastion. I don't remember off the top of my head what they are. They are not currently something that can be changed per user. [20:50:23] *CPUs [20:50:45] https://wikitech.wikimedia.org/wiki/Help:Toolforge/Kubernetes#%22failed_to_create_new_OS_thread%22_from_kubectl is related to those limits [20:52:17] bd808: I meant disk quota [20:54:19] There isn't anything that enforces a strict limit on the NFS share for Toolforge. We periodically make tasks like T358554 and T357882 when the NFS server is being overwhelmed. [20:54:20] T358554: 2024-02-27: toolforge NFS cleanup - https://phabricator.wikimedia.org/T358554 [20:54:20] T357882: 2024-02-19: toolforge NFS cleanup - https://phabricator.wikimedia.org/T357882 [20:56:00] @Yetkin: questions like what do you want to store, how big do you think it will grow, how long do you need to keep it come to mind in response to your question. :) [20:57:49] In addition to the shared NFS storage within Toolforge there are other potential solutions like https://wikitech.wikimedia.org/wiki/Help:Object_storage_user_guide [21:02:32] Broadly speaking, nobody is likely to notice if you store a few GiB of files on NFS. When you get into 50GiB and higher range then you might be noticed and asked to find ways to reduce your NFS usage. [21:05:05] bd808: Thanks for the answer. The reason why I am asking is that I am currently uploading some XML files (5-10 GB in total) that I will process on Toolforge [21:06:13] that sounds like it will be fine, especially if you clean up the files after you use them [21:10:27] !log bd808@tools-bastion-12 tools.wikibugs Restart Phorge task to pick up lower backoff limit from eabd23c9 [21:10:31] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikibugs/SAL