[00:42:12] <wb-test>	 !log tools.wikibugs Updated gerrit-channels.yaml to: 87e88d59ab285c1f990579f37a1ceb7def84c5b8 gerrit: use wikibugs2.channelfilter.ChannelFilter
[00:42:16] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikibugs/SAL
[00:47:42] <wb-test>	 !log tools.wikibugs Updated channels.yaml to: 87e88d59ab285c1f990579f37a1ceb7def84c5b8 gerrit: use wikibugs2.channelfilter.ChannelFilter
[00:47:45] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikibugs/SAL
[00:55:39] <wb-test>	 !log tools.wikibugs Updated channels.yaml to: 7874c409dcef0ecc4bebc22914c540b37fd06574 gerrit: use wikibugs2.channelfilter.ChannelFilter
[00:55:45] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikibugs/SAL
[08:55:54] <wm-bot>	 !log taavi@tools-sgebastion-11 tools.wikibugs toolforge jobs restart irc
[08:55:57] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikibugs/SAL
[09:00:22] <wm-bot>	 !log lucaswerkmeister@tools-sgebastion-10 tools.bridgebot Double IRC messages to other bridges
[09:00:24] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.bridgebot/SAL
[09:25:06] <CountCount>	 I am suddenly getting 400 errors for all  Rest API calls in my spamcheck tool. E.g. for https://en.wikipedia.org/api/rest_v1/page/html/Veljko_Ra%C5%BEnatovi%C4%87/1212334420?redirect=false
[09:25:26] <CountCount>	 Is there an outage?
[09:32:49] <CountCount>	 https://phabricator.wikimedia.org/T359509
[10:16:08] <dcaro>	 nothing major https://www.wikimediastatus.net/
[10:16:27] <dcaro>	 you can try asking in -ops/-sre
[10:27:45] <Epantaleo>	 hello i have an issue with a tool that I maintain on toolforge. it's called etytree
[10:30:08] <Epantaleo>	 can anyone help me out please? i think I might only need to restart the etytree-b instance
[10:30:28] <Epantaleo>	 but I'm afraid i might mess it up if i don't do it correctly
[10:31:23] <dcaro>	 Hi Epantaleo, can you elaborate a bit more? do you have a ticket?
[10:31:28] <dcaro>	 are you trying to migrate the tool?
[10:31:32] <dcaro>	 (from gridengine to k8s)
[10:33:04] <Epantaleo>	 i do not have a ticket, but I have an event where i will show the tool and apparently it is down now
[10:34:16] <Epantaleo>	 should I create a ticket? I' am not sure what is going on but when I search for a word in the tool etytree I don't see anything and the sparql endpoint is off as well
[10:34:31] <Epantaleo>	 I will migrat hopefully next week to a new operating system
[10:34:58] <dcaro>	 I don't know any details on how the tool works, only the platform it runs on, but I can try helping debug some stuff
[10:35:12] <dcaro>	 but I'll need some information on how the tool is supposed to work :)
[10:35:14] <Epantaleo>	 http://etytree-virtuoso.wmflabs.org/sparql this is the endpoint
[10:35:18] <Epantaleo>	 sure
[10:35:20] <Epantaleo>	 I'll try
[10:35:21] <Epantaleo>	 thanks
[10:35:36] <dcaro>	 that is a cloudvps project?
[10:35:42] <Epantaleo>	 yes
[10:35:43] <dcaro>	 (a VM of it's own)
[10:35:47] <dcaro>	 okok, so not toolforge?
[10:36:11] <dcaro>	 (trying to understand the layout of the tool)
[10:36:13] <Epantaleo>	 I think it's both
[10:36:15] <Epantaleo>	 https://etytree.toolforge.org/
[10:36:24] <Epantaleo>	 because there is a graphical interface on toolforge
[10:36:35] <Epantaleo>	 and a sparql endpoint + virtuoso on the VM
[10:36:45] <dcaro>	 okok, nice, that helps
[10:37:16] <dcaro>	 and what does not work, is searching on the backend?
[10:37:33] <Epantaleo>	 for sure the sparql endpoint is not working
[10:37:52] <Epantaleo>	 that's essential so it might be that restarting the VM will fix it
[10:38:03] <Epantaleo>	 maybe someone sent a very big query
[10:38:16] <Epantaleo>	 which messed the endpoint
[10:38:39] <dcaro>	 as far as I can see, the VM that's responding to the request on etytree-virtuoso.wmflabs.org is etytree-a
[10:39:03] <dcaro>	 etytree-virtuoso.wmflabs.org 	http://172.16.0.251:8890
[10:39:06] <Epantaleo>	 oh ok
[10:40:49] <dcaro>	 there's a virtuoso instance running on that port, but has no logs on journalctl
[10:40:57] <dcaro>	 the VM seems mostly idle
[10:41:44] <Epantaleo>	 it's been down for a while, because of OS issues.  recently I asked to reinstate it
[10:41:52] <Epantaleo>	 and it worked
[10:42:12] <Epantaleo>	 two months ago
[10:42:21] <Epantaleo>	 and i just found out is not working again
[10:42:53] <Epantaleo>	 i am suppsed to migrate it to a different OS but I haven't managed to yet. Maybe that's the issue?
[10:43:11] <Epantaleo>	 *more recent OS
[10:44:48] <dcaro>	 I doubt that if it was working before it stopped suddenly
[10:44:52] <dcaro>	 we can try restarting the service
[10:45:06] <dcaro>	 (could be though)
[10:46:06] <dcaro>	 Epantaleo: should I restart the service?
[10:46:28] <dcaro>	 (systemctl restart virtuoso-opensource-7.service)
[10:47:16] <Epantaleo>	 yes please
[10:48:23] <Epantaleo>	 do you have an idea of what the issue might be? when it was restarted it worked, I checked it. And now before the meeting i checked again and it is not...
[10:48:45] <dcaro>	 done ,it seems to be working again, can yo utry?
[10:49:22] <dcaro>	 how did you restart it last time?
[10:49:34] <dcaro>	 3 months ago?
[10:50:24] <dcaro>	 cool tool btw :)
[10:52:03] <Epantaleo>	 oh great thank you so much
[10:52:06] <Epantaleo>	 so
[10:52:19] <Epantaleo>	 I used to be much more familiar with it a few years ago
[10:52:27] <Epantaleo>	 i was able to restart it by myself
[10:52:43] <Epantaleo>	 someone from this chat helped me last time and francesco negre
[10:52:51] <Epantaleo>	 negri which I met at a conference
[10:53:02] <Epantaleo>	 so i do restart it from horizon?
[10:53:11] <Epantaleo>	 just in case...
[10:53:15] <dcaro>	 for future reference, I just issued that command  `systemctl restart virtuoso-opensource-7.service`
[10:53:20] <Epantaleo>	 from where?
[10:53:23] <dcaro>	 from the VM itself
[10:53:40] <Epantaleo>	 you mean after ssh?
[10:53:57] <dcaro>	 !log etytree ran `systemctl restart virtuoso-opensource-7.service` from the VM ssh
[10:54:00] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Etytree/SAL
[10:54:07] <dcaro>	 yes, added the log just in case too :)
[10:54:35] <Epantaleo>	 thanks
[10:54:39] <Epantaleo>	 a lot!
[10:55:35] <dcaro>	 np, you should move to the new OS though ;), when you have a minute, feel free to ask for help too if you find issues (a task might help more, so we can keep track/reply at different hours/timezones)
[10:57:22] <Epantaleo>	 ok thanks
[11:26:45] <CountCount>	 In general it would be ideal if internal errors like this could be translated to 500 errors in the user-facing APIs. It is the same for other cases, e.g. https://phabricator.wikimedia.org/T350672
[11:27:02] <CountCount>	 oops, wrong channel
[12:48:59] <wm-bb>	 <Yetkin> One of my jobs is no longer being executed ton Kubernetes. Last schedule time is displayed as 2024-03-05T00:00:00Z but this job is set to run daily. What might be the issue and how to troubleshoot?
[12:56:58] <arturo>	 @Yetkin, could you share the tool name and job name?
[12:58:10] <wm-bb>	 <Yetkin> @arturo Tool name: superyetkin
[12:58:11] <wm-bb>	 <Yetkin> Job name: job-popular-pages
[13:03:32] <arturo>	 let me take a look
[13:06:11] <arturo>	 @Yetkin: the first thing I would try is to schedule the job on a different timeframe. The schedule `0 0 * * *` is usually very busy. Try `--schedule @daily`, or some other hour, like `9 23 * * *`
[13:06:32] <arturo>	 if the system fails to schedule the job because too busy, it wont be scheduled outside the timeframe
[13:09:00] <wm-bb>	 <Yetkin> @arturo Can I schedule without deleting the job first?
[13:10:28] <arturo>	 @Yetkin try `toolforge jobs restart job-popular-pages` to run the job now without waiting for the schedule time
[13:15:48] <wm-bb>	 <Yetkin> @arturo The job complated successfully. Is there any chance I can obtain the command I used to create the job? I do not want to delete the job and create it again to schedule it at a different time
[13:16:25] <arturo>	 @Yetkin: it should have been something like this:
[13:17:19] <arturo>	 `toolforge jobs run job-popular-pages --command "php ybot/wikiproject_popular_pages.php" --image "php7.4" --schedule "@daily"`
[13:17:34] <arturo>	 usually, for users with a long list of jobs like you have, we suggest storing them in a yaml file
[13:17:48] <arturo>	 see here: https://wikitech.wikimedia.org/wiki/Help:Toolforge/Jobs_framework#Loading_jobs_from_a_YAML_file
[13:18:44] <arturo>	 I'm also planning to work on a way to export the currently defined list of jobs to a YAML file: T320575 which would help in your case
[13:18:45] <stashbot>	 T320575: Allow exporting jobs list in YAML format - https://phabricator.wikimedia.org/T320575
[13:35:08] <wm-bb>	 <Yetkin> @arturo Is there any way to find out if my job has not been scheduled to run due to the busy hours you mentioned above? I have observed this by chance 😊
[13:35:57] <arturo>	 we currently don't have a way to notify about this. It would be a good thing to have though
[13:36:02] <arturo>	 let me create a phab ticket
[13:43:43] <dcaro>	 arturo: https://phabricator.wikimedia.org/T306790
[13:46:36] <arturo>	 dcaro: thanks
[15:19:45] <wm-bot>	 !log urbanecm@tools-sgebastion-10 tools.sal webservice restart # 503 Service Unavailable
[15:19:48] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.sal/SAL
[18:44:42] <wm-bot>	 !log anticomposite@tools-sgebastion-10 tools.stewardbots SULWatcher/manage.sh restart # SULWatchers disconnected
[18:44:46] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stewardbots/SAL
[20:51:44] <wm-bot>	 !log bd808@tools-sgebastion-11 tools.wikibugs Temporarily disable git webhook handler while landing and deploying changes to channel file handling.
[20:51:48] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikibugs/SAL
[21:01:14] <Rook>	 !log paws increase worker count to manage outreachy load T359591
[21:01:18] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Paws/SAL
[21:01:19] <stashbot>	 T359591: add worker to paws - https://phabricator.wikimedia.org/T359591
[21:22:04] <wm-bot>	 !log bd808@tools-sgebastion-11 tools.wikibugs Updated to c42300f (T359230, T359202, T359228)
[21:22:10] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikibugs/SAL
[21:32:00] <wm-bot>	 !log bd808@tools-sgebastion-11 tools.wikibugs Restored git webhook handler
[21:32:03] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikibugs/SAL