[08:57:09] !log toolsbeta Depooling and removing worker , will pick the oldest. (T267140) - cookbook ran by dcaro@vulcanus [08:57:13] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [08:57:16] !log toolsbeta Draining node toolsbeta-test-k8s-worker-1... (T267140) - cookbook ran by dcaro@vulcanus [08:57:20] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [08:58:47] !log toolsbeta Depooling and removing worker , will pick the oldest. (T267140) - cookbook ran by dcaro@vulcanus [08:58:50] !log toolsbeta Draining node toolsbeta-test-k8s-worker-1... (T267140) - cookbook ran by dcaro@vulcanus [08:58:53] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [08:58:56] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [08:59:38] !log toolsbeta Drained node toolsbeta-test-k8s-worker-1. (T267140) - cookbook ran by dcaro@vulcanus [08:59:43] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [08:59:55] !log toolsbeta Depooled and removed worker toolsbeta-test-k8s-worker-1.toolsbeta.eqiad1.wikimedia.cloud. (T267140) - cookbook ran by dcaro@vulcanus [08:59:59] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [09:00:23] !log toolsbeta Adding a new k8s worker node - cookbook ran by dcaro@vulcanus [09:00:26] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [09:09:34] !log toolsbeta Added a new k8s worker toolsbeta-test-k8s-worker-5.toolsbeta.eqiad1.wikimedia.cloud to the worker pool - cookbook ran by dcaro@vulcanus [09:09:37] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [09:12:16] !log toolsbeta Depooling and removing worker , will pick the oldest. (T267140) - cookbook ran by dcaro@vulcanus [09:12:19] !log toolsbeta Draining node toolsbeta-test-k8s-worker-2... (T267140) - cookbook ran by dcaro@vulcanus [09:12:19] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [09:12:23] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [09:13:12] !log toolsbeta Drained node toolsbeta-test-k8s-worker-2. (T267140) - cookbook ran by dcaro@vulcanus [09:13:15] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [09:13:29] !log toolsbeta Depooled and removed worker toolsbeta-test-k8s-worker-2.toolsbeta.eqiad1.wikimedia.cloud. (T267140) - cookbook ran by dcaro@vulcanus [09:13:33] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [09:13:40] !log toolsbeta Adding a new k8s worker node - cookbook ran by dcaro@vulcanus [09:13:42] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [09:14:33] xd, the new workers are a bit heavier, exceeded the ram quota [09:18:54] !log toolsbeta Adding a new k8s worker node - cookbook ran by dcaro@vulcanus [09:18:58] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [09:27:17] !log toolsbeta Added a new k8s worker toolsbeta-test-k8s-worker-6.toolsbeta.eqiad1.wikimedia.cloud to the worker pool - cookbook ran by dcaro@vulcanus [09:27:20] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [09:27:38] !log toolsbeta Depooling and removing worker , will pick the oldest. (T267140) - cookbook ran by dcaro@vulcanus [09:27:41] !log toolsbeta Draining node toolsbeta-test-k8s-worker-3... (T267140) - cookbook ran by dcaro@vulcanus [09:27:42] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [09:27:45] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [09:28:26] !log toolsbeta Drained node toolsbeta-test-k8s-worker-3. (T267140) - cookbook ran by dcaro@vulcanus [09:28:30] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [09:28:43] !log toolsbeta Depooled and removed worker toolsbeta-test-k8s-worker-3.toolsbeta.eqiad1.wikimedia.cloud. (T267140) - cookbook ran by dcaro@vulcanus [09:28:48] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [10:24:03] !log tools running puppet on the buster bastions after 20000 minutes failing... might break something [10:24:07] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [12:00:26] Anyone in here with some help? I am deploying a flask application on http://nimble.toolforge.org/ but it returns a 404 when i try to access the application whereas the server is running [12:07:08] @eugene233 you are running a php7.3 webservice, which servers files from a `public_html` directory, but your tool does not have a directory with that name [12:07:45] I was following https://wikitech.wikimedia.org/wiki/Help:Toolforge/My_first_Flask_OAuth_tool [12:08:09] which command did you use to start the web service? [12:08:56] `webservice --backend=kubernetes --canonical python3.7 start` [12:13:32] I stopped the web service (with `webservice stop`) and started it again with that command, not sure what happened but now it looks to be using the correct type (python3.7) and uwsgi is now failing to start with "unable to find "application" callable in file /data/project/nimble/www/python/src/app.py" [12:15:29] Thanks @wm-bb Let me do another check :) [12:17:37] wm-bb is just a bridge bot between irc, telegram and mattermost :P [12:19:12] @majavah :smile [12:19:20] @majavah 😄 [15:12:23] !log tools livehacking puppetmaster for T283238 [15:12:29] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [15:12:30] T283238: Toolforge: develop jobs-framework-api - https://phabricator.wikimedia.org/T283238 [15:33:59] !log tools remove duplicate definitions from tools-clushmaster-02 /root/.ssh/known_hosts [15:34:03] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [16:17:02] !log tools deployed jobs-framework-api in the k8s cluster [16:17:06] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [17:02:55] !log starting toolforge kubernetes 1.18 upgrade - T280299 [17:02:56] majavah: Unknown project "starting" [17:02:57] T280299: Upgrade Toolforge Kubernetes to latest 1.18 - https://phabricator.wikimedia.org/T280299 [17:03:02] !log tools starting toolforge kubernetes 1.18 upgrade - T280299 [17:03:07] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [20:11:50] !log tools toolforge kubernetes upgrade complete T280299 [20:11:56] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [20:11:56] T280299: Upgrade Toolforge Kubernetes to latest 1.18 - https://phabricator.wikimedia.org/T280299 [21:36:57] Hi. [21:38:07] I find I cannot kill a job for a long time: `jstop cron-20170515.signature_check.moegirl` [21:38:29] Can anyone help me? [21:47:00] It hangs on https://sge-status.toolforge.org/#host-tools-sgeexec-0911 [21:47:36] andrewbogott: ^ [21:50:52] Well, I think tasks `refreshfeeds3` and `paper2taxon` are also hang on this server. [21:51:44] Someone with the right buttons will be around soon [21:51:56] File a task if not [21:52:08] There's something weird with this server. [21:52:47] RhinosF1 THank you :) [21:53:05] bstorm: ^ [21:53:32] ? [21:53:47] ah ok [21:53:50] bstorm: any chance you could help kanashimi [21:54:47] kanashimi: can you run qstat for me and past the output quick? It's just faster if I have job ids [21:55:07] job-ID prior name user state submit/start at queue slots ja-task-ID [21:55:07] ----------------------------------------------------------------------------------------------------------------- [21:55:08] 3363257 0.46916 cron-20170 tools.signat dRr 03/25/2021 17:49:16 continuous@tools-sgeexec-0911. 1 [21:55:08] 2297310 0.34053 cron-20170 tools.signat r 05/09/2021 05:57:40 continuous@tools-sgeexec-0936. 1 [21:55:09] 2297312 0.34053 cron-20170 tools.signat r 05/09/2021 05:58:10 continuous@tools-sgeexec-0914. 1 [21:55:09] 2297323 0.34053 cron-20170 tools.signat r 05/09/2021 05:58:55 continuous@tools-sgeexec-0919. 1 [21:55:10] 2297341 0.34053 cron-20170 tools.signat r 05/09/2021 05:59:10 continuous@tools-sgeexec-0914. 1 [21:55:10] 2297344 0.34053 cron-20170 tools.signat r 05/09/2021 05:59:40 continuous@tools-sgeexec-0912. 1 [21:55:11] 2297345 0.34053 cron-20170 tools.signat r 05/09/2021 05:59:55 continuous@tools-sgeexec-0932. 1 [21:55:17] thank you [21:55:32] looks like 3363257 is the problem [21:55:53] forced deletion of the job [21:56:31] Yes. [21:57:09] By the way, maybe jobs on other servers are also hanged? For example, `welcome_ws` on tools-sgeexec-0909 [21:57:53] bstorm Thank you. It's OK now. [21:58:28] I'm cleaning up a bunch of jobs that died for one reason or another now [21:58:48] !log tools clearing one errored queue and a stack of discarded jobs [21:58:51] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [21:59:03] I'll check if things are stuck deleting [21:59:10] OK~ [22:00:18] Nothing else seems to be right now [22:00:48] Thank you all. Have a good day! [22:00:55] 👋🏻 [22:02:37] tools.jarbot is submitting loads of jobs that are stuck in queue wait and getting dropped due to user limits. It seems to just be pushing too hard. No idea why there. [22:04:26] I'll leave that alone