[01:28:29] I have some SGE jobs that are zombied qdel doesn't work. How might they be killed? [01:28:31] 1395686 0.26854 frwiktiona tools.botwik Rr 04/03/2023 13:57:10 continuous@tools-sgeexec-10-19 1 [01:28:32] 1395687 0.26854 enwiktiona tools.botwik Rr 04/03/2023 13:56:56 continuous@tools-sgeexec-10-17 1 [01:28:32] 1462071 0.26545 en.arcstat tools.botwik Rr 04/03/2023 13:57:25 continuous@tools-sgeexec-10-19 1 [01:28:33] 1465509 0.26529 en.pgcount tools.botwik dRr 04/03/2023 13:57:10 continuous@tools-sgeexec-10-20 1 [01:29:21] They went zombie during the recent Toolforge maintenance outage [01:33:32] one of the roots will need to delete them for you. not sure who's around right now [05:23:45] ALPS [05:23:47] SPSS [05:46:46] hi GreenC [05:48:45] !log tools.botwikiawk manually killed stuck jobs per request from GreenC [05:48:48] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.botwikiawk/SAL [05:58:21] 1465509 need a qdel -f, but those 4 are all gone now [06:40:19] ALPS [06:40:22] SPSS [06:40:52] wut [06:47:29] !log tools.coverage rebuilt venv, now working again [06:47:32] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.coverage/SAL [10:23:32] dcaro: I think I have to back out of the karma upgrade, I just noticed support for AM < 0.22 has been removed (we run 0.21) [10:23:54] we can upgrade once alert hosts are on bookworm though [10:25:10] dcaro: as a middle ground we can apply your patch on top of 0.99 if that's easy/possible ? [10:54:18] {{done}} (reset to 0.99 commit) [11:03:44] godog: ack, the patch might not be compatible, will have to check [11:10:00] dcaro: ok! [15:46:49] !log tools upload toolforge-jobs-framework-cli v11 to aptly [15:46:53] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [16:12:08] TheresNoTime, can we get some attention on T328691? As far as I know linkwatcher is currently down but there's been no response on that ticket in ages. [16:12:09] T328691: [toolsdb] Migrate linkwatcher db to Trove - https://phabricator.wikimedia.org/T328691 [16:21:10] andrewbogott: I'll take a look at where we are [16:23:15] Oh, I remember — it's quite difficult to get hold of the primary maintainer, Perl is not a language I have any competence in and afaics it generally needs some rewriting.. [16:24:24] TheresNoTime: at the moment I think the patch needed is a simple search and replace for the new fqdn. [16:24:42] Happy to poke the right things in regards to helping y'all migrate though, yeah [16:25:17] thx [16:29:01] !log tools.disabled-tools Update to eb64570 [16:29:04] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.disabled-tools/SAL [18:03:26] I know this is is wmflabs.org and probably old tool.. dont even know why I had it open in a tab anymore.. but the error message still was a bit unexpected.. not working or down but "try again later" "under heavy load" [18:03:30] https://outreachdashboard.wmflabs.org/ [18:12:20] interesting, I think someone complained about it being broken a bit ago but with a different error message [18:12:53] (bit being measured in weeks here) [18:40:31] mutante: that's maybe a question for ragesoss... He's at least the person I think of most when I think of the https://openstack-browser.toolforge.org/project/globaleducation project that owns that proxy. [18:41:56] bd808: ah, different proxy! got it. ok, thank you [18:43:57] yup. that error is from Apache, usually because enough slow DB queries are happening that Rails has saturated its max number of threads. [18:44:06] I will kick the server. [18:44:53] ah, you are here as well:) sounds good [18:45:35] "under heavy load" was unusual to me, that made me report it. cheers [18:46:00] nginx -> apache -> rails -> boom :) [18:57:55] mutante: for the next one of these that you run into outside of Toolforge, there is a report at https://openstack-browser.toolforge.org/proxy/ that you can use to figure out which Cloud VPS project currently owns a proxy host like outreachdashboard.wmflabs.org. [19:01:28] the database seems to be performing very badly at the moment... dashboard is back on line, but queries are taking much longer than they should. [19:01:46] and when i try SSH into the database server, it just hangs [19:02:57] bd808: ah, thanks I use openstack-browser but for role classes and did not occur to me for proxies! [19:58:20] hello! is it okay to create a new web proxy for xtools.wmcloud.org while xtools.wmflabs.org still exists (and is a different VM), or is that going to cause problems with the auto-redirect for the old domain? i.e. I'm hoping xtools.wmflabs.org won't redirect until I delete the old web proxy [19:59:32] musikanimal: it should be ok to have both and have them point to different backends [19:59:39] musikanimal: yeah, that's fine [19:59:43] okay, thanks! [20:00:38] another unrelated question: is https://tools-static.wmflabs.org going to stay at wmflabs.org? I noticed wmcloud.org or toolforge.org doesn't work [20:07:20] is using a proxy on wmcloud.org [20:07:59] misunderstood, nvm [20:15:03] musikanimal: nobody has poked at it is the quick answer. When we do add a new hostname there I think it would be .wmcloud.org just because it would be easier than trying to make it a *.toolforge.org hostname. [20:15:32] * bd808 last thought about static when Brooke was still here... [20:16:28] okay, I was just curious :) I've been basically doing searches in my code for wmflabs.org and updating them, and the assets pulled from tools-static are the only things left [20:16:41] sad to hear Brooke isn't here anymore :( [20:21:53] She left us in October 2021 to take a job at Digital Ocean. She got caught in the recent layoffs at DO, but has found a new gig at Akamai as of last month.