[03:12:59] looks like geohack is 504ing https://geohack.toolforge.org/geohack.php?pagename=Potton,_Quebec¶ms=45_05_N_72_22_W_region:CA-QC_type:city(2012) (other URLs too) [05:10:56] Is geo hack down for anyone else [08:30:49] Geohack seems to be working now for me. On windows you might want to try `ipconfig /flushdns` and/or restart the system. [08:32:11] Could somebody help with our WLM map? There is is an on-going contest... And the site returns 504 for me: https://zabytki.toolforge.org/ [08:33:15] !log tools.zabytki `webservice restart`, reports of issues on #wikimedia-cloud [08:33:17] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.zabytki/SAL [08:33:38] eccenux: I restarted it, seems to work better now [08:34:47] Thanks! It works 😊 [13:50:18] looks like xtools and acc are having connectivity issues (acc's error is the same as the recent DNS issues) [13:51:31] I can ssh to toolforge though [14:10:22] mdaniels5757: is there a particular domain that isn't resolving? [14:10:55] andrewbogott: seems to be fixed now, probably caching? [14:11:07] great [15:19:47] I've not seen any announcements but I gather there are problems with toolforge-based tools, and connecting to toolforge via SSH? [15:21:34] CurbSafeCharmer: there's been a couple of recent issues, but they should(?) be resolved by now. What sort of issues are you seeing/what error do you get when attempting to SSH to `login.toolforge.org` ? [15:25:38] CurbSafeCharmer: what problems are you experiencing? [15:35:46] I've logged in ok now, user error I think. Just trying to see why reFill stuck at 'Submitting your task...' [15:53:31] !log tools.refill-api webservice restart [15:53:33] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.refill-api/SAL [15:54:09] CurbSafeCharmer: ^, looks like a `webservice restart` resolved it [16:01:38] :) [17:34:02] I tried building a python 3.11 kubernetes container thing with oursql3 installed. Even if I had wheel installed in the container beforehand, the build of oursql3 fails with a really long and ugly error. But I have a workstation at home that also has Python 3.11 and I was able to build oursql3 inside of a venv just fine. Is there something Wikimedia Cloud-specific I’m missing? [17:34:34] can you share the error message you're getting? [17:34:46] Gladly, hold on [17:35:27] It’s stored in` /data/project/reports-bot/bootstrap-venv.err `if you want to look yourself [17:36:34] pasted: https://paste.toolforge.org/view/37d907d7 [17:37:11] that doesn't seem to be the full output of `pip install` [17:37:31] (sorry, I should have been more specific on what I need) [17:37:51] That’s the error file; there’s also an` out `file [17:38:15] thanks [17:38:40] https://paste.toolforge.org/view/1423fdd9 [17:38:53] (I appreciate that you don’t have to fill out the captcha a second time once you’ve done it once) [17:41:29] oursql was last updated in 2016, which is not an immediately promising sign. it also doesn't seem to have any prebuilt wheels available :/ [17:42:11] Can I prebuild my own wheel? I have no trouble building oursql3 on my own system which also uses Python 3.11 [17:44:51] you can try, I don't immediately know if that would work. honestly I would suggest using pymysql or some other maintained implementation instead [17:45:19] Heh, it *did* use PyMySQL before a commit replaced it with oursql3 :( [17:47:46] :/ [20:30:14] bd808: got a moment? [20:51:02] Who can I ask to either extract the secret of an OAuth consumer I just created, or just straight up delete it, so I can recreate it? I inadvertently clicked away from the page before I could copy the secret. [20:52:02] Can't you just create another one anyway? [20:57:55] I really don't feel like cluttering my already cluttered list of consumers. Plus, I kind of want all of my consumers at v1.3 [20:58:11] Which that one is, but now I am missing the secret for. [20:58:37] I think there's some button to reset the credentials for a consumer? [21:00:16] YES PLEASE [21:00:28] We can reject them, but can't delete (unless that deletes too?) [21:00:56] It only rejects, but maybe you can at least reversion that consumer to some arbitrary version number? [21:01:09] (Probably needs prod access) [21:01:28] We're not editing things directly in the DB [21:09:25] :-( [22:19:58] Krinkle, legoktm: could you whack krinklebot please? it hasn't edited since yesterday [22:20:52] probably would be a good task to move to k8s for reliability's sake [22:25:43] job-ID prior name user state submit/start at queue slots ja-task-ID [22:25:43] ----------------------------------------------------------------------------------------------------------------- [22:25:43] 2153961 0.25173 fileprotec tools.krinkl r 09/12/2023 06:40:17 task@tools-sgeexec-10-14.tools 1 [22:25:51] [Wed Sep 13 22:00:07 2023] there is a job named 'fileprotectionsync' already active [22:25:51] [Wed Sep 13 22:10:04 2023] there is a job named 'fileprotectionsync' already active [22:25:51] [Wed Sep 13 22:20:04 2023] there is a job named 'fileprotectionsync' already active [22:26:17] $ qdel fileprotectionsync [22:26:17] tools.krinklebot has registered the job 2153961 for deletion [22:27:51] 2153961 0.25173 fileprotec tools.krinkl dr 09/12/2023 06:40:17 task@tools-sgeexec-10-14.tools 1 [22:27:55] no details? [22:28:30] $ qstat -f -j 2153961 [22:28:44] scheduling info: queue instance "webgrid-lighttpd@tools-sgeweblight-10-17.tools.eqiad1.wikimedia.cloud" dropped because it is temporarily not available [22:28:44] queue instance "webgrid-lighttpd@tools-sgeweblight-10-22.tools.eqiad1.wikimedia.cloud" dropped because it is temporarily not available [22:28:44] queue instance "continuous@tools-sgeexec-10-14.tools.eqiad1.wikimedia.cloud" dropped because it is overloaded: np_load_avg=1.767500 (= 1.767500 + 0.50 * 0.000000 with nproc=4) >= 1.75 [22:28:44] queue instance "task@tools-sgeexec-10-14.tools.eqiad1.wikimedia.cloud" dropped because it is overloaded: np_load_avg=1.767500 (= 1.767500 + 0.50 * 0.000000 with nproc=4) >= 1.75 [22:28:44] queue instance "webgrid-generic@tools-sgewebgen-10-3.tools.eqiad1.wikimedia.cloud" dropped because it is overloaded: np_load_avg=3.425000 (= 3.425000 + 0.50 * 0.000000 with nproc=4) >= 2.75 [22:29:56] $ qdel -f fileprotectionsync [22:29:56] job 2153961 is already in deletion [22:29:56] .. [22:33:52] well, it returned but still hasn't actually deleted the job [22:34:29] anyone interested in migrating, let me know at T319848 and I'll add you to the respectioce tool admin group. [22:34:30] T319848: Migrate krinklebot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319848 [22:34:39] Iv'e run it manaully once for now [22:34:45] gotta get back to other things meanwhile [22:37:50] alternatively if a cloud admin knows how to actually kill a stuck job, that'd be great too [22:38:26] !help Can someone wave a magic wand to actually delete job 2153961? [22:38:26] If you don't get a response in 15-30 minutes, please create a phabricator task -- https://phabricator.wikimedia.org/maniphest/task/edit/form/1/?projects=wmcs-kanban