[02:38:53] as I recall, I was going to build the Toolviews UI at the Wikimedia Hackathon 2020, but that obviously didn't happen. Fortunately there's one coming up just next month, so maybe Toolviews will be a thing in the near future :) [02:47:17] a traffic tool for a traffic tool? O_O [08:20:57] Hi there. I have a (monthly) cron job defined using "toolforge-jobs". Is there a a way to trigger it once to run now (without forgetting the monthly schedule) ? thanks [08:34:33] huh, I see there's an open ticket on that https://phabricator.wikimedia.org/T321992 [08:36:06] Kotz: I think there has been even some code written to support that already, but I'd need to double check [08:42:11] Kotz: actually, reading the source code, it seems it does what you want! [08:42:22] give it a try, and I'll update the docs [08:45:32] arturo: this does not seem to work. "restart" does nothing and ends silently [08:45:33] tools.bothasava@tools-sgebastion-10:~$ toolforge-jobs restart monthly [08:45:33] tools.bothasava@tools-sgebastion-10:~$ toolforge-jobs show monthly [08:45:34] [...] [08:45:34] +-------------+------------------------------------+ [08:45:35] | Status: | Waiting for scheduled time | [08:45:35] +-------------+------------------------------------+ [08:45:36] | Hints: | No pods were created for this job. | [08:45:36] +-------------+------------------------------------+ [09:08:39] then that sounds like a bug! [09:09:29] let me try myself in your tool [09:10:10] arturo: iirc when implementing `restart` you specifically did not want it to also trigger manual starts for cronjobs [09:10:54] taavi: I remember that discussion, but also, there is a `_launch_manual_cronjob()` function that does just that [13:40:04] !log tools.stewardbots Restarting the bots [13:40:07] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stewardbots/SAL [14:03:46] I changed my registered email address with Wikitech via Special:Preferences a couple of minutes ago, confirmed it, but Special:Preferences is still displaying the old one. I purged cache, loged-in and out... still unchanged. But it's updated in LDAP though. Known issue? [14:04:52] log out and in again? [14:05:09] As I said, I already did that [14:06:52] it's changed as well in toolsadmin [14:07:59] What's your new one? A wikitech specific one? [14:08:00] "last confirmed 6 sep 2015 @ 17:13" heh nope [14:55:39] herzog: not know to me, but also not especially surprising that it could be broken somehow. The MediaWiki extension we use on Wikitech to sync up with the backing LDAP directory is fundamentally unmaintained and occasionally gets partially broken because of internal api changes in MediaWiki/AuthManager/etc. [14:56:22] herzog: please do file a bug and I will try to make some time to see if I can reproduce the issue [14:56:36] bd808: confirming the email twice fixed it for my account but curiously LDAP sync happened without issues :) [14:56:50] I'm trying to reproduce with my bot account [14:57:25] Alright, I'll file a task later [15:10:41] Filed as T334102 [15:10:41] T334102: Wikitech: Preferences not updating after email change - https://phabricator.wikimedia.org/T334102 [15:46:51] I wanted to give a shout out to taavi for a nice bit of bot+lua+template documentation helper magic he has done with https://wikitech.wikimedia.org/wiki/Template:Toolforge_latest_image. See edits like https://wikitech.wikimedia.org/wiki/Special:Diff/2066803 for usage. [15:54:13] TIL mw.loadJsonData(), nice [15:56:13] bd808: not sure if you did something in the background, but the UI is now displaying the new email after ~1 hour [15:56:41] cache expired :D [15:56:43] herzog: I didn't do anything, but this maybe confirms that there is a missing cache purge somewhere [15:57:07] so the email is actually changed in the db and ldap but the ui takes a while to refresh [15:57:28] no idea if this is exclusive for wikitech or it can be reproduced in prod [15:57:45] in any case, thanks all [15:57:57] Sounds a bit like T333925 and T249623 [15:57:58] T333925: Error during MFA setup for wikitech.wikimedia.org: MWException: CAS update failed on user_touched. The version of the user to be saved is older than the current version. - https://phabricator.wikimedia.org/T333925 [15:57:58] T249623: Logging in on Wikitech can throw fatal from LdapAuthentication "CAS update failed on user_touched" - https://phabricator.wikimedia.org/T249623 [15:58:16] both are cache vs db mismatch errors on wikitech [15:59:29] wikitech things like caching get messed up more often than we would like in part because wikitech is the only "production" wiki that is not hosted in the production wiki cluster. This leads to weird config drifts and mismatches. [17:59:07] I noticed there can be projects that do appear in openstack-browser after they are already deleted.. or that's how it seems. [17:59:31] example: here you see project "glampipe" as a user of a role: https://openstack-browser.toolforge.org/puppetclass/role::simplelamp2 [17:59:49] but here it says "Unknown project 'glampipe'. Are you just guessing?" https://openstack-browser.toolforge.org/project/glampipe [18:01:00] that's a known thing that's been on my mind a while :/ [18:02:03] taavi: ACK, that's exactly what I wanted to get out of this.. is it known already or not [18:02:16] let me know if I should make a ticket or that would be duplicate [18:14:49] I created T334127 [18:14:51] T334127: Puppet ENC has stale data for deleted projects - https://phabricator.wikimedia.org/T334127 [18:15:10] looks like a rather easy fix, not sure why I've been putting it off for so long [18:15:27] cool, thank you [18:15:52] I am trying to fix the situation with those mysql datadirs for users of simplamp2 [18:16:03] going through each project / instance .. [18:16:03] awesome [18:21:48] https://phabricator.wikimedia.org/T329571#8760338 [18:33:48] taavi: soo... I can ssh as root@ into any cloud VPS instance, but that does not mean I am member of every project in Horizon. as part of fixing this I want to edit project Hiera for some projects. I don't want to edit Hiera in the repo as usual because then the project admins can't change it themselves. But to be able to do that I would have to ask to be added to like 5 projects for 5 minutes and [18:33:54] then be removed again.. thoughts? should I ask for that or do you guys have admin power to edit Hiera for any project and would do that for me if it's a one-off [18:36:51] yeah, I don't see that being a problem. or for an one-off it's probably faster if you give me or someone else a list and I'll do it for you [18:39:15] ok, thanks, I will come up with the list for you in a bit [18:39:23] (via ticket, no worries0 [18:39:38] sgtm [18:50:12] while doing that..finds instances that are show as active but "Connection closed by UNKNOWN port 65535" when I try to ssh to them [18:50:34] while my global root access works on others, so assuming something wrong with them [18:50:45] do you have an example? [18:51:07] pixel.reading-web-staging.eqiad1.wikimedia.cloud [18:51:21] prototype1.vuessr.eqiad1.wikimedia.cloud [18:52:18] ooh.. wait. my bad I think [18:52:57] taavi: sorry, false alarm. I was not sending root as user name [19:21:08] !log tools.quickcategories deployed e27a219ad0 (remove some wmflabs references) [19:21:13] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.quickcategories/SAL [19:27:33] !log tools.wd-shex-infer updated ToolsDB domain name in config.yaml [19:27:36] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wd-shex-infer/SAL [19:59:54] Hi, I am running some scripts on Toolforge and a few day ago, one of them stopped working. I tried to rerun it but cron tells me that there is a job with the same name that is already running. This script is supposed to be running but it does not do anything more (no log, no work on the wiki). I typed "qstat" and here is the output: [19:59:54] job-ID  prior   name       user         state submit/start at     queue                          slots ja-task-ID [19:59:55] ----------------------------------------------------------------------------------------------------------------- [19:59:55]    1773 0.33008 cron-anagr tools.wiktio dr    02/13/2023 21:00:14 task@tools-sgeexec-10-14.tools     1 [19:59:56] I tried to delete it with "qdel -f 1773" [19:59:56] but it tells me "tools.wiktionnaire-pamputt has registered the job 1773 for deletion" and after a some time, it finishes but the job still appears in qstat. [19:59:57] I am not familiar with this grid stuff so maybe I am not doing the right thing. Does anyone knows what happens and how to solve this problem? [20:00:52] Pamputt: please use a pastebin next time instead of pasting several lines of text [20:00:55] Pamputt: that job is probably dead but still tracked in the job scheduler. Let me take a quick look... [20:01:04] but I have force deleted the job, you should be able to re-create it now [20:01:25] hehe. taavi is faster than me [20:01:55] Wow, very fast. And noted for pastbin [20:02:49] Does it mean that I could not do anything on my side, i.e. if it happens again in the future, I need to come here to ask for help :) ? [20:07:15] unfortunately yes, until that tools is migrated to Kubernetes. the grid engine does that sometimes and it needs an admin to tell it to force delete the job [20:10:07] taavi ok, thank you. No problem, now that I know what to do :)