[05:16:52] StrikerBot creates Phab tags such as https://phabricator.wikimedia.org/project/profile/6294/ . Where to find the corresponding code repo for that new tag? [05:31:52] andre_: https://phabricator.wikimedia.org/source/labs-striker/ [05:35:17] JJMC89, the source of the tag, not the source of Striker :) [05:38:46] you the tags are created in striker (toolsadmin). For a tool you maintain (e.g. https://toolsadmin.wikimedia.org/tools/id/jjmc89-bot) there is a button to create a phab project in the left pannel. [05:42:13] hmm, so there's no way to make any assumptions about the location of a code base of that project tag? [05:43:39] if it isn't added to toolsadmin, then no. https://toolsadmin.wikimedia.org/tools/id/animanga-db-import-bot doesn't have a git repo. [05:46:32] I was able to find https://github.com/PythonCoderAS/wikidata-anime-import [05:47:57] Hmm, I see. Thanks! [15:16:08] Rook: Good morning from East Liberty. [15:16:49] Rook: Some time between today and Wed noon I'll know if we can chew on that SPARQL kernel fork :) [15:17:07] Rook: And I get to use "yinz" in actual conversations. [15:27:11] pr3d4t0r: awesome welcome to East Liberty (If you're on Liberty blvd on south side, so the other side of the city from you, go to Fiori's, they have amazing pizza) [21:19:40] `toolforge-jobs show send-daily-report` is telling me that the job that's supposed to be run daily hasn't run since the 10th. [21:19:48] (for the anomiebot tool) [21:37:47] anomie: `kubectl get cronjobs` and `kubectl describe cronjobs/send-daily-report` agree with that. That's only comforting in that the issue seems to be not in the toolforge-jobs API but instead in something in the kubernetes cluster itself or your namespace. [21:38:05] * bd808 tries to find events explaining why the job failed to schedule [21:50:38] !log tools.anomiebot Changed k8s cronjob send-daily-report schedule to 3 0 AnomieBOT.git bot botctl.sh botlogs bot-test generate-daily-report.sh grid jobs.yaml logs mysql2sqlite needed-perl-packges public_html README replica.my.cnf rotate-logs.pl send-daily-report.sh service.manifest task-status test-task-status.pl tmp www AnomieBOT.git bot botctl.sh botlogs bot-test generate-daily-report.sh grid jobs.yaml logs mysql2sqlite needed-perl-packges public_html README replica.my.cnf rotate-logs.pl send-daily-report.sh service.manifest task-status test-task-status.pl tmp www * to see if this makes the job run again. [21:50:40] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.anomiebot/SAL [21:50:55] lol [21:51:06] that was a cli logging fail on my part [21:51:20] !log tools.anomiebot Changed k8s cronjob send-daily-report schedule to "3 0 * * *" to see if this makes the job run again. [21:51:21] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.anomiebot/SAL [21:51:48] Theory being it doesn't like exactly midnight? [21:53:25] anomie: yeah, that's my only random guess at this point. I haven't found any "events" saying that the cronjob object itself is busted. My random guess is that too many things are scheduled for midnight and that's causing this missed job as a side effect. [21:56:57] anomie: I did the edit via `kubectl edit cronjobs send-daily-report` as your tool. That opens up the yaml describing the cronjob object in an editor and then updates the k8s state when you save and exit the editor. [22:03:47] bd808: if it's reassuring, my cron ran fine this afternoon [22:03:54] bd808: if it's reassuring, my cron ran fine this afternoon [22:03:59] (2pm utc) [22:13:15] (was double-sending that a bit or..? :p) [22:14:44] I was *just* thinking “you know, it’s been a while since the double-send bug happened” [22:15:24] It was double on the irc side too. PEBKAC from RhinosF1? [22:17:13] * bd808 is impressed that there are 1058 cronjob objects registered across all namespaces [22:17:38] RhinosF1 is not idempotent and should be run on Kubernetes with caution [22:20:22] !log tools.admin Deleted stuck updatetools pod launched by a CronJob object [22:20:24] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.admin/SAL [22:24:56] anomie: I think I'm finding a few other jobs globally that have a 00:00 start time and have not fired when expected. Not all of them are strictly '0 0 * * *' schedules but the minute and hour of 0 are a commonality. [22:26:27] as an example, borkedbot has a job with a '0 0 * * 4' schedule (midnight on Thursday) and it hasn't fired for 11 days. [22:30:02] citationhunt seems to have lots and lots of cronjobs that have never fired or last fired >200 days ago. [22:30:09] I had something similar happen to one of my jobs the week of 20 Nov. After a report to my on 24 Nov I found that a daily job hadn't run since 20 Nov. I couldn't firure out why so ended up deleting the cronjob and adding it back. [22:31:43] * bd808 grumbles about jobs that fire every minute [22:34:14] Rook: Will do! I'll crash early tonight. No sleep on the way in :) Tomorrow is genetic algorithms and ontologies day. Need to be fresh and ready to rock. [22:35:12] pr3d4t0r: Have a good time in pgh [22:48:19] I made T325027 to track the 00:00 cronjob problem. Maybe t.aavi or a.rturo will have some ideas about where to look for more info on what the scheduler is up to. [22:48:20] T325027: Kubernetes CronJobs scheduled for midnight not firing consistently - https://phabricator.wikimedia.org/T325027 [22:49:27] SantiComposite: your anticompositebot.catwatch showed up as one that has missed some days [22:53:46] is it possible you have over 500 jobs on the cluster? [22:53:57] " When the number of jobs exceeds 500, cronjob cannot schedule, bug of pager.List #77465 " [22:54:32] It's not just possible, it's a certainty. I counted 1058 a bit ago [22:54:38] "has been fixed in 1.14.7" [22:55:30] We are running v1.21.8 so in theory patched against that old bug [22:55:40] ack, that was too old then [23:11:03] The double message was my internet being stupid and irc saying it failed but actually sending [23:49:22] !log wm-bot clear apache logs to free up space [23:49:24] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wm-bot/SAL