[04:02:21] <MirahezeRelay>	 <paladox> See dms @orduin
[04:02:24] <MirahezeRelay>	 <paladox> Pls
[06:23:10] <MirahezeRelay>	 <orduin> [1/2] Currently with grafana reporting 2596 unclaimed LocalGlobalUserPageCacheUpdateJob, I'm seeing 236 responses for `LocalGlobalUserPageCacheUpdateJob` in redis. A spot check of one or two of them shows that whatever job is stored in redis is not stored in the job table of the database meaning there has been some drift between redis and the db. There's also global::rootjob keys for at l
[06:23:10] <MirahezeRelay>	 <orduin> [2/2] east one of the affected wikis.
[06:23:47] <MirahezeRelay>	 <rhinosf1> @orduin the job table is unused in our setup
[06:24:08] <MirahezeRelay>	 <orduin> Weird, then I wonder what else may be causing runJobs to say there is no jobs to run for that wiki?
[06:24:15] <MirahezeRelay>	 <rhinosf1> But grafana reporting 2596 jobs compared to 236 keys doesn't make sense
[06:24:31] <MirahezeRelay>	 <orduin> l-unclaimed is a list and may include multiple jobs
[06:24:42] <MirahezeRelay>	 <rhinosf1> @orduin my expectation is that there isn't any jobs and they are broken
[06:24:46] <MirahezeRelay>	 <rhinosf1> Or they'd have ran
[06:24:55] <MirahezeRelay>	 <rhinosf1> But I have no idea why they are there
[06:25:27] <MirahezeRelay>	 <orduin> There's still data in the keys though, but I can't make sense of it as it is encoded data
[06:25:55] <MirahezeRelay>	 <rhinosf1> If runJobs.php doesn't show them, they're broken
[06:26:02] <MirahezeRelay>	 <rhinosf1> So I'd just purge the keys
[06:26:36] <MirahezeRelay>	 <rhinosf1> But we need to eventually find out why it happens
[06:29:16] <MirahezeRelay>	 <orduin> Uhh, there's > 200k jobs matching `global:jobqueue::rootjob:`
[06:29:55] <MirahezeRelay>	 <rhinosf1> We should be purging rootjob
[06:29:57] <MirahezeRelay>	 <orduin> Still listing and over 500k now
[06:30:10] <MirahezeRelay>	 <rhinosf1> John wrote code to drop them
[06:31:29] <MirahezeRelay>	 <orduin> Ended at just over 730k
[06:33:04] <RhinosF1>	 That's not supposed to happen
[06:37:30] <MirahezeRelay>	 <orduin> Do you happen to know where the script was? It's 1:30am here, and am tired
[06:37:40] <MirahezeRelay>	 <orduin> Want to find it and make sure it's running before going to bed
[06:38:27] <RhinosF1>	 He built it into the jobchron server
[06:43:26] <MirahezeRelay>	 <orduin> Ah, could have sworn it was in puppet somewhere, but I see that the cause for both problems appears to be <https://github.com/miraheze/jobrunner-service/commit/b0be7a3b3c24684a114e16325e01e2202edc108a>
[06:47:49] <MirahezeRelay>	 <orduin> Added a revert in <https://github.com/miraheze/jobrunner-service/commit/4f1bc6cb71002919df18548a087c332db7087c84>
[06:49:57] <MirahezeRelay>	 <orduin> I've let puppet handle the deployment, but it doesn't look like any service(s) were restarted by the merge, so I don't know if the change will do anything tonight.
[06:50:03] <MirahezeRelay>	 <orduin> Anyway, it's late, and I'm going to bed.