[12:09:36] <kostajh>	 hi, T285538 was just upgraded to unbreak now, it looks like everything that uses foreachwikiindblist is broken (so a bunch of maintenance scripts running in cron jobs). could someone from SRE take a look please?
[12:09:36] <stashbot>	 T285538: All cronjobs using foreachwikiindblist broken in production: Fatal error: Uncaught Exception: MWWikiversions::readDbListFile: unable to read . in /srv/mediawiki/multiversion/MWWikiversions.php:94 - https://phabricator.wikimedia.org/T285538
[12:10:51] <kormat>	 effie: ^
[12:14:46] <apergos>	 it's being discussed in a backchannel
[12:15:04] <apergos>	 change might get rolled back, stay tuned for updates
[12:15:40] <kormat>	 📻
[12:39:59] <effie>	 kormat: tx, not much I can do I am afraid 
[12:40:09] <kormat>	 effie: np :)
[13:56:49] <joe>	 kostajh: sorry saw your ping just now, we reverted and then fixed the issue
[13:57:02] <joe>	 kostajh: but the logs fro the scripts are kinda worrisome
[13:57:24] <joe>	 Jun 25 13:57:17 mwmaint1002 mediawiki_job_growthexperiments-refreshLinkRecommendations[215788]: * Error fetching URL: Failed to connect to linkrecommendation.discovery.wmnet port 4005: Connection refused
[13:57:28] <joe>	 quite a lot of those
[13:57:36] <kostajh>	 Hmm
[13:57:37] <joe>	 and the service is up and reachable
[13:57:52] <joe>	 so I suspect you're making too many requests to it and filling all available workers?
[13:59:25] <kostajh>	 not sure. The grafana charts for the service look a bit odd for the lat two days https://w.wiki/3YLk part of it was the refreshLinkRecommendations script not running
[13:59:41] <kostajh>	 but we shouldn't be making more requests to the internal service than we have at any other time in the past
[14:00:58] <kostajh>	 I'm about to be afk but cc'ing tgr|away in case he's around a bit later to help troubleshoot.