[00:00:12] 06serviceops, 06MediaWiki-Engineering: Clean up UcfirstOverrides.php following PHP 7.4 -> 8.1 transition - https://phabricator.wikimedia.org/T394556#10855589 (10tstarling) >>! In T394556#10846102, @Scott_French wrote: > After spending some time reading through T292552 and the code in `uppercaseTitlesForUnicode... [07:43:48] 06serviceops, 10MW-on-K8s: Create ValidatingAdmissionPolicy to permit hostPath mounts for mediawiki - https://phabricator.wikimedia.org/T395225 (10jijiki) 03NEW [09:42:37] 06serviceops, 10Ceph, 06Data-Persistence, 10SRE-swift-storage: Onboard the Docker Registry to apus - https://phabricator.wikimedia.org/T394476#10856107 (10akosiaris) 1 single bucket, at least at the beginning. Reading https://distribution.github.io/distribution/about/configuration/, I don't think the softw... [09:43:06] 06serviceops, 10Ceph, 06Data-Persistence, 10SRE-swift-storage: Onboard the Docker Registry to apus - https://phabricator.wikimedia.org/T394476#10856108 (10akosiaris) > If you want to do some testing, I could set you up with a test account on apus. That would be swell! [10:21:32] 06serviceops, 06Security-Team, 07SecTeam-Processed: Migrate Security-Team jobs to mw-cron - https://phabricator.wikimedia.org/T388531#10856187 (10Clement_Goubert) 05Stalled→03In progress @Joe is going to try to make the approach outlined in [[ https://phabricator.wikimedia.org/T388531#10851842 | my last... [10:29:43] 06serviceops, 06Security-Team, 07SecTeam-Processed: Migrate Security-Team jobs to mw-cron - https://phabricator.wikimedia.org/T388531#10856196 (10A_smart_kitten) Slight side note (I mentioned this on the merged-in task, but I just wanted to make sure it's seen by #serviceops!): if/given that the `generatecap... [10:41:43] 06serviceops, 06Security-Team, 07SecTeam-Processed: Migrate Security-Team jobs to mw-cron - https://phabricator.wikimedia.org/T388531#10856208 (10Clement_Goubert) >>! In T388531#10856196, @A_smart_kitten wrote: > Slight side note (I mentioned this on the merged-in task, but I just wanted to make sure it's se... [11:37:14] 06serviceops, 06Trust and Safety Product Team: Migrate trust_and_safety_product_team jobs to mw-cron - https://phabricator.wikimedia.org/T388542#10856378 (10Clement_Goubert) 05Resolved→03In progress Ok so the issue is a little worse than I thought. The old wrapper didn't `set -e`, which meant the loop kept... [11:39:58] 06serviceops, 06Trust and Safety Product Team: Add a flag to the mwscript wrapper to set +e when required - https://phabricator.wikimedia.org/T395245 (10Clement_Goubert) 03NEW [11:40:05] 06serviceops, 06Trust and Safety Product Team: Add a flag to the mwscript wrapper to set +e when required - https://phabricator.wikimedia.org/T395245#10856395 (10Clement_Goubert) p:05Triage→03High [12:11:05] 06serviceops, 06Growth-Team, 10GrowthExperiments, 10MW-on-K8s, 07Wikimedia-production-error: growthexperiments-refreshlinkrecommendations periodic jobs error out on search too busy - https://phabricator.wikimedia.org/T395247 (10Clement_Goubert) 03NEW p:05Triage→03High [12:21:54] 06serviceops, 10Add-Link, 10GrowthExperiments, 10MW-on-K8s, and 2 others: growthexperiments-refreshlinkrecommendations periodic jobs error out on search too busy - https://phabricator.wikimedia.org/T395247#10856502 (10Michael) @Clement_Goubert: Retries sound like the preferred solution if we can implement... [12:22:22] 06serviceops, 06Growth-Team, 10GrowthExperiments, 10MW-on-K8s, 07Wikimedia-production-error: growthexperiments-fixlinkrecommendationdata-dryrun 'link-recommendation' is not a link recommendation task type error - https://phabricator.wikimedia.org/T395249 (10Clement_Goubert) 03NEW p:05Triage→03High [12:28:11] 06serviceops, 06Growth-Team, 10GrowthExperiments, 10MW-on-K8s, 07Wikimedia-production-error: growthexperiments-fixlinkrecommendationdata-dryrun 'link-recommendation' is not a link recommendation task type error - https://phabricator.wikimedia.org/T395249#10856545 (10Michael) The current error message is... [13:02:24] 06serviceops, 06Growth-Team, 10GrowthExperiments, 10MW-on-K8s, 07Wikimedia-production-error: growthexperiments-fixlinkrecommendationdata-dryrun 'link-recommendation' is not a link recommendation task type error - https://phabricator.wikimedia.org/T395249#10856691 (10Clement_Goubert) Once the changes for... [13:04:33] 06serviceops, 10function-orchestrator, 10Abstract Wikipedia team (25Q4 (Apr–Jun)), 07OKR-Work: Tidy app chart modules in function-orchestrator - https://phabricator.wikimedia.org/T395254 (10jijiki) 03NEW [13:11:45] 06serviceops, 10function-orchestrator, 10Abstract Wikipedia team (25Q4 (Apr–Jun)), 07OKR-Work: Tidy app chart modules in function-orchestrator - https://phabricator.wikimedia.org/T395254#10856717 (10jijiki) [13:11:48] 06serviceops, 10function-orchestrator, 10Abstract Wikipedia team (25Q4 (Apr–Jun)), 07OKR-Work: Enable memcached in the orchestrator - https://phabricator.wikimedia.org/T391986#10856718 (10jijiki) [13:19:32] 06serviceops, 06collaboration-services, 06Data-Platform-SRE, 10Prod-Kubernetes, 07Kubernetes: Ensure all required kubectl versions are installed on deploy hosts - https://phabricator.wikimedia.org/T388388#10856732 (10JMeybohm) 05Open→03Resolved a:03JMeybohm Thanks for following up on this! I've... [13:27:05] 06serviceops, 10Add-Link, 10GrowthExperiments, 10MW-on-K8s, and 2 others: growthexperiments-refreshlinkrecommendations periodic jobs error out on search too busy - https://phabricator.wikimedia.org/T395247#10856746 (10Joe) >>! In T395247#10856502, @Michael wrote: > @Clement_Goubert: Retries sound like the... [13:40:12] 06serviceops: Replace Nutcracker for Redis (Thumbor, API Gateway, Changeprop) - https://phabricator.wikimedia.org/T333019#10856778 (10kamila) a:05kamila→03None [13:42:02] 06serviceops, 10function-orchestrator, 10Abstract Wikipedia team (25Q4 (Apr–Jun)), 07OKR-Work, 13Patch-For-Review: Enable memcached in the orchestrator - https://phabricator.wikimedia.org/T391986#10856797 (10jijiki) @cmassaro if you would like to do any tests, mcrouter's address is 127.0.0.1:11213 [13:42:18] 06serviceops, 10function-orchestrator, 10Abstract Wikipedia team (25Q4 (Apr–Jun)), 07OKR-Work, 13Patch-For-Review: Enable memcached in the orchestrator - https://phabricator.wikimedia.org/T391986#10856799 (10jijiki) a:03jijiki [13:44:02] 06serviceops, 10function-orchestrator, 10Abstract Wikipedia team (25Q4 (Apr–Jun)), 07OKR-Work, 13Patch-For-Review: Enable memcached in the orchestrator - https://phabricator.wikimedia.org/T391986#10856803 (10jijiki) [14:37:47] 06serviceops, 10envoy, 10Observability-Metrics, 13Patch-For-Review, 10SRE Observability (FY2024/2025-Q4): Revisit default envoy histogram buckets - https://phabricator.wikimedia.org/T391333#10856995 (10akosiaris) >>! In T391333#10853430, @elukey wrote: >>>! In T391333#10852614, @akosiaris wrote: >> I... [15:03:12] 06serviceops, 06Trust and Safety Product Team, 13Patch-For-Review: Add a flag to the mwscript wrapper to set +e when required - https://phabricator.wikimedia.org/T395245#10857082 (10Clement_Goubert) 05Open→03Resolved [15:08:27] 06serviceops, 06Trust and Safety Product Team: Migrate trust_and_safety_product_team jobs to mw-cron - https://phabricator.wikimedia.org/T388542#10857103 (10Clement_Goubert) 05In progress→03Resolved I've added the necessary flag to the job and ran it manually to verify it ran correctly. It does log an... [15:24:06] 06serviceops, 10CampaignEvents, 06Campaigns-Product-Team, 10MW-on-K8s: Migrate CampaignEvents jobs to mw-cron - https://phabricator.wikimedia.org/T385867#10857141 (10Daimona) Thank you! [15:37:53] 06serviceops, 10API Platform (RESTBase Deprecation Roadmap): Remove recommendation-api from the REST API offerings - https://phabricator.wikimedia.org/T390517#10857163 (10akosiaris) >>! In T390517#10692685, @akosiaris wrote: > Per https://w.wiki/DeQh, we are well within our estimations about the day amount of... [15:40:09] 06serviceops, 10ChangeProp: Exclude 404s from changeprop/changeprop-jobqueue logs - https://phabricator.wikimedia.org/T395132#10857169 (10hnowlan) Done {F60583750} [15:46:29] 06serviceops, 10Wikidata, 10Discovery-Search (2025.05.24 - 2025.06.13): MediaWikiCronJobFailed - https://phabricator.wikimedia.org/T395058#10857182 (10Gehel) This cronjob error should ping #wikidata and not Search. See discussion in T388538#10851664 for more details. [15:46:59] 06serviceops, 10Wikidata, 10Discovery-Search (2025.05.24 - 2025.06.13): MediaWikiCronJobFailed - https://phabricator.wikimedia.org/T395058#10857187 (10Gehel) @hoo : could you have a look at this failure? [15:47:04] 06serviceops, 10Wikidata: MediaWikiCronJobFailed - https://phabricator.wikimedia.org/T395058#10857188 (10Gehel) [15:49:13] 06serviceops, 10Wikidata: MediaWikiCronJobFailed - https://phabricator.wikimedia.org/T395058#10857197 (10Clement_Goubert) >>! In T395058#10857182, @Gehel wrote: > This cronjob error should ping #wikidata and not Search. See discussion in T388538#10851664 for more details. I'll switch it. [15:53:36] 06serviceops, 10Wikidata: MediaWikiCronJobFailed - https://phabricator.wikimedia.org/T395058#10857207 (10Clement_Goubert) 05Open→03Resolved a:03Clement_Goubert The failure was transient by the way, so I deleted the failing job to rearm alerting. [16:08:06] 06serviceops, 10Wikidata, 10Discovery-Search (2025.05.24 - 2025.06.13), 13Patch-For-Review: Migrate discovery-search jobs to mw-cron - https://phabricator.wikimedia.org/T388538#10857240 (10Clement_Goubert) 05In progress→03Resolved a:03Clement_Goubert >>! In T388538#10851661, @Clement_Goubert wrot... [16:08:21] 06serviceops, 10Add-Link, 10GrowthExperiments, 10MW-on-K8s, and 3 others: growthexperiments-refreshlinkrecommendations periodic jobs error out on search too busy - https://phabricator.wikimedia.org/T395247#10857244 (10Clement_Goubert) The loop for this job now ignores errors and continues, as it was in the... [16:08:24] 06serviceops, 06Growth-Team, 10GrowthExperiments, 10MW-on-K8s, and 2 others: growthexperiments-fixlinkrecommendationdata-dryrun 'link-recommendation' is not a link recommendation task type error - https://phabricator.wikimedia.org/T395249#10857245 (10Clement_Goubert) The loop for this job now ignores erro... [16:11:29] 06serviceops, 06SRE, 10WMF-General-or-Unknown: Re-consider ` >/dev/null 2>&1` as output of many cron'd MW maintenance scripts - https://phabricator.wikimedia.org/T187078#10857252 (10Clement_Goubert) 05Open→03Resolved All scripts now have alerting, and log to logstash. [16:12:23] 06serviceops, 10MW-on-K8s: Convert cirrus_build_completion_indices.sh to kubernetes CronJob - https://phabricator.wikimedia.org/T380171#10857256 (10Clement_Goubert) 05Open→03Invalid We've moved to sharding this job in {T388538} [16:14:32] 06serviceops, 10MW-on-K8s, 10Observability-Alerting, 10SRE Observability (FY2024/2025-Q3): Periodic job alerting - https://phabricator.wikimedia.org/T385709#10857266 (10Clement_Goubert) 05In progress→03Resolved Alerting is setup for all jobs, resolving, we can attach future improvements to a future... [16:18:25] 06serviceops, 10Add-Link, 10GrowthExperiments, 10MW-on-K8s, and 3 others: growthexperiments-refreshlinkrecommendations periodic jobs error out on search too busy - https://phabricator.wikimedia.org/T395247#10857291 (10Clement_Goubert) I have deleted the failed jobs to clear alerting. [16:18:26] 06serviceops, 06Growth-Team, 10GrowthExperiments, 10MW-on-K8s, and 2 others: growthexperiments-fixlinkrecommendationdata-dryrun 'link-recommendation' is not a link recommendation task type error - https://phabricator.wikimedia.org/T395249#10857292 (10Clement_Goubert) I have deleted the failed jobs to clea... [17:18:46] 06serviceops, 06Growth-Team, 10GrowthExperiments, 10MW-on-K8s, 07Wikimedia-production-error: growthexperiments-fixlinkrecommendationdata-dryrun 'link-recommendation' is not a link recommendation task type error - https://phabricator.wikimedia.org/T395249#10857444 (10Michael) >>! In T395249#10857292, @Cl... [20:52:07] 06serviceops, 10MW-on-K8s, 10Release-Engineering-Team (Priority Backlog 📥): Allow hostPath mounting of /srv/mediawiki in MediaWiki Helm chart - https://phabricator.wikimedia.org/T395284 (10jijiki) 03NEW