[09:07:28] 06serviceops, 10Sustainability (Incident Followup): Broken / outdated links from Api Gateway & application servers docs - https://phabricator.wikimedia.org/T394584#10833806 (10Clement_Goubert) > https://wikitech.wikimedia.org/wiki/Application_servers/Runbook#Average_latency_exceeded links to the application se... [09:31:07] 06serviceops, 06Growth-Team, 10GrowthExperiments, 10MW-on-K8s, 13Patch-For-Review: Migrate GrowthExperiments maintenance jobs to mw-cron - https://phabricator.wikimedia.org/T385782#10833909 (10Clement_Goubert) We are kind of abusing CronJobs if they are meant to run continuously, but as of now we have no... [09:48:51] hnowlan: Regarding T394582 indeed expressjs reported it as 400 error but we overwritten it in mobileapps error handling [09:49:07] I sent a patch [09:49:55] nemo-yiannis: ah great, thank you! [09:50:38] might this also reduce other 500 cases? [09:55:00] not sure, this is very specific to URIError that we considered as a generic error [09:55:13] but maybe (?) [09:55:28] ahh okay, cool [09:58:06] nemo-yiannis: <3 [10:22:17] i tried to deploy the change on mobileapps but i see diffs in helm from https://phabricator.wikimedia.org/T391333 [10:22:23] should i go ahead and deploy them too ? [10:29:04] cc hnowlan [10:29:58] nemo-yiannis: yep, should be fine [10:36:42] 06serviceops: Implement KeyModifyRoute in mcrouter configuration - https://phabricator.wikimedia.org/T393281#10834177 (10jijiki) [10:36:48] 06serviceops: Implement KeyModifyRoute in mcrouter configuration - https://phabricator.wikimedia.org/T393281#10834178 (10jijiki) 05In progress→03Resolved a:03jijiki [10:52:45] 06serviceops, 10MW-on-K8s, 13Patch-For-Review: Investigate startingDeadlineSeconds setting for kubernetes CronJobs - https://phabricator.wikimedia.org/T394423#10834245 (10Clement_Goubert) 05Open→03In progress p:05Triage→03High [11:50:35] 06serviceops, 10MW-on-K8s: Implement continuously running maintenance jobs - https://phabricator.wikimedia.org/T394657 (10Clement_Goubert) 03NEW [11:50:42] 06serviceops, 10MW-on-K8s: Implement continuously running maintenance jobs - https://phabricator.wikimedia.org/T394657#10834408 (10Clement_Goubert) p:05Triage→03Medium [12:16:46] 06serviceops: Implement KeyModifyRoute in mcrouter configuration - https://phabricator.wikimedia.org/T393281#10834469 (10cmassaro) Thank you! [12:25:18] 06serviceops, 10function-orchestrator, 10Abstract Wikipedia team (25Q4 (Apr–Jun)), 07OKR-Work: Enable memcached in the orchestrator - https://phabricator.wikimedia.org/T391986#10834485 (10cmassaro) @jijiki Are we now able to address mcrouter in the orchestrator service? If so, what is the URI we should use? [12:29:01] 06serviceops, 10Page Content Service: Production error: worker died, restarting - https://phabricator.wikimedia.org/T394659 (10Jgiannelos) 03NEW [13:51:15] 06serviceops, 10MW-on-K8s, 13Patch-For-Review: Implement periodic maintenance scripts for mw-on-k8s - https://phabricator.wikimedia.org/T341555#10834838 (10Clement_Goubert) >>! In T341555#10832816, @A_smart_kitten wrote: > Thank you for the detailed response @clement_goubert! > >> As far as duplicates go, a... [13:56:43] 06serviceops, 10MW-on-K8s: Investigate startingDeadlineSeconds setting for kubernetes CronJobs - https://phabricator.wikimedia.org/T394423#10834873 (10Clement_Goubert) The jobs in {P76154} should also get a proper `startingDeadlineSeconds` and `concurrency` so they have a chance to complete. [14:34:58] 06serviceops, 10Page Content Service, 06SRE: mobileapps returns 500 error in response to malformed /v1/page/summary path - https://phabricator.wikimedia.org/T394610#10835045 (10hnowlan) This issue was hopefully addressed in https://gerrit.wikimedia.org/r/c/mediawiki/services/mobileapps/+/1147736 [14:41:40] 06serviceops, 10Page Content Service, 06SRE: mobileapps returns 500 error in response to malformed /v1/page/summary path - https://phabricator.wikimedia.org/T394610#10835083 (10Jgiannelos) 05Open→03Resolved a:03Jgiannelos Just verified this: GET https://en.wikipedia.org/api/rest_v1/page/summary/ext... [14:51:24] 06serviceops, 10function-orchestrator, 10Abstract Wikipedia team (25Q4 (Apr–Jun)), 07OKR-Work: Enable memcached in the orchestrator - https://phabricator.wikimedia.org/T391986#10835109 (10jijiki) >>! In T391986#10834485, @cmassaro wrote: > @jijiki Are we now able to address mcrouter in the orchestrator ser... [20:30:50] 06serviceops, 10MediaWiki-Special-pages, 13Patch-For-Review: Migrate updatequerypages/update_special_pages/initsitestats jobs to mw-cron - https://phabricator.wikimedia.org/T388534#10837061 (10Scott_French) [22:02:45] 06serviceops, 10function-orchestrator, 10Abstract Wikipedia team (25Q4 (Apr–Jun)), 07OKR-Work: Enable memcached in the orchestrator - https://phabricator.wikimedia.org/T391986#10837445 (10cmassaro) >>! In T391986#10835109, @jijiki wrote: >>>! In T391986#10834485, @cmassaro wrote: >> @jijiki Are we now able...