[07:46:35] 06serviceops, 10Prod-Kubernetes, 07Kubernetes: Migration to containerd and away from docker - https://phabricator.wikimedia.org/T362408#10862327 (10JMeybohm) 05Stalled→03Resolved 🎉 [09:02:02] 06serviceops, 10Ceph, 06Data-Persistence, 10SRE-swift-storage, 13Patch-For-Review: Onboard the Docker Registry to apus - https://phabricator.wikimedia.org/T394476#10862692 (10MatthewVernon) @elukey OK, I've set that up for you. Quota is 3T, but can be adjusted as needed - it's really there so we can keep... [10:09:54] 06serviceops, 10DiscussionTools, 10MediaWiki-REST-API, 06MW-Interfaces-Team, and 3 others: Reroute RESTbase Endpoints for Linter Errors to MediaWiki REST Endpoints - https://phabricator.wikimedia.org/T384216#10862835 (10hnowlan) [11:47:19] 06serviceops, 10MW-on-K8s: Limit CPU usage for mw-on-k8s cli deployments - https://phabricator.wikimedia.org/T395436 (10Clement_Goubert) 03NEW [11:47:34] 06serviceops, 10MW-on-K8s: Limit CPU usage for mw-on-k8s cli deployments - https://phabricator.wikimedia.org/T395436#10863170 (10Clement_Goubert) 05Open→03In progress p:05Triage→03Medium [11:51:18] 06serviceops, 10DiscussionTools, 10MediaWiki-REST-API, 06MW-Interfaces-Team, and 3 others: Reroute RESTbase Endpoints for Linter Errors to MediaWiki REST Endpoints - https://phabricator.wikimedia.org/T384216#10863179 (10MSantos) @hnowlan yes they are and I believe this is safe to switch-over. Any thoughts... [12:35:02] Hello ServiceOps. We would like to merge this patch, which will (hopefully) enable a scap deployment of mediawiki to the dse-k8s-eqiad cluster. https://gerrit.wikimedia.org/r/c/operations/puppet/+/1148203 [12:35:56] Would this be a convenient time and would someone perhaps be available to help verify that it works as intended? Thanks. [12:51:13] 06serviceops: low rate of mw-memcached errors - https://phabricator.wikimedia.org/T371881#10863351 (10jijiki) [12:51:15] 06serviceops, 06Infrastructure-Foundations, 10Prod-Kubernetes, 07Kubernetes: Race condition in iptables rules during puppet runs on k8s nodes - https://phabricator.wikimedia.org/T374366#10863352 (10jijiki) [12:51:41] 06serviceops, 06Infrastructure-Foundations, 10Prod-Kubernetes, 07Kubernetes: Race condition in iptables rules during puppet runs on k8s nodes - https://phabricator.wikimedia.org/T374366#10863354 (10jijiki) 05Resolved→03Open Re-opening, as it is partially the reason of T371881 [13:24:29] btullis: we'd prefer to wait for s.wfrench-wmf , since he's been working on it [13:29:12] Raine: Cool, that's fine by me. [13:51:39] 06serviceops, 10DiscussionTools, 10MediaWiki-REST-API, 06MW-Interfaces-Team, and 3 others: Reroute RESTbase Endpoints for Linter Errors to MediaWiki REST Endpoints - https://phabricator.wikimedia.org/T384216#10863707 (10MSantos) Had a conversation with @daniel and it seems that these are not enabled yet.... [13:55:56] 06serviceops, 10MW-on-K8s, 13Patch-For-Review: Limit CPU usage for mw-on-k8s cli deployments - https://phabricator.wikimedia.org/T395436#10863722 (10Clement_Goubert) 05In progress→03Resolved a:03Clement_Goubert Limited resources for mediawiki containers created in these namespaces: - CPU request/li... [14:11:18] 06serviceops, 10DiscussionTools, 10MediaWiki-REST-API, 06MW-Interfaces-Team, and 3 others: Reroute RESTbase Endpoints for Linter Errors to MediaWiki REST Endpoints - https://phabricator.wikimedia.org/T384216#10863819 (10Jgiannelos) From a quick look at the RESTBase metrics it looks like that even though we... [14:36:42] 06serviceops, 10DiscussionTools, 10MediaWiki-REST-API, 06MW-Interfaces-Team, and 3 others: Reroute RESTbase Endpoints for Linter Errors to MediaWiki REST Endpoints - https://phabricator.wikimedia.org/T384216#10863969 (10hnowlan) There are also external hits for `v1/page/lint/$title` and `v1/page/lint/$titl... [14:40:12] 06serviceops, 06Security-Team, 13Patch-For-Review, 07SecTeam-Processed: Migrate Security-Team jobs to mw-cron - https://phabricator.wikimedia.org/T388531#10863988 (10Kappakayala) this is blocking PHP upgrade and hence the mediawiki release. Could we get this looked into as high priority? @Krinkle , @MSanto... [14:51:00] 06serviceops, 10DiscussionTools, 10MediaWiki-REST-API, 10RESTBase Sunsetting, and 3 others: Reroute RESTbase Endpoints for Linter Errors to MediaWiki REST Endpoints - https://phabricator.wikimedia.org/T384216#10864059 (10daniel) [14:51:29] 06serviceops, 06Security-Team, 13Patch-For-Review, 07SecTeam-Processed: Migrate Security-Team jobs to mw-cron - https://phabricator.wikimedia.org/T388531#10864063 (10Tgr) >>! In T388531#10849408, @Clement_Goubert wrote: > Find a way around `limit.sh` in `UnboxedCommand` (which doesn't seem possible as is)... [14:53:08] btullis: thanks for checking in! so, the scap release that picked up the new feature was deployed yesterday. as I mentioned on the puppet patch, we should leave a bit of time for it to be rollback-safe before merging and deploying, but 24h is probably enough :) [14:55:50] 06serviceops, 06Security-Team, 13Patch-For-Review, 07SecTeam-Processed: Migrate Security-Team jobs to mw-cron - https://phabricator.wikimedia.org/T388531#10864081 (10Reedy) >>! In T388531#10863987, @Kappakayala wrote: > this is blocking PHP upgrade and hence the mediawiki release. Could we get this looked... [14:56:42] btullis: so, someone from serviceops should work with you to get this merged and run through the first deployment. I'm happy to be that person, but the range of times during which we could do it during overlap hours are somewhat limited ... let me follow up with you out of band to try to schedule something. [15:07:48] 06serviceops, 10function-orchestrator, 10Abstract Wikipedia team (25Q4 (Apr–Jun)), 07OKR-Work: Tidy app chart modules in function-orchestrator - https://phabricator.wikimedia.org/T395254#10864160 (10jijiki) @cmassaro parts of this chart were written before our chart modules either existed or they were matu... [15:17:46] 06serviceops, 06Security-Team, 13Patch-For-Review, 07SecTeam-Processed: Migrate Security-Team jobs to mw-cron - https://phabricator.wikimedia.org/T388531#10864254 (10MSantos) >>! In T388531#10864081, @Reedy wrote: >>>! In T388531#10863987, @Kappakayala wrote: >> this is blocking PHP upgrade and hence the m... [15:18:50] 06serviceops, 06Security-Team, 13Patch-For-Review, 07SecTeam-Processed: Migrate Security-Team jobs to mw-cron - https://phabricator.wikimedia.org/T388531#10864262 (10Clement_Goubert) >>! In T388531#10864063, @Tgr wrote: >>>! In T388531#10849408, @Clement_Goubert wrote: >> Find a way around `limit.sh` in `U... [15:34:28] 06serviceops, 06Security-Team, 13Patch-For-Review, 07SecTeam-Processed: Migrate Security-Team jobs to mw-cron - https://phabricator.wikimedia.org/T388531#10864366 (10Tgr) So if the goal is to just not use `limit.sh` (based on the presence of an env flag or whatever), that's a trivial code change. Convertin... [15:38:36] 06serviceops, 10Sustainability (Incident Followup): Broken / outdated links from Api Gateway & application servers docs - https://phabricator.wikimedia.org/T394584#10864403 (10hnowlan) 05Open→03Resolved a:03hnowlan API gateway logging and documentation updated to be more useful. I think that covers e... [16:21:50] 06serviceops, 06SRE: Silence RESTGatewayBackendErrorsHigh for envoy_cluster_name: mobileapps_cluster - https://phabricator.wikimedia.org/T394609#10864810 (10hnowlan) I think this can probably be closed given the improvements to mobileapps since. [16:23:41] 06serviceops, 06Security-Team, 13Patch-For-Review, 07SecTeam-Processed: Migrate Security-Team jobs to mw-cron - https://phabricator.wikimedia.org/T388531#10864816 (10Reedy) >>! In T388531#10864366, @Tgr wrote: > So if the goal is to just not use `limit.sh` (based on the presence of an env flag or whatever)... [16:36:34] 06serviceops, 10Wikidata, 10Discovery-Search (2025.05.24 - 2025.06.13), 13Patch-For-Review: Migrate discovery-search jobs to mw-cron - https://phabricator.wikimedia.org/T388538#10864869 (10Clement_Goubert) >>! In T388538#10851395, @Clement_Goubert wrote: > ` > cgoubert@deploy1003:~$ kubectl logs cirrus... [16:41:15] 06serviceops, 10Wikidata, 10Discovery-Search (2025.05.24 - 2025.06.13), 13Patch-For-Review: Migrate discovery-search jobs to mw-cron - https://phabricator.wikimedia.org/T388538#10864889 (10EBernhardson) >>! In T388538#10864868, @Clement_Goubert wrote: > Should we configure the jobs to keep going with t... [16:42:29] 06serviceops, 10DiscussionTools, 10MediaWiki-REST-API, 10RESTBase Sunsetting, and 3 others: Reroute RESTbase Endpoints for Linter Errors to MediaWiki REST Endpoints - https://phabricator.wikimedia.org/T384216#10864899 (10daniel) >>! In T384216#10863969, @hnowlan wrote: > There are also external hits for `v... [18:56:17] 06serviceops, 06SRE Observability: Stale labels applied when the pod IP of a terminated k8s job is reused - https://phabricator.wikimedia.org/T395052#10865493 (10Scott_French) 05Open→03Resolved Spot checking examples like the one in the task description (envoy metrics misattributed to mw-videoscaler) 2... [20:32:47] 06serviceops, 06Security-Team, 13Patch-For-Review, 07SecTeam-Processed: Migrate Security-Team jobs to mw-cron - https://phabricator.wikimedia.org/T388531#10865838 (10Krinkle) >>! In T388531#10864816, @Reedy wrote: >>>! In T388531#10864366, @Tgr wrote: >> So if the goal is to just not use `limit.sh` […] >... [20:53:00] 06serviceops, 10MW-on-K8s, 13Patch-For-Review: Allow running one-off scripts manually - https://phabricator.wikimedia.org/T341553#10865900 (10Zabe) I would like to point out that as long as there is no solution for T379675 at least I am dependent on mwmaint and ask that there is a solution for persistent dis... [21:08:19] 06serviceops, 06Security-Team, 13Patch-For-Review, 07SecTeam-Processed: Migrate Security-Team jobs to mw-cron - https://phabricator.wikimedia.org/T388531#10865923 (10Reedy) Do we log the commands run anywhere? As if that's the actual solution, to just explicitly set all the limits to 0... `lang=php public... [21:30:14] 06serviceops, 06MediaWiki-Engineering: Clean up UcfirstOverrides.php following PHP 7.4 -> 8.1 transition - https://phabricator.wikimedia.org/T394556#10865961 (10Scott_French) Thank you very much for confirming, Tim, and for noting the fact that the transitional overrides need removed promptly once we're done.... [21:31:52] 06serviceops, 06MediaWiki-Engineering: Clean up UcfirstOverrides.php following PHP 7.4 -> 8.1 transition - https://phabricator.wikimedia.org/T394556#10865967 (10Scott_French) [22:18:53] 06serviceops, 10MW-on-K8s: Members of restricted cannot use --dblist - https://phabricator.wikimedia.org/T395521 (10jrbs) 03NEW [22:21:42] 06serviceops, 10MW-on-K8s: Members of restricted cannot use --dblist - https://phabricator.wikimedia.org/T395521#10866103 (10Reedy) I'm wondering if this is because of the lack of being in the `deployment` group and/or because of being in `restricted`? I'm in `deployment` and was able to run the commands as m... [22:21:43] 06serviceops, 10MW-on-K8s: Allow members of restricted to run maintenance scripts - https://phabricator.wikimedia.org/T378429#10866101 (10Reedy)