[02:29:56] 06serviceops, 06Content-Transform-Team, 06MediaWiki-Engineering, 07OKR-Work: Migrate parsoidtest functionality to kubernetes - https://phabricator.wikimedia.org/T386246#10720004 (10Scott_French) In the interest of having a solution ready in the very near term, I'd like to propose that we move ahead with th... [02:30:25] 06serviceops, 06Content-Transform-Team, 06MediaWiki-Engineering, 07OKR-Work: Migrate parsoidtest functionality to kubernetes - https://phabricator.wikimedia.org/T386246#10720005 (10Scott_French) [10:19:21] 06serviceops, 06MediaWiki-Platform-Team: Migrate "startupregistrystats" maintenance script to k8s-mw-cron (mediawiki-platform-team) - https://phabricator.wikimedia.org/T388540#10720811 (10Clement_Goubert) [10:49:22] 06serviceops, 06Trust and Safety Product Team: Migrate trust_and_safety_product_team jobs to mw-cron - https://phabricator.wikimedia.org/T388542#10720869 (10kostajh) >>! In T388542#10667863, @Dreamy_Jazz wrote: >>>! In T388542#10667714, @Clement_Goubert wrote: >> I was looking at setting up {T385709}, and real... [11:04:19] 06serviceops, 06Growth-Team, 10GrowthExperiments, 10MW-on-K8s, 13Patch-For-Review: Migrate GrowthExperiments maintenance jobs to mw-cron - https://phabricator.wikimedia.org/T385782#10720899 (10Clement_Goubert) `growthexperiments-deleteOldSurveys` migrated to k8s. Do you mind if I do a manual run to test... [11:05:50] 06serviceops, 06Growth-Team, 10GrowthExperiments, 10MW-on-K8s, 13Patch-For-Review: Migrate GrowthExperiments maintenance jobs to mw-cron - https://phabricator.wikimedia.org/T385782#10720905 (10Clement_Goubert) [11:10:10] 06serviceops, 10Page Content Service, 10RESTBase Sunsetting, 10Content-Transform-Team (Work In Progress): Rollout more wikis: week 3 - https://phabricator.wikimedia.org/T390724#10720924 (10MSantos) [11:12:19] 06serviceops, 10Page Content Service, 10RESTBase Sunsetting, 10Content-Transform-Team (Work In Progress): Rollout more wikis: week 3 - https://phabricator.wikimedia.org/T390724#10720930 (10MSantos) [11:27:19] 06serviceops, 06DC-Ops, 10ops-codfw: hw troubleshooting: hard down for wikikube-worker2142 - https://phabricator.wikimedia.org/T391341 (10Clement_Goubert) 03NEW [11:28:05] 06serviceops, 06DC-Ops, 10ops-codfw: hw troubleshooting: hard down for wikikube-worker2142 - https://phabricator.wikimedia.org/T391341#10720990 (10ops-monitoring-bot) depool host wikikube-worker2142.codfw.wmnet by cgoubert@cumin1002 with reason: Hardware failure [11:28:56] 06serviceops, 06DC-Ops, 10ops-codfw: hw troubleshooting: hard down for wikikube-worker2142 - https://phabricator.wikimedia.org/T391341#10721004 (10Clement_Goubert) [11:29:20] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: hw troubleshooting: hard down for wikikube-worker2142 - https://phabricator.wikimedia.org/T391341#10721007 (10Clement_Goubert) a:03Papaul [11:33:04] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: hw troubleshooting: hard down for wikikube-worker2142 - https://phabricator.wikimedia.org/T391341#10721015 (10Clement_Goubert) Host drained forcefully and depooled. [12:45:26] 06serviceops, 06Trust and Safety Product Team, 13Patch-For-Review: Migrate trust_and_safety_product_team jobs to mw-cron - https://phabricator.wikimedia.org/T388542#10721300 (10fgiunchedi) By default one task per alert name, the task title is the alert name. When multiple alerts fire with the same name then... [13:39:05] 06serviceops, 06Content-Transform-Team, 07Epic, 10Maps (Kartotherian): Move Kartotherian to Kubernetes - https://phabricator.wikimedia.org/T216826#10721508 (10elukey) 05Open→03Resolved a:03elukey Despite a memory leak tracked in T389776, Kartotherian now runs only on k8s and we have cleaned up ev... [13:59:55] 06serviceops, 06Trust and Safety Product Team, 13Patch-For-Review: Migrate trust_and_safety_product_team jobs to mw-cron - https://phabricator.wikimedia.org/T388542#10721612 (10Clement_Goubert) >>! In T388542#10721300, @fgiunchedi wrote: > By default one task per alert name, the task title is the alert name.... [14:13:50] 06serviceops, 10MW-on-K8s, 10Observability-Alerting, 13Patch-For-Review, 10SRE Observability (FY2024/2025-Q3): Periodic job alerting - https://phabricator.wikimedia.org/T385709#10721792 (10Clement_Goubert) Updated https://wikitech.wikimedia.org/wiki/Periodic_jobs#Monitoring to be clearer on what alerting... [15:30:45] 06serviceops, 10MW-Interfaces-Team (MWI-Sprint-6 (2025-03-25 to 2025-04-08)): Migrate mw-interfaces-team jobs to mw-cron - https://phabricator.wikimedia.org/T388541#10722321 (10HCoplin-WMF) [15:32:45] 06serviceops, 10MW-Interfaces-Team (MWI-Sprint-7 (2025-04-08 to 2025-04-22)): Migrate mw-interfaces-team jobs to mw-cron - https://phabricator.wikimedia.org/T388541#10722339 (10HCoplin-WMF) [16:02:39] 06serviceops: docker-registry.wikimedia.org keeps serving bad blobs - https://phabricator.wikimedia.org/T390251#10722539 (10dancy) Train presync failed last week due to this issue, but it ran successfully this week. [17:59:15] 06serviceops, 10Scap: Migrate scap's maintenance script invocations to PHP 8.1 - https://phabricator.wikimedia.org/T390225#10722976 (10Scott_French) I was able to use `-Dmediawiki_runtime_image:docker-registry.wikimedia.org/php8.1-fpm-multiversion-base` (thanks @dancy for reminding me this was possible) to swi... [18:31:01] 06serviceops, 10Deployments, 10Release-Engineering-Team (Radar), 07Wikimedia-production-error: httpb sometimes fails upon deployment with a HTTP 503 - https://phabricator.wikimedia.org/T380958#10723184 (10brennen) Hit one of these again today. {P74744} [18:47:32] 06serviceops, 06Release-Engineering-Team, 10Scap, 06SRE-OnFire, and 2 others: Should scap be able to update helmfile-defaults when -Dbuild_mw_container_image:False ? - https://phabricator.wikimedia.org/T390531#10723243 (10dancy) scap 4.152.0 has been deployed to address the `update_helmfile_files()` issue. [21:07:15] 06serviceops, 13Patch-For-Review: MediaWiki on PHP 8.1 production traffic ramp-up - https://phabricator.wikimedia.org/T383845#10723801 (10Jdforrester-WMF) Is this as-written Resolved? The remaining work AIUI is on migrating the last remaining cron jobs over to the new infrastructure (T341555). [21:25:13] 06serviceops, 13Patch-For-Review: Build php-uuid package, and add to WMF production and CI - https://phabricator.wikimedia.org/T373752#10723913 (10Reedy) I guess at this point, we probably really don't care about getting this for PHP 7.4 ({T391057} etc)... So mostly it's just getting this into our CI images l... [22:20:07] 06serviceops, 10Scap: Migrate scap's maintenance script invocations to PHP 8.1 - https://phabricator.wikimedia.org/T390225#10724078 (10Scott_French) Alright, after a bit of head-scratching, I believe we're good to go with this. Specifically, I initially did not realize that the `cdb` files embed timestamps fo... [23:35:19] 06serviceops, 13Patch-For-Review: MediaWiki on PHP 8.1 production traffic ramp-up - https://phabricator.wikimedia.org/T383845#10724254 (10Scott_French) 05In progress→03Resolved Thanks, @Jdforrester-WMF - Indeed it is, though I was holding onto it for a bit to track immediate follow-on cleanups to get o... [23:51:51] 06serviceops: Clean up and abstract PHP_ENGINE 8.1 routing - https://phabricator.wikimedia.org/T391421 (10Scott_French) 03NEW [23:51:56] 06serviceops: Clean up and abstract PHP_ENGINE 8.1 routing - https://phabricator.wikimedia.org/T391421#10724306 (10Scott_French) p:05Triage→03Low