[06:52:02] 06serviceops, 06MediaWiki-Platform-Team, 07Epic: Migrate Wikimedia production from PHP 8.1 to PHP 8.3 - https://phabricator.wikimedia.org/T360995#11067197 (10Jdforrester-WMF) > 4. MW: Make CI for MediaWiki core and WMF extensions pass on PHP 8.3. T353362, Jun 2024 This is not fixed yet. It's still waiting... [08:40:20] 06serviceops, 10MW-on-K8s: Turn down mwmaint production servers - https://phabricator.wikimedia.org/T397017#11067429 (10hashar) As part of deploying the train, `deploy1003` had a disk space alarm for `/srv`: ` CRITICAL: DISK CRITICAL - free space: /srv 8607 MB (3% inode=69%) ` ` name=df -h /srv Filesystem... [09:51:44] 06serviceops, 10ChangeProp, 10Page Content Service, 13Patch-For-Review: Investigate significant changeprop backlogs after PCS migration - https://phabricator.wikimedia.org/T397072#11067583 (10Jgiannelos) 05Open→03Resolved a:03Jgiannelos Backlog has been consistency low manageable for the past few... [11:02:52] 06serviceops, 10MW-on-K8s: Turn down mwmaint production servers - https://phabricator.wikimedia.org/T397017#11067803 (10Clement_Goubert) I plan on removing the homedirs soon-ish, but I will add 30GB to the vg to negate the issue in the meantime. [11:03:58] 06serviceops, 10MW-on-K8s: Turn down mwmaint production servers - https://phabricator.wikimedia.org/T397017#11067807 (10Clement_Goubert) 05Open→03Resolved [11:05:09] 06serviceops, 10MW-on-K8s: Document the new periodic cronjobs infrastructure on Wikitech - https://phabricator.wikimedia.org/T385783#11067811 (10Clement_Goubert) 05In progress→03Resolved [11:06:14] 06serviceops, 10MW-on-K8s: Migrate mwmaint server functionality to mw-on-k8s - https://phabricator.wikimedia.org/T341560#11067819 (10Clement_Goubert) 05Open→03Resolved a:03Clement_Goubert All main use cases of `mwmaint` servers are now migrated to either `mw-cron` or `mwscript-k8s`. Resolving, the st... [11:20:14] 06serviceops, 10MW-on-K8s: Turn down mwmaint production servers - https://phabricator.wikimedia.org/T397017#11067833 (10Clement_Goubert) ` root@deploy1003:~# lvextend -L+30G /dev/vg0/srv Size of logical volume vg0/srv changed from <281.92 GiB (72171 extents) to <311.92 GiB (79851 extents). Logical volu... [12:30:58] 06serviceops, 06collaboration-services, 06Data-Platform-SRE, 10Prod-Kubernetes, and 2 others: Update Kubernetes clusters to 1.31 - https://phabricator.wikimedia.org/T341984#11068042 (10Clement_Goubert) [14:21:38] 06serviceops, 10MW-on-K8s: Turn down mwmaint production servers - https://phabricator.wikimedia.org/T397017#11068484 (10hashar) I am happy to see there was plenty of disk space on the volume group \o/ Thank you! [14:28:53] 06serviceops, 07Epic, 10MW-Interfaces-Team (MWI-Roadmap), 07OKR-Work: Revisit backend routing for rest-gateway - https://phabricator.wikimedia.org/T401396 (10Clement_Goubert) 03NEW [15:07:58] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q1:rack/setup/install deploy2003 - https://phabricator.wikimedia.org/T400485#11068783 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin1003 for host deploy2003.codfw.wmnet with OS bookworm [15:48:12] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q1:rack/setup/install deploy2003 - https://phabricator.wikimedia.org/T400485#11068941 (10Jhancock.wm) a:03Jhancock.wm [16:26:39] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q1:rack/setup/install deploy2003 - https://phabricator.wikimedia.org/T400485#11069127 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin1003 for host deploy2003.codfw.wmnet with OS bookworm executed with errors: - deploy2003 (**... [16:45:10] 06serviceops: Configure title-case consistency mapping for PHP 8.1 -> 8.3 transition - https://phabricator.wikimedia.org/T401252#11069244 (10Scott_French) 05In progress→03Resolved [16:45:51] 06serviceops: Upgrade mw-debug/next to PHP 8.3 - https://phabricator.wikimedia.org/T401254#11069251 (10Scott_French) [17:34:45] 06serviceops, 10Dumps-Generation, 10MW-on-K8s: Investigate memcache errors during wikidata and commons dumps runs - https://phabricator.wikimedia.org/T401425 (10Scott_French) 03NEW [17:34:50] 06serviceops, 10Dumps-Generation, 10MW-on-K8s: Investigate memcache errors during wikidata and commons dumps runs - https://phabricator.wikimedia.org/T401425#11069375 (10Scott_French)