[07:09:28] 06serviceops, 10Dumps-Generation, 06MediaWiki-Platform-Team: Migrate WMF production from PHP 7.4 to PHP 8.1 - https://phabricator.wikimedia.org/T319432#10347278 (10kostajh) [[ https://www.php.net/archive/2024.php#2024-11-21-4 | PHP 8.4 was released yesterday ]], and PHP 8.1 is now receiving critical security... [07:32:15] 06serviceops, 10Dumps-Generation, 06MediaWiki-Platform-Team: Migrate WMF production from PHP 7.4 to PHP 8.1 - https://phabricator.wikimedia.org/T319432#10347290 (10MoritzMuehlenhoff) >>! In T319432#10347278, @kostajh wrote: > [[ https://www.php.net/archive/2024.php#2024-11-21-4 | PHP 8.4 was released yesterd... [08:48:27] 06serviceops, 10Dumps-Generation, 06MediaWiki-Platform-Team: Migrate WMF production from PHP 7.4 to PHP 8.1 - https://phabricator.wikimedia.org/T319432#10347333 (10kostajh) >>! In T319432#10347290, @MoritzMuehlenhoff wrote: >>>! In T319432#10347278, @kostajh wrote: >> [[ https://www.php.net/archive/2024.php#... [09:51:11] 06serviceops, 06Commons, 10WMF-JobQueue: Temporarily run more refreshLinks jobs on Commons - https://phabricator.wikimedia.org/T380544#10347422 (10akosiaris) This type of Kafka Consumer lag for that job isn't unheard of. In fact, just recently we had way [higher consumer lags](https://grafana.wikimedia.org/d... [09:57:59] 06serviceops: The mwdebug cluster has inconsistent AAAA DNS records for the primary IPv6 of the hosts - https://phabricator.wikimedia.org/T380254#10347433 (10akosiaris) 05Open→03Declined >>! In T380254#10335636, @akosiaris wrote: > The cluster is slated to eventually be decommissioned and sooner rather t... [09:58:30] 06serviceops, 06DC-Ops, 06Infrastructure-Foundations, 10netops, and 4 others: Reimage one of the wikikube-worker1240 to wikikube-worker1304 node in eqiad as a replacement for wikikube-ctrl1001 - https://phabricator.wikimedia.org/T379790#10347436 (10akosiaris) Cool thanks, I 'll take over this one. [09:59:19] 06serviceops, 06MW-Interfaces-Team, 10RESTBase Sunsetting, 10MW-1.44-notes (1.44.0-wmf.4; 2024-11-19), and 2 others: Switchover plan from RESTbase to REST Gateway for rest_v1/page/html and rest_v1/page/title endpoints - https://phabricator.wikimedia.org/T374683#10347438 (10akosiaris) >>! In T374683#1034617... [10:19:15] 06serviceops: wikikube-worker21[56-70] implementation tracking - https://phabricator.wikimedia.org/T376966#10347533 (10Clement_Goubert) [10:19:36] 06serviceops: wikikube-worker21[36-55] implementation tracking - https://phabricator.wikimedia.org/T377028#10347534 (10Clement_Goubert) [10:21:29] 06serviceops: wikikube-worker21[56-70] implementation tracking - https://phabricator.wikimedia.org/T376966#10347537 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1002 for host wikikube-worker2159.codfw.wmnet with OS bookworm [10:34:28] 06serviceops, 06DC-Ops, 10ops-eqiad: Degraded RAID on wikikube-worker1256 - https://phabricator.wikimedia.org/T379454#10347563 (10Clement_Goubert) Re-imaging because I accidentaly overwrote the partition table on the good disk with the partition table on the new disk... [11:02:18] 06serviceops: wikikube-worker21[56-70] implementation tracking - https://phabricator.wikimedia.org/T376966#10347650 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1002 for host wikikube-worker2159.codfw.wmnet with OS bookworm completed: - wikikube-worker2159 (**PASS**) - D... [11:03:26] 06serviceops: wikikube-worker21[56-70] implementation tracking - https://phabricator.wikimedia.org/T376966#10347652 (10Clement_Goubert) [11:06:11] 06serviceops, 06Data-Persistence, 10Prod-Kubernetes: Reevaluate the requirement for dedicated sessionstore/kask nodes in wikikube clusters - https://phabricator.wikimedia.org/T379599#10347656 (10Clement_Goubert) I'm good with removing them as well. [11:09:25] 06serviceops: wikikube-worker21[36-55] implementation tracking - https://phabricator.wikimedia.org/T377028#10347676 (10Clement_Goubert) [11:09:28] 06serviceops: wikikube-worker21[36-55] implementation tracking - https://phabricator.wikimedia.org/T377028#10347677 (10ops-monitoring-bot) pool host wikikube-worker2140.codfw.wmnet by cgoubert@cumin1002 with reason: New nodes [11:10:22] 06serviceops: wikikube-worker21[36-55] implementation tracking - https://phabricator.wikimedia.org/T377028#10347681 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by cgoubert@cumin1002 pool for host wikikube-worker2140.codfw.wmnet completed: - wikikube-worker2140.codfw.wmnet (**PA... [11:10:33] 06serviceops: wikikube-worker21[36-55] implementation tracking - https://phabricator.wikimedia.org/T377028#10347683 (10Clement_Goubert) 05Stalled→03Resolved a:03Clement_Goubert [11:16:16] 06serviceops: wikikube-worker21[56-70] implementation tracking - https://phabricator.wikimedia.org/T376966#10347714 (10Clement_Goubert) [11:26:24] 06serviceops: wikikube-worker21[56-70] implementation tracking - https://phabricator.wikimedia.org/T376966#10347750 (10ops-monitoring-bot) pool host wikikube-worker[2156-2170].codfw.wmnet by cgoubert@cumin1002 with reason: None [11:26:30] 06serviceops: wikikube-worker21[56-70] implementation tracking - https://phabricator.wikimedia.org/T376966#10347751 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by cgoubert@cumin1002 pool for host wikikube-worker[2156-2170].codfw.wmnet completed: - wikikube-worker[2156-2170].codfw.w... [11:26:51] 06serviceops: wikikube-worker21[56-70] implementation tracking - https://phabricator.wikimedia.org/T376966#10347752 (10Clement_Goubert) 05In progress→03Resolved [13:19:00] 06serviceops, 10Dumps-Generation, 06MediaWiki-Platform-Team: Migrate WMF production from PHP 7.4 to PHP 8.1 - https://phabricator.wikimedia.org/T319432#10348042 (10Reedy) Noting also that getting the k8s stuff done was done before this in attempt to make future upgrades easier.. I’m not sure we’re immediate... [13:24:12] 06serviceops, 06Commons, 10WMF-JobQueue: Temporarily run more refreshLinks jobs on Commons - https://phabricator.wikimedia.org/T380544#10348050 (10Nikki) > @LucasWerkmeister how was the 10 year estimation calculated? I've been using the number of results of a `hastemplate:` search between two points in time... [13:42:47] 06serviceops, 06collaboration-services, 06Data-Platform-SRE, 10Prod-Kubernetes, 07Kubernetes: Update Kubernetes clusters to >1.25 - https://phabricator.wikimedia.org/T341984#10348073 (10Jelto) [13:53:03] 06serviceops: Package prometheus-mcrouter-exporter v0.4.0 - https://phabricator.wikimedia.org/T380212#10348140 (10jijiki) [14:25:41] 06serviceops, 07Kubernetes: Add 2 more nodes per DC to wikikube-staging - https://phabricator.wikimedia.org/T380043#10348213 (10JMeybohm) 05Open→03Resolved a:03JMeybohm Staging clusters have 4 nodes each now. Resolving this [14:50:09] 06serviceops: Decommission parse20[01-20] - https://phabricator.wikimedia.org/T380473#10348312 (10ops-monitoring-bot) depool host parse[2001-2020].codfw.wmnet by cgoubert@cumin1002 with reason: decom [14:59:46] 06serviceops: Decommission parse20[01-20] - https://phabricator.wikimedia.org/T380473#10348396 (10Clement_Goubert) [15:09:49] 06serviceops, 13Patch-For-Review: Decommission parse20[01-20] - https://phabricator.wikimedia.org/T380473#10348465 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by cgoubert@cumin1002 depool for host parse[2001-2020].codfw.wmnet completed: - parse[2001-2020].codfw.wmnet (**PASS**)... [15:30:00] 06serviceops, 13Patch-For-Review: Decommission parse20[01-20] - https://phabricator.wikimedia.org/T380473#10348605 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by cgoubert@cumin1002 for hosts: `parse2001.codfw.wmnet` - parse2001.codfw.wmnet (**PASS**) - Downtimed host on Icinga/Alertmanag... [15:50:03] 06serviceops, 06Content-Transform-Team, 06MediaWiki-Engineering, 06MW-Interfaces-Team, and 3 others: Transition parsoidtest1001 to PHP 8.1 - https://phabricator.wikimedia.org/T380485#10348779 (10larissagaulia) [16:11:11] 06serviceops, 13Patch-For-Review: Decommission parse20[01-20] - https://phabricator.wikimedia.org/T380473#10348987 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by cgoubert@cumin1002 for hosts: `parse[2002-2020].codfw.wmnet` - parse2002.codfw.wmnet (**PASS**) - Downtimed host on Icinga/Ale... [16:18:28] 06serviceops, 06Infrastructure-Foundations, 10netops, 10Prod-Kubernetes: WikiKube clusters close to exhausting Calico IPPool allocations - https://phabricator.wikimedia.org/T375845#10349021 (10JMeybohm) This did bite us again and we had to {T380473} in a hurry. Quick fix to free up IPAM blocks without deco... [16:23:35] 06serviceops, 06DC-Ops, 10decommission-hardware, 10ops-codfw: Decommission parse20[01-20] - https://phabricator.wikimedia.org/T380473#10349027 (10Clement_Goubert) [16:29:34] 06serviceops, 06Infrastructure-Foundations, 10netops, 10Prod-Kubernetes: WikiKube clusters close to exhausting Calico IPPool allocations - https://phabricator.wikimedia.org/T375845#10349049 (10JMeybohm) >>! In T375845#10246786, @akosiaris wrote: > Now, for the actual change. The biggest issue is coordinati... [21:14:15] 06serviceops, 07Wikimedia-production-error: Spike of impossible "Cannot declare class" fatal errors (opcache) - https://phabricator.wikimedia.org/T254209#10349911 (10Paladox) This looks similar to T378006 which there's a patch for https://github.com/wikimedia/mediawiki/commit/8f7b4abd751f9f4e402b104f19b1e48559...