[06:10:45] 06serviceops, 10Arc-Lamp: Gather PHP 8.1 profiling data - https://phabricator.wikimedia.org/T385199#10655786 (10ori) Ack, thanks for the heads up. [07:10:42] 06serviceops, 06Data-Engineering, 06Data-Engineering-Radar, 10Dumps-Generation, and 2 others: Migrate WMF production from PHP 7.4 to PHP 8.1 - https://phabricator.wikimedia.org/T319432#10655908 (10BTullis) I am rolling out PHP version 8.1 to all snapshot hosts except snapshot1016. They should all start the... [08:37:30] hnowlan: o/ if you have some old memories of imposm on maps: https://phabricator.wikimedia.org/T389462 [08:37:45] not sure if you have ever seen the issue [09:06:37] 06serviceops, 10Observability-Alerting, 07Kubernetes, 10SRE Observability (FY2024/2025-Q3): Alert on unscrapable pods - https://phabricator.wikimedia.org/T372242#10656143 (10fgiunchedi) [09:12:58] 06serviceops, 10Observability-Alerting, 07Kubernetes, 10SRE Observability (FY2024/2025-Q3): Alert on unscrapable pods - https://phabricator.wikimedia.org/T372242#10656168 (10fgiunchedi) [09:14:48] 06serviceops, 10Observability-Alerting, 07Kubernetes, 10SRE Observability (FY2024/2025-Q3): Alert on unscrapable pods - https://phabricator.wikimedia.org/T372242#10656176 (10fgiunchedi) [10:09:28] 06serviceops, 06Data-Engineering, 06Data-Engineering-Radar, 10Dumps-Generation, and 2 others: Migrate WMF production from PHP 7.4 to PHP 8.1 - https://phabricator.wikimedia.org/T319432#10656294 (10BTullis) 7 of the 8 snapshot servers are now running PHP 8.1 for dumps. ` btullis@cumin1002:~$ sudo cumin A:sn... [10:54:24] 06serviceops, 10Cassandra, 13Patch-For-Review: mediawiki: migrate from image-suggestion to data-gateway - https://phabricator.wikimedia.org/T368096#10656490 (10Cparle) Hey Eric - we've had big problems with the data pipeline since Jan (flaky upstream dependencies), and we kinda have to deal with those before... [11:01:15] 06serviceops, 10Observability-Alerting, 07Kubernetes, 10SRE Observability (FY2024/2025-Q3): Alert on unscrapable pods - https://phabricator.wikimedia.org/T372242#10656520 (10fgiunchedi) [11:03:36] 06serviceops, 10Observability-Alerting, 07Kubernetes, 10SRE Observability (FY2024/2025-Q3): Alert on unscrapable pods - https://phabricator.wikimedia.org/T372242#10656523 (10fgiunchedi) [11:42:34] 06serviceops, 10Observability-Alerting, 07Kubernetes, 10SRE Observability (FY2024/2025-Q3): mcrouter declares unscrapable port to prometheus - https://phabricator.wikimedia.org/T389480 (10Clement_Goubert) 03NEW [12:20:18] 06serviceops, 10Observability-Alerting, 07Kubernetes, 13Patch-For-Review, 10SRE Observability (FY2024/2025-Q3): mcrouter and thumbor declare unscrapable ports to prometheus - https://phabricator.wikimedia.org/T389480#10656765 (10jijiki) [12:33:05] 06serviceops, 06Data-Engineering, 10Dumps-Generation, 06Experimentation Lab, and 3 others: Create a mediawiki-cli image - https://phabricator.wikimedia.org/T389484 (10Clement_Goubert) 03NEW [12:34:06] 06serviceops, 06Data-Engineering, 10Dumps-Generation, 06Experimentation Lab, and 3 others: Create a mediawiki-cli image - https://phabricator.wikimedia.org/T389484#10656829 (10Clement_Goubert) p:05Triage→03High [12:34:46] 06serviceops, 06Data-Engineering, 10Dumps-Generation, 06Experimentation Lab, and 3 others: Create a mediawiki-cli image - https://phabricator.wikimedia.org/T389484#10656831 (10Clement_Goubert) [12:34:46] 06serviceops, 10MW-on-K8s, 13Patch-For-Review: Implement periodic maintenance scripts for mw-on-k8s - https://phabricator.wikimedia.org/T341555#10656832 (10Clement_Goubert) [12:34:47] 06serviceops, 10MW-on-K8s: Allow running one-off scripts manually - https://phabricator.wikimedia.org/T341553#10656833 (10Clement_Goubert) [12:43:33] 06serviceops, 06Data-Engineering, 10Dumps-Generation, 06Experimentation Lab, and 3 others: Create a mediawiki-cli image - https://phabricator.wikimedia.org/T389484#10656844 (10Clement_Goubert) [12:43:58] 06serviceops, 06Data-Engineering, 10Dumps-Generation, 06Experimentation Lab, and 3 others: Create a mediawiki-cli image - https://phabricator.wikimedia.org/T389484#10656845 (10Clement_Goubert) [13:41:55] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: wipe-cluster cookbook should check if systemd services have started properly - https://phabricator.wikimedia.org/T389086#10657113 (10Gehel) Removing DPE SRE as I don't think we need to be involved. Please add us again if needed. [13:42:50] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Fix dependencies between admin_ng deployments - https://phabricator.wikimedia.org/T389080#10657125 (10Gehel) Removing DPE SRE as it does not seem that we need to be involved. Please add us again if you need us to do something. [13:43:55] 06serviceops, 06collaboration-services, 10Prod-Kubernetes, 07Kubernetes: Update kube-state-metrics for k8s 1.31 - https://phabricator.wikimedia.org/T388387#10657132 (10Gehel) Removing DPE SRE as it does not seem that we need to be involved. Please add us again if you need us to do something. [14:00:04] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q3:rack/setup/install wikikube-worker2248-2331, wikikube-ctrl2004-2005 - https://phabricator.wikimedia.org/T384970#10657175 (10Jhancock.wm) [14:25:38] 06serviceops, 10Cassandra, 06Content-Transform-Team: restbase service crashing - https://phabricator.wikimedia.org/T389410#10657243 (10Jgiannelos) [15:43:06] 06serviceops, 06Data-Engineering, 10Dumps-Generation, 06Experimentation Lab, and 4 others: Create a mediawiki-cli image - https://phabricator.wikimedia.org/T389484#10657560 (10Clement_Goubert) Build succeeded, the image is currently being tested by @brouberol If it's conclusive, we can resolve this, and w... [15:48:20] 06serviceops, 10MW-on-K8s, 06Release-Engineering-Team: Refactor scap's kubernetes DeploymentsConfig - https://phabricator.wikimedia.org/T389499 (10Clement_Goubert) 03NEW [15:53:06] 06serviceops, 06Data-Engineering, 10Dumps-Generation, 06Experimentation Lab, and 4 others: Create a mediawiki-cli image - https://phabricator.wikimedia.org/T389484#10657651 (10Clement_Goubert) 05Open→03Resolved a:03Clement_Goubert Image has what's needed for `dumps`, resolving. [16:29:08] 06serviceops, 10MW-on-K8s: Convert captchaloop to kubernetes CronJob - https://phabricator.wikimedia.org/T380167#10657775 (10Clement_Goubert) →14Duplicate dup:03T388531 [16:29:10] 06serviceops, 06Security-Team, 07SecTeam-Processed: Migrate Security-Team jobs to mw-cron - https://phabricator.wikimedia.org/T388531#10657777 (10Clement_Goubert) [16:30:55] 06serviceops, 06Security-Team, 07SecTeam-Processed: Migrate Security-Team jobs to mw-cron - https://phabricator.wikimedia.org/T388531#10657798 (10Clement_Goubert) {T389484} adds a way for us to have `python3` installed in a non-web facing image, I've already added `python3-pil` to the build process. I need t... [16:56:47] 06serviceops, 10Observability-Alerting, 07Kubernetes, 13Patch-For-Review, 10SRE Observability (FY2024/2025-Q3): mcrouter and thumbor declare unscrapable ports to prometheus - https://phabricator.wikimedia.org/T389480#10657951 (10jijiki) When using `values.monitoring.named_ports: true`, adds the correct a... [17:34:35] 06serviceops, 10MW-on-K8s, 06Release-Engineering-Team: Refactor scap's kubernetes DeploymentsConfig - https://phabricator.wikimedia.org/T389499#10658190 (10Scott_French) Taking a step back, there are a couple of ways we could go about this. IMO, the two most obvious are as follows: **One option** is what we... [17:59:24] 06serviceops, 10MW-on-K8s, 06Release-Engineering-Team: Refactor scap's kubernetes DeploymentsConfig to support selection of image kinds - https://phabricator.wikimedia.org/T389499#10658458 (10Scott_French) [18:09:22] 06serviceops, 10MW-on-K8s, 06Release-Engineering-Team: Refactor scap's kubernetes DeploymentsConfig to support selection of image kinds - https://phabricator.wikimedia.org/T389499#10658653 (10Scott_French) [18:27:02] 06serviceops, 13Patch-For-Review: Align mw-on-k8s alerts with capacity pools - https://phabricator.wikimedia.org/T389224#10658752 (10Scott_French) Enough time has passed since https://gerrit.wikimedia.org/r/1129358 was merged that it should be live. I've cleared the silence the silence (`7112e3a2-4430-401a-b5d... [19:09:16] 06serviceops, 13Patch-For-Review: MediaWiki on PHP 8.1 production traffic ramp-up - https://phabricator.wikimedia.org/T383845#10658925 (10Scott_French) Alright, after a bit or manual testing (h/t to Joe for doing so as well), mw-misc - i.e., [[ https://noc.wikimedia.org | noc.wikimedia.org ]] - appears to be w... [19:09:54] 06serviceops, 13Patch-For-Review: MediaWiki on PHP 8.1 production traffic ramp-up - https://phabricator.wikimedia.org/T383845#10658926 (10Scott_French) [21:02:57] 06serviceops: Align mw-on-k8s alerts with capacity pools - https://phabricator.wikimedia.org/T389224#10659354 (10Scott_French) 05Open→03Resolved