[00:05:11] <wikibugs>	 06serviceops, 13Patch-For-Review: Migrate production Shellbox variants to PHP 8.1 - https://phabricator.wikimedia.org/T377038#10603578 (10Scott_French) Now that the `display_startup_errors` fix is available, I've started a second single-replica-per-DC pilot for shellbox-media.  I'll be checking throughout the...
[00:11:42] <wikibugs>	 06serviceops, 13Patch-For-Review: Migrate production Shellbox variants to PHP 8.1 - https://phabricator.wikimedia.org/T377038#10603609 (10Scott_French)
[02:12:29] <wikibugs>	 06serviceops, 13Patch-For-Review: Migrate production Shellbox variants to PHP 8.1 - https://phabricator.wikimedia.org/T377038#10603763 (10Scott_French) About two hours in, we've seen a handful of `POST Content-Length of {size} bytes exceeds the limit of 104857600 bytes in Unknown on line 0` warnings pop up in...
[07:40:17] <jelto>	 ^ ryans ingress change still has to be deployed in admin_ng. The diff (helmfile ...  diff --context 5 --selector name=namespace-certificates) looks good, just the new query endpoint is added. Any objections? jayme?
[08:44:29] <jayme>	 jelto: I think I lack context
[08:44:53] <jayme>	 but fine by me addind ingress endpoints/certs
[08:45:38] <jelto>	 In https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1122678 they added a new tlsExtraSANs "query-legacy-full.wikidata.org"
[08:47:04] <jelto>	 ok then I'll deploy admin-ng with --selector name=namespace-certificates in a moment
[09:05:49] <wikibugs>	 06serviceops, 06Release-Engineering-Team: deploy1003 reports  helmfileAdminPendingChanges - https://phabricator.wikimedia.org/T387900#10604216 (10JMeybohm) This is very common and happens for example on infra changes that are to be reflected in `external-services` deployments
[09:06:06] <jelto>	 ryankemper: the ingress change is deployed, see also my comment in the task T384422
[12:31:55] <wikibugs>	 06serviceops, 06Infrastructure-Foundations, 06SRE, 07Kubernetes: Remove `.cluster.local.` suffix in PTR responses - https://phabricator.wikimedia.org/T376762#10604979 (10MoritzMuehlenhoff)
[14:56:39] <kamila_>	 oh, thanks jelto <3 (I was just about to do it, had doctor earlier)
[15:13:14] <wikibugs>	 06serviceops, 10MW-on-K8s, 10Observability-Alerting, 10SRE Observability (FY2024/2025-Q3): Periodic job alerting - https://phabricator.wikimedia.org/T385709#10605761 (10fgiunchedi)
[15:37:32] <wikibugs>	 06serviceops, 06Infrastructure-Foundations, 10netops, 10Prod-Kubernetes: WikiKube clusters close to exhausting Calico IPPool allocations - https://phabricator.wikimedia.org/T375845#10605886 (10cmooney) FYI I've updated the prefix-list on our switches and routers in eqiad/codfw from the old /18 to the wider...
[16:04:28] <wikibugs>	 06serviceops, 07Datacenter-Switchover: Spicerack support for mw-cron in periodic_jobs functions - https://phabricator.wikimedia.org/T387753#10606063 (10hnowlan) a:03jasmine_
[17:18:00] <wikibugs>	 06serviceops, 06SRE Observability, 13Patch-For-Review: chartmuseum prometheus metrics cardinality spam - https://phabricator.wikimedia.org/T386808#10606601 (10kamila) > Maybe let's drop `url` label for `404` CM metrics for now, it seems like a good enough solution to me. I'm happy to assist/brainstorm on the...
[17:38:07] <wikibugs>	 06serviceops, 13Patch-For-Review: MediaWiki on PHP 8.1 production traffic ramp-up - https://phabricator.wikimedia.org/T383845#10606681 (10Scott_French) Next steps: 1. Right-size the main and next releases of mw-api-ext and mw-web, as they're both still considerably over-provisioned (more so main). 2. Begin cap...
[18:08:08] <wikibugs>	 06serviceops, 06collaboration-services, 06Data-Platform-SRE, 10Prod-Kubernetes, 07Kubernetes: Set cert-manager leader election namespace to cert-manager - https://phabricator.wikimedia.org/T383553#10606868 (10JMeybohm) p:05Triage→03High
[18:08:47] <wikibugs>	 06serviceops, 06collaboration-services, 06Data-Platform-SRE, 10Prod-Kubernetes, 07Kubernetes: Fix installed key in dependend helmfile releases - https://phabricator.wikimedia.org/T387837#10606880 (10JMeybohm) p:05Triage→03High
[18:10:35] <wikibugs>	 06serviceops, 10Prod-Kubernetes, 10Wikifunctions, 10Abstract Wikipedia team (25Q3 (Jan–Mar)), 07Kubernetes: Migrate deprecated apparmor.security.beta.kubernetes.io annotations to SecurityContext - https://phabricator.wikimedia.org/T384429#10606889 (10JMeybohm) →14Duplicate dup:03T367880
[18:10:36] <wikibugs>	 06serviceops, 10Prod-Kubernetes, 07Kubernetes: Set AppArmor profile via SecurityContext rather than annotations (k8s >=1.30) - https://phabricator.wikimedia.org/T367880#10606891 (10JMeybohm)
[18:11:11] <wikibugs>	 06serviceops, 06Infrastructure-Foundations, 10netops, 10Prod-Kubernetes: WikiKube clusters close to exhausting Calico IPPool allocations - https://phabricator.wikimedia.org/T375845#10606894 (10JMeybohm) p:05Medium→03High
[18:12:35] <wikibugs>	 06serviceops, 07Kubernetes: Add pod ip address blocks to staging-eqiad - https://phabricator.wikimedia.org/T386232#10606898 (10JMeybohm) @cmooney this probably needs a prefix update on "your" side as well, right (like T375845)?
[23:51:46] <wikibugs>	 06serviceops, 13Patch-For-Review: MediaWiki on PHP 8.1 production traffic ramp-up - https://phabricator.wikimedia.org/T383845#10608161 (10Scott_French) I was able to wrap up the right-sizing described in T383845#10606681 this afternoon with https://gerrit.wikimedia.org/r/1124848 and some additional tweaks in h...