[00:48:11] 06serviceops, 06Content-Transform-Team-WIP, 10MW-on-K8s, 06SRE, and 4 others: A lot of `[info] Wikitext for this page has duplicate ids:` in logstash for mw-parsoid. Possibly related to PageBundle - https://phabricator.wikimedia.org/T358588#10220378 (10ABreault-WMF) a:03ABreault-WMF [07:01:29] 06serviceops, 10MW-on-K8s, 10Sustainability (Incident Followup): Remove memory limits from critical cluster components (calico) - https://phabricator.wikimedia.org/T376976 (10JMeybohm) 03NEW [07:01:50] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 10Sustainability (Incident Followup): Remove memory limits from critical cluster components (calico) - https://phabricator.wikimedia.org/T376976#10220527 (10JMeybohm) [07:05:06] 06serviceops, 06Data-Persistence, 13Patch-For-Review: Sessionstore's discovery TLS cert will expire before end of May 2024 - https://phabricator.wikimedia.org/T363996#10220536 (10elukey) @hnowlan if echostore turns out to work as expected (it sounds so from the other task), we could keep the ball rolling and... [07:15:15] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 10Sustainability (Incident Followup): Remove memory limits from critical cluster components (calico) - https://phabricator.wikimedia.org/T376976#10220538 (10akosiaris) We 've already discussed this in a 1on1 and just for transparency's sake, this finds me in a... [08:05:15] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 10Sustainability (Incident Followup): Remove memory limits from critical cluster components (calico) - https://phabricator.wikimedia.org/T376976#10220597 (10JMeybohm) p:05Triage→03High a:03JMeybohm [08:11:29] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 10Sustainability (Incident Followup): Remove memory limits from critical cluster components (calico) - https://phabricator.wikimedia.org/T376976#10220618 (10Joe) >>! In T376976#10220538, @akosiaris wrote: > We 've already discussed this in a 1on1 and just for... [09:38:59] 06serviceops, 06Infrastructure-Foundations, 06SRE: Clean up the Docker Registry catalog and Swift storage from old images - https://phabricator.wikimedia.org/T375645#10220815 (10elukey) I've created a Python script to dry-run what I highlighted above, this is how it would look like: ==== No tags, registryct... [10:29:19] 06serviceops, 10MW-on-K8s, 13Patch-For-Review, 10Sustainability (Incident Followup): mwscript-k8s creates too many resources - https://phabricator.wikimedia.org/T376795#10220900 (10JMeybohm) I've confirmed in the staging cluster that creating 1k network-policies (even if they don't apply to anything) bump... [10:35:22] 06serviceops, 06SRE, 13Patch-For-Review: mw2420-mw2451 do have unnecessary raid controllers (configured) - https://phabricator.wikimedia.org/T358489#10220912 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1002 for host wikikube-worker2092.codfw.wmnet with OS bullseye [11:20:12] 06serviceops, 06SRE, 13Patch-For-Review: mw2420-mw2451 do have unnecessary raid controllers (configured) - https://phabricator.wikimedia.org/T358489#10221004 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1002 for host wikikube-worker2092.codfw.wmnet with OS bullseye com... [11:28:50] 06serviceops, 06SRE, 13Patch-For-Review: mw2420-mw2451 do have unnecessary raid controllers (configured) - https://phabricator.wikimedia.org/T358489#10221059 (10Clement_Goubert) [11:29:37] 06serviceops, 06SRE, 13Patch-For-Review: mw2420-mw2451 do have unnecessary raid controllers (configured) - https://phabricator.wikimedia.org/T358489#10221064 (10Clement_Goubert) 05Open→03In progress [11:29:56] 06serviceops, 06DC-Ops, 10ops-codfw, 10Prod-Kubernetes: Degraded RAID on wikikube-worker2092 - https://phabricator.wikimedia.org/T374409#10221060 (10Clement_Goubert) 05In progress→03Resolved Hardware RAID removed, server reimaged and repooled. [12:34:35] 06serviceops, 10decommission-hardware: decommission scandium - https://phabricator.wikimedia.org/T376632#10221257 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by akosiaris@cumin1002 for hosts: `scandium.eqiad.wmnet` - scandium.eqiad.wmnet (**FAIL**) - Downtimed host on Icinga/Alertmanager... [12:35:35] 06serviceops, 10decommission-hardware: decommission scandium - https://phabricator.wikimedia.org/T376632#10221259 (10akosiaris) machine powered off manually [12:35:57] 06serviceops, 10decommission-hardware: decommission scandium - https://phabricator.wikimedia.org/T376632#10221260 (10akosiaris) [13:48:45] 06serviceops, 10MW-on-K8s, 13Patch-For-Review, 10Sustainability (Incident Followup): mwscript-k8s creates too many resources - https://phabricator.wikimedia.org/T376795#10221449 (10JMeybohm) Adding 6k configmaps does not really do anything to calico, cert-manager, helm-state metrics. It might have an impac... [13:50:51] 06serviceops, 10Deployments, 06Release-Engineering-Team, 13Patch-For-Review: sync-testservers-k8s takes 4 minutes when deploying a mediawiki-config change - https://phabricator.wikimedia.org/T374907#10221451 (10akosiaris) @hashar Patch merged today. Starting next week, there should be some speed improvemen... [14:01:36] 06serviceops, 10MW-on-K8s, 13Patch-For-Review: mw-debug-repl fails with `container mediawiki-pinkunicorn-app is not valid for pod mw-debug.codfw.next-5d785576b4-sq6dv` - https://phabricator.wikimedia.org/T376895#10221465 (10Clement_Goubert) 05In progress→03Resolved ` cgoubert@deploy2002:~$ sudo /usr/... [15:23:15] 06serviceops, 06DC-Ops, 10ops-codfw: Q2:rack/setup/install wikikube-worker21[28-35] - https://phabricator.wikimedia.org/T377007 (10RobH) 03NEW [15:24:08] 06serviceops: wikikube-worker21[28-35] implementation tracking - https://phabricator.wikimedia.org/T377008 (10RobH) 03NEW [15:24:50] 06serviceops, 06DC-Ops, 10ops-codfw: Q2:rack/setup/install wikikube-worker21[28-35] - https://phabricator.wikimedia.org/T377007#10221735 (10RobH) [15:33:33] 06serviceops, 06DC-Ops, 10ops-codfw: Q2:rack/setup/install kubestage200[3-4] - https://phabricator.wikimedia.org/T377009 (10RobH) 03NEW [15:34:16] 06serviceops, 06DC-Ops, 10ops-codfw: Q2:rack/setup/install kubestage200[3-4] - https://phabricator.wikimedia.org/T377009#10221782 (10RobH) [15:34:41] 06serviceops: kubestage200[3-4] implementation tracking - https://phabricator.wikimedia.org/T377011 (10RobH) 03NEW [15:40:07] 06serviceops, 06DC-Ops, 10ops-codfw: Q2:rack/setup/install kubestage200[3-4] - https://phabricator.wikimedia.org/T377009#10221838 (10RobH) a:03Clement_Goubert @Clement_Goubert, Please note the workflow for racking tasks has changed this fiscal year, and we now require the puppet updates from the sub-team... [15:40:17] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q2:rack/setup/install wikikube-worker21[28-35] - https://phabricator.wikimedia.org/T377007#10221841 (10RobH) a:03Clement_Goubert @Clement_Goubert, Please note the workflow for racking tasks has changed this fiscal year, and we now require the puppet updates fr... [15:41:04] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q2:rack/setup/install mc-gp200[4-6] - https://phabricator.wikimedia.org/T376968#10221843 (10RobH) a:03Clement_Goubert @Clement_Goubert, Please note the workflow for racking tasks has changed this fiscal year, and we now require the puppet updates from the sub-... [15:41:27] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q2:rack/setup/install wikikube-worker21[56-70] - https://phabricator.wikimedia.org/T376965#10221845 (10RobH) a:03Clement_Goubert @Clement_Goubert, Please note the workflow for racking tasks has changed this fiscal year, and we now require the puppet updates fr... [16:22:45] 06serviceops, 06DC-Ops, 10ops-eqiad: Q2:rack/setup/install wikikube-worker12[35-42] - https://phabricator.wikimedia.org/T377021 (10RobH) 03NEW [16:23:51] 06serviceops: wikikube-worker12[35-42] implementation tracking - https://phabricator.wikimedia.org/T377022 (10RobH) 03NEW [16:24:07] 06serviceops, 06DC-Ops, 10ops-eqiad: Q2:rack/setup/install wikikube-worker12[35-42] - https://phabricator.wikimedia.org/T377021#10222049 (10RobH) [16:25:42] 06serviceops, 06DC-Ops, 10ops-eqiad: Q2:rack/setup/install wikikube-worker12[35-42] - https://phabricator.wikimedia.org/T377021#10222051 (10RobH) a:03Clement_Goubert @Clement_Goubert, Please note the workflow for racking tasks has changed this fiscal year, and we now require the puppet updates from the su... [16:58:13] 06serviceops, 06DC-Ops, 10ops-codfw: Q2:rack/setup/install wikikube-worker21[36-55] - https://phabricator.wikimedia.org/T377027 (10RobH) 03NEW [16:58:52] 06serviceops, 06DC-Ops, 10ops-codfw: Q2:rack/setup/install wikikube-worker21[36-55] - https://phabricator.wikimedia.org/T377027#10222192 (10RobH) [16:59:10] 06serviceops: wikikube-worker21[36-55] implementation tracking - https://phabricator.wikimedia.org/T377028 (10RobH) 03NEW [17:00:29] 06serviceops, 06DC-Ops, 10ops-codfw: Q2:rack/setup/install wikikube-worker21[36-55] - https://phabricator.wikimedia.org/T377027#10222205 (10RobH) a:03Clement_Goubert @Clement_Goubert, Please note the workflow for racking tasks has changed this fiscal year, and we now require the puppet updates from the su... [17:46:18] 06serviceops, 06DC-Ops, 10ops-eqiad: Q2:rack/setup/install mc-gp100[4-6] - https://phabricator.wikimedia.org/T377032 (10RobH) 03NEW [17:46:39] 06serviceops, 06DC-Ops, 10ops-eqiad: Q2:rack/setup/install mc-gp100[4-6] - https://phabricator.wikimedia.org/T377032#10222328 (10RobH) [17:47:09] 06serviceops: mc-gp100[4-6] implementation tracking - https://phabricator.wikimedia.org/T377033 (10RobH) 03NEW [17:47:11] 06serviceops: mc-gp100[4-6] implementation tracking - https://phabricator.wikimedia.org/T377033#10222340 (10RobH) a:03Clement_Goubert [17:48:19] 06serviceops, 06DC-Ops, 10ops-eqiad: Q2:rack/setup/install mc-gp100[4-6] - https://phabricator.wikimedia.org/T377032#10222345 (10RobH) a:03Clement_Goubert @Clement_Goubert, Please note the workflow for racking tasks has changed this fiscal year, and we now require the puppet updates from the sub-team rece... [18:07:58] 06serviceops, 07Datacenter-Switchover, 13Patch-For-Review: Southward Datacenter Switchover (September 2024) - https://phabricator.wikimedia.org/T370962#10222420 (10Scott_French) 05Open→03Resolved All remaining follow-ups in T370962#10183874 have been split off to tasks or other discussion venues. Sin... [19:03:31] 06serviceops: Migrate production Shellbox variants to PHP 8.1 - https://phabricator.wikimedia.org/T377038 (10Scott_French) 03NEW [19:04:01] 06serviceops: Migrate production Shellbox variants to PHP 8.1 - https://phabricator.wikimedia.org/T377038#10222559 (10Scott_French) [19:04:02] 06serviceops: Turn up PHP 8.1 Shellbox deployments - https://phabricator.wikimedia.org/T375243#10222560 (10Scott_French) [19:33:56] 06serviceops: Turn up PHP 8.1-flavored k8s deployments for all MediaWiki services - https://phabricator.wikimedia.org/T377040 (10Scott_French) 03NEW [19:36:16] 06serviceops: Turn up PHP 8.1-flavored k8s deployments for all MediaWiki services - https://phabricator.wikimedia.org/T377040#10222609 (10Scott_French) As with the mw-debug "next" deployment, we can start work on these immediately even though 8.1-based MediaWiki images are not yet available (we'll just point the... [19:38:09] 06serviceops: Turn up PHP 8.1-flavored k8s deployments for all MediaWiki services - https://phabricator.wikimedia.org/T377040#10222625 (10Scott_French) [20:01:45] 06serviceops: Support cookie-driven fractional migration to PHP 8.1 deployments of mw-web and mw-api-ext - https://phabricator.wikimedia.org/T377042 (10Scott_French) 03NEW [20:02:02] 06serviceops: Support cookie-driven fractional migration to PHP 8.1 deployments of mw-web and mw-api-ext - https://phabricator.wikimedia.org/T377042#10222657 (10Scott_French) [20:02:02] 06serviceops: Turn up PHP 8.1-flavored k8s deployments for all MediaWiki services - https://phabricator.wikimedia.org/T377040#10222658 (10Scott_French) [20:10:10] 06serviceops: Turn up PHP 8.1-flavored k8s deployments for all MediaWiki services - https://phabricator.wikimedia.org/T377040#10222673 (10Scott_French) Service ports for #1 * 4454 mw-web-next * 4455 mw-api-ext-next [21:48:18] 06serviceops, 13Patch-For-Review: Support cookie-driven fractional migration to PHP 8.1 deployments of mw-web and mw-api-ext - https://phabricator.wikimedia.org/T377042#10222845 (10Scott_French) It appears that the phpEngine instrument is still fully wired into the WikimediaEvents extension - i.e., it seems no...