[09:27:12] 06serviceops: The mwdebug cluster has inconsistent AAAA DNS records for the primary IPv6 of the hosts - https://phabricator.wikimedia.org/T380254 (10Volans) 03NEW [09:28:17] 06serviceops: The mwdebug cluster has inconsistent AAAA DNS records for the primary IPv6 of the hosts - https://phabricator.wikimedia.org/T380254#10334960 (10Volans) [10:15:46] 06serviceops, 10Prod-Kubernetes, 07Kubernetes: Update kubeconform schema and CI checks to new target Kubernetes version - https://phabricator.wikimedia.org/T379919#10335120 (10Jelto) I’ve added the new Kubernetes API schema in the [MR above](https://gitlab.wikimedia.org/repos/sre/kubernetes-json-schema/-/mer... [10:16:12] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Update kubeconform schema and CI checks to new target Kubernetes version - https://phabricator.wikimedia.org/T379919#10335123 (10Jelto) [10:20:16] 06serviceops, 10MediaWiki-extensions-PropertySuggester, 10MW-on-K8s, 10Wikidata, and 2 others: [PS] Update PropertySuggester update process for mwscript-k8s - https://phabricator.wikimedia.org/T376604#10335129 (10ArthurTaylor) a:03ArthurTaylor [12:04:15] 06serviceops: The mwdebug cluster has inconsistent AAAA DNS records for the primary IPv6 of the hosts - https://phabricator.wikimedia.org/T380254#10335364 (10Clement_Goubert) p:05Triage→03Low [12:05:19] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q2:rack/setup/install wikikube-worker21[56-70] - https://phabricator.wikimedia.org/T376965#10335368 (10Clement_Goubert) Thanks @Jhancock.wm :) [12:17:47] 06serviceops, 13Patch-For-Review: wikikube-worker21[36-55] implementation tracking - https://phabricator.wikimedia.org/T377028#10335427 (10Clement_Goubert) [12:38:13] 06serviceops, 06Structured-Data-Backlog, 10Thumbor: Thumbor workers hang indefinitely when conducting some tiff operations, leading to user-facing error - https://phabricator.wikimedia.org/T374350#10335482 (10hnowlan) >>! In T374350#10333391, @Don-vip wrote: > If it helps, I still face problems, last one thr... [12:52:48] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Update kubeconform schema and CI checks to new target Kubernetes version - https://phabricator.wikimedia.org/T379919#10335514 (10Jelto) [12:55:21] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Reimaging a kubernetes control-plane invalidates service-account tokens issued by it - https://phabricator.wikimedia.org/T380142#10335528 (10JMeybohm) >>! In T380142, @JMeybohm wrote: > I don't think filtering out the "own" public key is st... [12:55:33] 06serviceops, 13Patch-For-Review: wikikube-worker21[36-55] implementation tracking - https://phabricator.wikimedia.org/T377028#10335529 (10Clement_Goubert) [12:59:28] 06serviceops, 06DC-Ops, 10ops-codfw: hw troubleshooting: Link down for wikikube-worker2140.codfw.wmnet - https://phabricator.wikimedia.org/T380265#10335539 (10Clement_Goubert) [13:24:01] 06serviceops, 06Structured-Data-Backlog, 10Thumbor: Thumbor workers hang indefinitely when conducting some tiff operations, leading to user-facing error - https://phabricator.wikimedia.org/T374350#10335600 (10hnowlan) Things done to address this issue so far: * Alerting added to detect (unlikely) recurrence... [13:34:16] 06serviceops: The mwdebug cluster has inconsistent AAAA DNS records for the primary IPv6 of the hosts - https://phabricator.wikimedia.org/T380254#10335636 (10akosiaris) The cluster is slated to eventually be decommissioned and sooner rather than later. I think we can just `decline` this. Any objections? [13:40:21] 06serviceops, 06Content-Transform-Team-WIP, 10Electron-PDFs, 07Essential-Work, 13Patch-For-Review: Download to PDF: HTTP 500 error on some wikis for some users - https://phabricator.wikimedia.org/T376438#10335652 (10TheDJ) The error rate is quickly increasing again: {F57721570} [14:00:20] 06serviceops, 06Content-Transform-Team-WIP, 10Electron-PDFs, 07Essential-Work, 13Patch-For-Review: Download to PDF: HTTP 500 error on some wikis for some users - https://phabricator.wikimedia.org/T376438#10335741 (10JayCubby) I got this on enwiki w/ Chromium v126 (Falklands War) {F57721594} Reloading ht... [14:10:15] 06serviceops, 06Infrastructure-Foundations, 10netops, 07Kubernetes: Reimage one of the wikikube-worker1240 to wikikube-worker1304 node in eqiad as a replacement for wikikube-ctrl1001 - https://phabricator.wikimedia.org/T379790#10335773 (10ops-monitoring-bot) depool host wikikube-worker1290.eqiad.wmnet by a... [14:10:57] 06serviceops, 06Infrastructure-Foundations, 10netops, 07Kubernetes: Reimage one of the wikikube-worker1240 to wikikube-worker1304 node in eqiad as a replacement for wikikube-ctrl1001 - https://phabricator.wikimedia.org/T379790#10335776 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node star... [14:15:23] 06serviceops, 06Content-Transform-Team-WIP, 10Electron-PDFs, 07Essential-Work, 13Patch-For-Review: Download to PDF: HTTP 500 error on some wikis for some users - https://phabricator.wikimedia.org/T376438#10335795 (10CDanis) @ihurbain just deployed the crashpad flag flip patch and (at least for now) Proto... [14:19:29] 06serviceops, 06Infrastructure-Foundations, 10netops, 07Kubernetes, 13Patch-For-Review: Reimage one of the wikikube-worker1240 to wikikube-worker1304 node in eqiad as a replacement for wikikube-ctrl1001 - https://phabricator.wikimedia.org/T379790#10335855 (10akosiaris) >>! In T379790#10330660, @cmooney w... [14:41:04] 06serviceops: wikikube-worker21[36-55] implementation tracking - https://phabricator.wikimedia.org/T377028#10335971 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1002 for host wikikube-worker2136.codfw.wmnet with OS bookworm [14:41:58] 06serviceops: wikikube-worker21[36-55] implementation tracking - https://phabricator.wikimedia.org/T377028#10335972 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1002 for host wikikube-worker2137.codfw.wmnet with OS bookworm [14:42:22] 06serviceops: wikikube-worker21[36-55] implementation tracking - https://phabricator.wikimedia.org/T377028#10335976 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1002 for host wikikube-worker2138.codfw.wmnet with OS bookworm [14:43:31] 06serviceops: wikikube-worker21[36-55] implementation tracking - https://phabricator.wikimedia.org/T377028#10335982 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1002 for host wikikube-worker2139.codfw.wmnet with OS bookworm [14:45:21] 06serviceops: wikikube-worker21[36-55] implementation tracking - https://phabricator.wikimedia.org/T377028#10335988 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1002 for host wikikube-worker2141.codfw.wmnet with OS bookworm [14:45:28] 06serviceops: wikikube-worker21[36-55] implementation tracking - https://phabricator.wikimedia.org/T377028#10335990 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1002 for host wikikube-worker2142.codfw.wmnet with OS bookworm [14:47:12] 06serviceops: wikikube-worker21[36-55] implementation tracking - https://phabricator.wikimedia.org/T377028#10335994 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1002 for host wikikube-worker2140.codfw.wmnet with OS bookworm [15:21:11] 06serviceops: wikikube-worker21[36-55] implementation tracking - https://phabricator.wikimedia.org/T377028#10336179 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1002 for host wikikube-worker2136.codfw.wmnet with OS bookworm executed with errors: - wikikube-worker2136 (**FA... [15:21:12] 06serviceops: wikikube-worker21[36-55] implementation tracking - https://phabricator.wikimedia.org/T377028#10336180 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1002 for host wikikube-worker2137.codfw.wmnet with OS bookworm executed with errors: - wikikube-worker2137 (**FA... [15:21:24] 06serviceops: wikikube-worker21[36-55] implementation tracking - https://phabricator.wikimedia.org/T377028#10336181 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1002 for host wikikube-worker2141.codfw.wmnet with OS bookworm executed with errors: - wikikube-worker2141 (**FA... [15:21:30] 06serviceops: wikikube-worker21[36-55] implementation tracking - https://phabricator.wikimedia.org/T377028#10336184 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1002 for host wikikube-worker2142.codfw.wmnet with OS bookworm executed with errors: - wikikube-worker2142 (**FA... [15:22:31] 06serviceops: wikikube-worker21[36-55] implementation tracking - https://phabricator.wikimedia.org/T377028#10336189 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1002 for host wikikube-worker2136.codfw.wmnet with OS bookworm [15:25:41] 06serviceops: wikikube-worker21[36-55] implementation tracking - https://phabricator.wikimedia.org/T377028#10336223 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1002 for host wikikube-worker2138.codfw.wmnet with OS bookworm executed with errors: - wikikube-worker2138 (**FA... [15:25:57] 06serviceops: wikikube-worker21[36-55] implementation tracking - https://phabricator.wikimedia.org/T377028#10336229 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1002 for host wikikube-worker2139.codfw.wmnet with OS bookworm executed with errors: - wikikube-worker2139 (**FA... [15:28:42] 06serviceops: wikikube-worker21[36-55] implementation tracking - https://phabricator.wikimedia.org/T377028#10336244 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1002 for host wikikube-worker2137.codfw.wmnet with OS bookworm [15:28:58] 06serviceops: wikikube-worker21[36-55] implementation tracking - https://phabricator.wikimedia.org/T377028#10336245 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1002 for host wikikube-worker2138.codfw.wmnet with OS bookworm [15:29:13] 06serviceops: wikikube-worker21[36-55] implementation tracking - https://phabricator.wikimedia.org/T377028#10336246 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1002 for host wikikube-worker2139.codfw.wmnet with OS bookworm [15:29:22] 06serviceops: wikikube-worker21[36-55] implementation tracking - https://phabricator.wikimedia.org/T377028#10336248 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1002 for host wikikube-worker2141.codfw.wmnet with OS bookworm [15:29:35] 06serviceops: wikikube-worker21[36-55] implementation tracking - https://phabricator.wikimedia.org/T377028#10336251 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1002 for host wikikube-worker2142.codfw.wmnet with OS bookworm [15:36:46] 06serviceops, 10Prod-Kubernetes, 07Kubernetes: Update kubeconform schema and CI checks to new target Kubernetes version - https://phabricator.wikimedia.org/T379919#10336310 (10Jelto) [15:37:41] 06serviceops, 06MediaWiki-Platform-Team, 13Patch-For-Review: Extend x-wikimedia-debug-routing.lua to support PHP 8.1 mw-debug deployment - https://phabricator.wikimedia.org/T372605#10336318 (10Krinkle) [15:39:19] 06serviceops, 06Data-Platform-SRE, 10Prod-Kubernetes, 07Kubernetes: Update Kubernetes clusters to >1.25 - https://phabricator.wikimedia.org/T341984#10336331 (10Jelto) [15:40:15] 06serviceops, 10Prod-Kubernetes, 07Kubernetes: Update kubeconform schema and CI checks to new target Kubernetes version - https://phabricator.wikimedia.org/T379919#10336327 (10Jelto) 05Open→03Resolved Kubeconform now lints and validates the `deployments-charts` repository against version `1.31.2`. No... [15:54:44] 06serviceops: wikikube-worker21[36-55] implementation tracking - https://phabricator.wikimedia.org/T377028#10336443 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1002 for host wikikube-worker2140.codfw.wmnet with OS bookworm executed with errors: - wikikube-worker2140 (**FA... [15:54:54] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: hw troubleshooting: Link down for wikikube-worker2140.codfw.wmnet - https://phabricator.wikimedia.org/T380265#10336410 (10Jhancock.wm) @Papaul you might need to check the switch. I looked in the idrac and the link shows as up. physically up as well. [15:55:24] 06serviceops: wikikube-worker21[36-55] implementation tracking - https://phabricator.wikimedia.org/T377028#10336445 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1002 for host wikikube-worker2140.codfw.wmnet with OS bookworm [16:04:56] 06serviceops, 10MediaWiki-Platform-Team (Radar), 13Patch-For-Review: Extend x-wikimedia-debug-routing.lua to support PHP 8.1 mw-debug deployment - https://phabricator.wikimedia.org/T372605#10336526 (10larissagaulia) [16:04:58] 06serviceops: wikikube-worker21[36-55] implementation tracking - https://phabricator.wikimedia.org/T377028#10336527 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1002 for host wikikube-worker2136.codfw.wmnet with OS bookworm completed: - wikikube-worker2136 (**PASS**) - R... [16:09:55] 06serviceops: wikikube-worker21[36-55] implementation tracking - https://phabricator.wikimedia.org/T377028#10336573 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1002 for host wikikube-worker2137.codfw.wmnet with OS bookworm completed: - wikikube-worker2137 (**PASS**) - R... [16:13:10] 06serviceops: wikikube-worker21[36-55] implementation tracking - https://phabricator.wikimedia.org/T377028#10336582 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1002 for host wikikube-worker2138.codfw.wmnet with OS bookworm completed: - wikikube-worker2138 (**PASS**) - R... [16:19:03] 06serviceops: wikikube-worker21[36-55] implementation tracking - https://phabricator.wikimedia.org/T377028#10336604 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1002 for host wikikube-worker2141.codfw.wmnet with OS bookworm completed: - wikikube-worker2141 (**PASS**) - R... [16:20:07] 06serviceops: wikikube-worker21[36-55] implementation tracking - https://phabricator.wikimedia.org/T377028#10336630 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1002 for host wikikube-worker2139.codfw.wmnet with OS bookworm completed: - wikikube-worker2139 (**PASS**) - R... [16:24:11] 06serviceops: wikikube-worker21[36-55] implementation tracking - https://phabricator.wikimedia.org/T377028#10336654 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1002 for host wikikube-worker2142.codfw.wmnet with OS bookworm completed: - wikikube-worker2142 (**PASS**) - R... [16:28:56] 06serviceops: wikikube-worker21[36-55] implementation tracking - https://phabricator.wikimedia.org/T377028#10336663 (10Clement_Goubert) [17:15:38] 06serviceops: wikikube-worker21[36-55] implementation tracking - https://phabricator.wikimedia.org/T377028#10336884 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1002 for host wikikube-worker2140.codfw.wmnet with OS bookworm executed with errors: - wikikube-worker2140 (**FA... [17:31:39] 06serviceops, 07Kubernetes: Revisit use of the wmf-deployment Gerrit group for deployment-charts rights - https://phabricator.wikimedia.org/T380299 (10hnowlan) 03NEW [17:45:00] 06serviceops, 10Add-Link, 06Growth-Team, 10Prod-Kubernetes, 07Kubernetes: Use ingress for linkrecommendation - https://phabricator.wikimedia.org/T302717#10337161 (10Aklapper) a:05jijiki→03None @jijiki: Removing task assignee as this open task has been assigned for more than two years - See the email... [18:00:58] 06serviceops, 06DC-Ops, 06Infrastructure-Foundations, 10netops, and 3 others: Reimage one of the wikikube-worker1240 to wikikube-worker1304 node in eqiad as a replacement for wikikube-ctrl1001 - https://phabricator.wikimedia.org/T379790#10337359 (10cmooney) Ok. So I've tested the "[[ https://netbox.wikime... [18:12:18] 06serviceops, 10MediaWiki-extensions-OAuth: OAuth extension - update\add logic of userCanSeeSecret() method of Backend\ConsumerAcceptance class. - https://phabricator.wikimedia.org/T265362#10337532 (10Aklapper) a:05roman-stolar→03None @roman-stolar: Removing task assignee as this open task has been assign... [19:38:36] 06serviceops, 13Patch-For-Review: Monitoring to surface "low-traffic" jobs isolation failure - https://phabricator.wikimedia.org/T378609#10337981 (10Scott_French) Initial versions of the runbook and low-traffic jobs debugging dashboard are now available: * https://wikitech.wikimedia.org/wiki/MediaWiki_JobQueue...