[08:46:50] 06serviceops, 06Data-Persistence, 10Prod-Kubernetes, 13Patch-For-Review: Reevaluate the requirement for dedicated sessionstore/kask nodes in wikikube clusters - https://phabricator.wikimedia.org/T379599#10356430 (10ops-monitoring-bot) depool host kubernetes[2005-2006,2015-2016].codfw.wmnet by jayme@cumin20... [08:46:53] 06serviceops, 06Data-Persistence, 10Prod-Kubernetes, 13Patch-For-Review: Reevaluate the requirement for dedicated sessionstore/kask nodes in wikikube clusters - https://phabricator.wikimedia.org/T379599#10356431 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by jayme@cumin2002 d... [08:49:01] 06serviceops, 06Data-Persistence, 10Prod-Kubernetes, 13Patch-For-Review: Reevaluate the requirement for dedicated sessionstore/kask nodes in wikikube clusters - https://phabricator.wikimedia.org/T379599#10356432 (10ops-monitoring-bot) depool host kubernetes[1005-1006,1015-1016].eqiad.wmnet by jayme@cumin20... [08:49:09] 06serviceops, 06Data-Persistence, 10Prod-Kubernetes, 13Patch-For-Review: Reevaluate the requirement for dedicated sessionstore/kask nodes in wikikube clusters - https://phabricator.wikimedia.org/T379599#10356433 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by jayme@cumin2002 d... [09:23:53] 06serviceops, 06Data-Persistence, 10Prod-Kubernetes, 13Patch-For-Review: Reevaluate the requirement for dedicated sessionstore/kask nodes in wikikube clusters - https://phabricator.wikimedia.org/T379599#10356498 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by jayme@cumin2002 for hosts:... [12:28:37] 06serviceops, 10MediaWiki-extensions-PropertySuggester, 10MW-on-K8s, 10Wikidata, and 3 others: [PS] Update PropertySuggester update process for mwscript-k8s - https://phabricator.wikimedia.org/T376604#10357272 (10ArthurTaylor) We (@hoo , @AudreyPenven_WMDE ) just reimported the 20240429 data using the lega... [13:36:26] 06serviceops, 10decommission-hardware: decommission mw[1349-1413] - https://phabricator.wikimedia.org/T375842#10357574 (10Clement_Goubert) [13:37:47] 06serviceops, 10decommission-hardware: decommission mw[1349-1413] - https://phabricator.wikimedia.org/T375842#10357578 (10akosiaris) Adding @jasmine_ now that she has her sudo privileges so that she can run some decoms. [13:38:06] 06serviceops, 06collaboration-services, 10GitLab (CI & Job Runners): Create a staging apt repository for CI-based builds of Debian packages - https://phabricator.wikimedia.org/T347004#10357580 (10eoghan) 05Open→03Resolved The pipeline seems to be working correctly, and we've got [[ https://wikitech.w... [15:14:59] 06serviceops, 10MW-on-K8s, 13Patch-For-Review: Allow running one-off scripts manually - https://phabricator.wikimedia.org/T341553#10357946 (10Tgr) Is there a way to pass an env variable? The need came up in {T380575}; old mwscript doesn't do it smoothly either, but it can be made to work, which I suspect won... [15:17:28] 06serviceops, 06Data-Platform-SRE, 06Machine-Learning-Team, 10Prod-Kubernetes, 07Kubernetes: Update kserve to v0.13.0 on ML clusters - https://phabricator.wikimedia.org/T380722#10357967 (10klausman) [15:17:54] 06serviceops, 06Data-Platform-SRE, 06Machine-Learning-Team, 10Prod-Kubernetes, and 2 others: Update knative-serving+net-istio to v1.12.x on ML clusters - https://phabricator.wikimedia.org/T380723#10357971 (10klausman) [15:28:53] 06serviceops, 06Content-Transform-Team-WIP, 10Electron-PDFs, 07Essential-Work: Download to PDF: HTTP 500 error on some wikis for some users - https://phabricator.wikimedia.org/T376438#10357989 (10TheDJ) 05Open→03Resolved There have not been significant 5xx errors for 7+ days now. Calling this fixed... [15:44:43] 06serviceops: wikikube-worker13[13-27] implementation tracking - https://phabricator.wikimedia.org/T380350#10358036 (10Clement_Goubert) [15:45:37] 06serviceops: wikikube-worker13[13-27] implementation tracking - https://phabricator.wikimedia.org/T380350#10358037 (10Clement_Goubert) [15:46:02] 06serviceops: wikikube-worker13[13-27] implementation tracking - https://phabricator.wikimedia.org/T380350#10358041 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1002 for host wikikube-worker1321.eqiad.wmnet with OS bookworm [15:46:40] 06serviceops: wikikube-worker13[13-27] implementation tracking - https://phabricator.wikimedia.org/T380350#10358042 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1002 for host wikikube-worker1322.eqiad.wmnet with OS bookworm [15:47:34] 06serviceops: wikikube-worker13[13-27] implementation tracking - https://phabricator.wikimedia.org/T380350#10358045 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1002 for host wikikube-worker1323.eqiad.wmnet with OS bookworm [15:48:16] 06serviceops: wikikube-worker13[13-27] implementation tracking - https://phabricator.wikimedia.org/T380350#10358048 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1002 for host wikikube-worker1324.eqiad.wmnet with OS bookworm [15:48:53] 06serviceops: wikikube-worker13[13-27] implementation tracking - https://phabricator.wikimedia.org/T380350#10358049 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1002 for host wikikube-worker1325.eqiad.wmnet with OS bookworm [15:49:26] 06serviceops: wikikube-worker13[13-27] implementation tracking - https://phabricator.wikimedia.org/T380350#10358052 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1002 for host wikikube-worker1326.eqiad.wmnet with OS bookworm [15:49:58] 06serviceops: wikikube-worker13[13-27] implementation tracking - https://phabricator.wikimedia.org/T380350#10358053 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1002 for host wikikube-worker1327.eqiad.wmnet with OS bookworm [16:20:48] 06serviceops: wikikube-worker13[13-27] implementation tracking - https://phabricator.wikimedia.org/T380350#10358220 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1002 for host wikikube-worker1323.eqiad.wmnet with OS bookworm executed with errors: - wikikube-worker1323 (**FA... [16:22:53] 06serviceops: wikikube-worker13[13-27] implementation tracking - https://phabricator.wikimedia.org/T380350#10358235 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1002 for host wikikube-worker1323.eqiad.wmnet with OS bookworm [16:27:03] 06serviceops: wikikube-worker13[13-27] implementation tracking - https://phabricator.wikimedia.org/T380350#10358254 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1002 for host wikikube-worker1322.eqiad.wmnet with OS bookworm executed with errors: - wikikube-worker1322 (**FA... [16:27:32] 06serviceops: wikikube-worker13[13-27] implementation tracking - https://phabricator.wikimedia.org/T380350#10358255 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1002 for host wikikube-worker1324.eqiad.wmnet with OS bookworm executed with errors: - wikikube-worker1324 (**FA... [16:27:41] 06serviceops: wikikube-worker13[13-27] implementation tracking - https://phabricator.wikimedia.org/T380350#10358256 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1002 for host wikikube-worker1325.eqiad.wmnet with OS bookworm executed with errors: - wikikube-worker1325 (**FA... [16:27:53] 06serviceops: wikikube-worker13[13-27] implementation tracking - https://phabricator.wikimedia.org/T380350#10358257 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1002 for host wikikube-worker1326.eqiad.wmnet with OS bookworm executed with errors: - wikikube-worker1326 (**FA... [16:28:06] 06serviceops: wikikube-worker13[13-27] implementation tracking - https://phabricator.wikimedia.org/T380350#10358258 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1002 for host wikikube-worker1327.eqiad.wmnet with OS bookworm executed with errors: - wikikube-worker1327 (**FA... [16:28:38] 06serviceops: wikikube-worker13[13-27] implementation tracking - https://phabricator.wikimedia.org/T380350#10358261 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1002 for host wikikube-worker1321.eqiad.wmnet with OS bookworm executed with errors: - wikikube-worker1321 (**FA... [16:29:18] 06serviceops: wikikube-worker13[13-27] implementation tracking - https://phabricator.wikimedia.org/T380350#10358262 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1002 for host wikikube-worker1321.eqiad.wmnet with OS bookworm [16:29:37] 06serviceops: wikikube-worker13[13-27] implementation tracking - https://phabricator.wikimedia.org/T380350#10358263 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1002 for host wikikube-worker1322.eqiad.wmnet with OS bookworm [16:29:58] 06serviceops: wikikube-worker13[13-27] implementation tracking - https://phabricator.wikimedia.org/T380350#10358264 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1002 for host wikikube-worker1324.eqiad.wmnet with OS bookworm [16:30:15] 06serviceops: wikikube-worker13[13-27] implementation tracking - https://phabricator.wikimedia.org/T380350#10358277 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1002 for host wikikube-worker1325.eqiad.wmnet with OS bookworm [16:30:39] 06serviceops: wikikube-worker13[13-27] implementation tracking - https://phabricator.wikimedia.org/T380350#10358278 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1002 for host wikikube-worker1326.eqiad.wmnet with OS bookworm [16:30:56] 06serviceops: wikikube-worker13[13-27] implementation tracking - https://phabricator.wikimedia.org/T380350#10358281 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1002 for host wikikube-worker1327.eqiad.wmnet with OS bookworm [16:44:23] 06serviceops: wikikube-worker13[13-27] implementation tracking - https://phabricator.wikimedia.org/T380350#10358354 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1002 for host wikikube-worker1321.eqiad.wmnet with OS bookworm executed with errors: - wikikube-worker1321 (**FA... [16:46:08] 06serviceops, 10MW-on-K8s, 13Patch-For-Review: Allow running one-off scripts manually - https://phabricator.wikimedia.org/T341553#10358334 (10Urbanecm_WMF) Circling back to the topic of job termination (previously discussed at T341553#10259819 //et seq//). I needed to run userOptions.php today, and I discove... [16:46:14] 06serviceops: wikikube-worker13[13-27] implementation tracking - https://phabricator.wikimedia.org/T380350#10358368 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1002 for host wikikube-worker1321.eqiad.wmnet with OS bookworm [17:04:10] 06serviceops: wikikube-worker13[13-27] implementation tracking - https://phabricator.wikimedia.org/T380350#10358506 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1002 for host wikikube-worker1323.eqiad.wmnet with OS bookworm completed: - wikikube-worker1323 (**PASS**) - R... [17:11:37] 06serviceops: wikikube-worker13[13-27] implementation tracking - https://phabricator.wikimedia.org/T380350#10358550 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1002 for host wikikube-worker1322.eqiad.wmnet with OS bookworm completed: - wikikube-worker1322 (**PASS**) - R... [17:13:45] 06serviceops: wikikube-worker13[13-27] implementation tracking - https://phabricator.wikimedia.org/T380350#10358571 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1002 for host wikikube-worker1327.eqiad.wmnet with OS bookworm completed: - wikikube-worker1327 (**PASS**) - R... [17:16:20] 06serviceops, 06Content-Transform-Team, 06MediaWiki-Engineering, 06MW-Interfaces-Team, and 3 others: Transition parsoidtest1001 to PHP 8.1 - https://phabricator.wikimedia.org/T380485#10358580 (10ABreault-WMF) While this is being sketched out, see T298046 as another itch to scratch [17:17:10] 06serviceops: wikikube-worker13[13-27] implementation tracking - https://phabricator.wikimedia.org/T380350#10358590 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1002 for host wikikube-worker1324.eqiad.wmnet with OS bookworm completed: - wikikube-worker1324 (**PASS**) - R... [17:24:25] 06serviceops: wikikube-worker13[13-27] implementation tracking - https://phabricator.wikimedia.org/T380350#10358620 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1002 for host wikikube-worker1325.eqiad.wmnet with OS bookworm completed: - wikikube-worker1325 (**PASS**) - R... [17:28:34] 06serviceops, 10MW-on-K8s, 13Patch-For-Review: Allow running one-off scripts manually - https://phabricator.wikimedia.org/T341553#10358629 (10RLazarus) >>! In T341553#10357946, @Tgr wrote: > Is there a way to pass an env variable? The need came up in {T380575}; old mwscript doesn't do it smoothly either, but... [17:29:02] 06serviceops: wikikube-worker13[13-27] implementation tracking - https://phabricator.wikimedia.org/T380350#10358642 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1002 for host wikikube-worker1321.eqiad.wmnet with OS bookworm completed: - wikikube-worker1321 (**PASS**) - R... [17:47:43] 06serviceops: wikikube-worker13[13-27] implementation tracking - https://phabricator.wikimedia.org/T380350#10358697 (10ops-monitoring-bot) pool host wikikube-worker[1313-1327].eqiad.wmnet by cgoubert@cumin1002 with reason: None [17:47:44] 06serviceops: wikikube-worker13[13-27] implementation tracking - https://phabricator.wikimedia.org/T380350#10358709 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by cgoubert@cumin1002 pool for host wikikube-worker[1313-1327].eqiad.wmnet completed: - wikikube-worker[1313-1327].eqiad.w... [17:48:33] 06serviceops: wikikube-worker13[13-27] implementation tracking - https://phabricator.wikimedia.org/T380350#10358718 (10Clement_Goubert) [17:48:37] 06serviceops: wikikube-worker13[13-27] implementation tracking - https://phabricator.wikimedia.org/T380350#10358715 (10Clement_Goubert) 05Open→03Resolved a:03Clement_Goubert [18:07:54] 06serviceops, 06DC-Ops, 10decommission-hardware, 10ops-codfw, 06SRE: Decommission parse20[01-20] - https://phabricator.wikimedia.org/T380473#10358812 (10Papaul) [18:11:00] 06serviceops, 06DC-Ops, 10decommission-hardware, 10ops-codfw, 06SRE: Decommission parse20[01-20] - https://phabricator.wikimedia.org/T380473#10358814 (10Papaul) 05Open→03Resolved a:03Papaul This complete [18:21:53] 06serviceops, 10MW-on-K8s, 13Patch-For-Review: Allow running one-off scripts manually - https://phabricator.wikimedia.org/T341553#10358853 (10RLazarus) >>! In T341553#10358334, @Urbanecm_WMF wrote > 1. Hitting control-c no longer terminates the job (so blindly following the script's guidance will execute th... [18:45:09] 06serviceops, 10MW-on-K8s, 13Patch-For-Review: Allow running one-off scripts manually - https://phabricator.wikimedia.org/T341553#10358961 (10Urbanecm_WMF) >>! In T341553#10358853, @RLazarus wrote: > I wouldn't be surprised if it works under mwscript-k8s with `--attach` rather than `--follow` It works...kin... [20:20:26] 06serviceops, 10MediaWiki-Platform-Team (Radar), 13Patch-For-Review: Extend x-wikimedia-debug-routing.lua to support PHP 8.1 mw-debug deployment - https://phabricator.wikimedia.org/T372605#10359399 (10Scott_French) Thanks to @Krinkle for confirming that testing is now unblocked. Given that, I'm going ahead a... [20:31:53] 06serviceops, 10MW-on-K8s, 13Patch-For-Review: Allow running one-off scripts manually - https://phabricator.wikimedia.org/T341553#10359460 (10RLazarus) >>! In T341553#10358961, @Urbanecm_WMF wrote: >Do we plan to fix the race conditions related to `--attach`? Unfortunately the bug is in the underlying kuber... [20:35:56] 06serviceops, 10MediaWiki-Platform-Team (Radar): Extend x-wikimedia-debug-routing.lua to support PHP 8.1 mw-debug deployment - https://phabricator.wikimedia.org/T372605#10359479 (10Scott_French) 05Open→03Resolved The `debug.json` changes are live, and I've validated that selecting "k8s-mwdebug-next" (o... [20:37:02] 06serviceops, 06Content-Transform-Team, 06MediaWiki-Engineering, 06MediaWiki-Platform-Team, and 3 others: Testing and verification of MediaWiki on PHP 8.1 in mwdebug-next - https://phabricator.wikimedia.org/T379986#10359486 (10Scott_French) [20:40:40] 06serviceops, 06Content-Transform-Team, 06MediaWiki-Engineering, 06MediaWiki-Platform-Team, and 3 others: Testing and verification of MediaWiki on PHP 8.1 in mwdebug-next - https://phabricator.wikimedia.org/T379986#10359503 (10Scott_French) As now noted in the (edited) task description, mwdebug-next is ava... [21:26:40] 06serviceops, 10MW-on-K8s: Support passing env variables to maintenance scripts in mwscript-k8s - https://phabricator.wikimedia.org/T380925 (10RLazarus) 03NEW [21:27:09] 06serviceops, 10MW-on-K8s: MWScript.php doesn't allow wikiless scripts without the .php suffix - https://phabricator.wikimedia.org/T376616#10359661 (10RLazarus) a:05RLazarus→03None [23:09:51] 06serviceops, 10MW-on-K8s: Support passing env variables to maintenance scripts in mwscript-k8s - https://phabricator.wikimedia.org/T380925#10360022 (10Tgr) Thanks @RLazarus! > whether it's a good idea to support it at all It's a way to communicate state to random parts of the codebase which are too deep in...