[05:16:44] 06serviceops: Clean up and abstract PHP_ENGINE 8.1 routing - https://phabricator.wikimedia.org/T391421#10724540 (10Joe) I would suggest we make this "permanent" in lua, with some configured escape so we don't actually execute the whole lua for every request when we have no experiment running. [07:50:25] 06serviceops: docker-registry.wikimedia.org keeps serving bad blobs - https://phabricator.wikimedia.org/T390251#10724721 (10elukey) @dancy just to be on the safe side, when you have a moment can you tell me what you think about T390251#10716962 ? 90% chance it is nothing but I want to make sure :) [09:36:13] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: hw troubleshooting: hard down for wikikube-worker2142 - https://phabricator.wikimedia.org/T391341#10724965 (10ops-monitoring-bot) check host wikikube-worker2142.codfw.wmnet by cgoubert@cumin1002 with reason: Hardware failure [09:37:18] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: hw troubleshooting: hard down for wikikube-worker2142 - https://phabricator.wikimedia.org/T391341#10724973 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=b0849406-4915-4da3-8220-1f360a73f331) set by cgoubert@cumin1002 for 7 days, 0:00:00 on 1... [10:04:32] 06serviceops, 10MW-on-K8s, 10Release-Engineering-Team (Priority Backlog 📥): Migrate mwdebug* hosts to PHP8.1 - https://phabricator.wikimedia.org/T391452 (10jijiki) 03NEW [10:29:06] 06serviceops: Move Wikikube services to Istio ingress (where possible) - https://phabricator.wikimedia.org/T391457 (10elukey) 03NEW [10:30:46] 06serviceops: Move Wikikube services to Istio ingress (where possible) - https://phabricator.wikimedia.org/T391457#10725368 (10elukey) Currently being tested: Citoid https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1135378 https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1135396 T... [10:35:22] 06serviceops, 13Patch-For-Review: Move Wikikube services to Istio ingress (where possible) - https://phabricator.wikimedia.org/T391457#10725383 (10elukey) [10:36:56] 06serviceops, 10MW-on-K8s, 06Release-Engineering-Team, 13Patch-For-Review: Refactor scap's kubernetes DeploymentsConfig to support selection of image kinds - https://phabricator.wikimedia.org/T389499#10725388 (10BTullis) >>! In T389499#10671841, @Scott_French wrote: > Thanks, @BTullis - I was aware of Joe'... [10:50:30] 06serviceops, 10decommission-hardware: decommission mw2278, mw2279 - https://phabricator.wikimedia.org/T391001#10725403 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by hnowlan@cumin1002 for hosts: `mw[2278-2279].codfw.wmnet` - mw2278.codfw.wmnet (**FAIL**) - Downtimed host on Icinga/Alert... [10:59:24] 06serviceops, 10decommission-hardware, 13Patch-For-Review: decommission mw13[49-51], mw1407 - https://phabricator.wikimedia.org/T383226#10725426 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by hnowlan@cumin1002 for hosts: `mw[1349-1351].eqiad.wmnet` - mw1349.eqiad.wmnet (**PASS**) - Dow... [11:01:58] 06serviceops, 10decommission-hardware: decommission mw2278, mw2279 - https://phabricator.wikimedia.org/T391001#10725429 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by hnowlan@cumin1002 for hosts: `mw2278.codfw.wmnet` - mw2278.codfw.wmnet (**FAIL**) - //Missing DNSName in Nebox for mw2278... [11:02:01] 06serviceops, 10Citoid, 06Editing QA, 06Editing-team, and 5 others: Switch from restbase to api gateway for Citoid - https://phabricator.wikimedia.org/T361576#10725430 (10Mvolz) [11:03:36] 06serviceops, 10Citoid, 06Editing QA, 06Editing-team, and 5 others: Switch from restbase to api gateway for Citoid - https://phabricator.wikimedia.org/T361576#10725432 (10Mvolz) >>! In T361576#10721517, @gerritbot wrote: > Change #1131008 **merged** by jenkins-bot: > %%%[mediawiki/extensions/Citoid@master]... [11:22:20] 06serviceops, 10decommission-hardware, 13Patch-For-Review: decommission mw13[49-51], mw1407 - https://phabricator.wikimedia.org/T383226#10725486 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by hnowlan@cumin1002 for hosts: `mw1407.eqiad.wmnet` - mw1407.eqiad.wmnet (**PASS**) - Downtimed... [11:23:48] 06serviceops, 06DC-Ops, 10decommission-hardware, 10ops-eqiad, 13Patch-For-Review: decommission mw13[49-51], mw1407 - https://phabricator.wikimedia.org/T383226#10725490 (10hnowlan) [11:27:38] 06serviceops, 06DC-Ops, 10decommission-hardware, 10ops-codfw: decommission mw2278, mw2279 - https://phabricator.wikimedia.org/T391001#10725497 (10hnowlan) [12:28:01] 06serviceops, 10MW-on-K8s, 13Patch-For-Review, 10Release-Engineering-Team (Priority Backlog 📥): Migrate mwdebug* hosts to PHP8.1 - https://phabricator.wikimedia.org/T391452#10725756 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jiji@cumin1002 for host mwdebug1002.eqiad.wmnet wi... [13:24:08] 06serviceops, 10MW-on-K8s, 13Patch-For-Review, 10Release-Engineering-Team (Priority Backlog 📥): Migrate mwdebug* hosts to PHP8.1 - https://phabricator.wikimedia.org/T391452#10726028 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jiji@cumin1002 for host mwdebug1002.eqiad.wmnet with O... [15:18:08] 06serviceops: docker-registry.wikimedia.org keeps serving bad blobs - https://phabricator.wikimedia.org/T390251#10726545 (10dancy) >>! In T390251#10716962, @elukey wrote: > My understanding is that those are rsync args (from https://gitlab.wikimedia.org/repos/releng/release/-/blob/main/make-container-image/buil... [16:42:55] 06serviceops, 10Abstract Wikipedia team (25Q4 (Apr–Jun)): Provide guidance on how to use apache bench to benchmark requests not through SSL for production services - https://phabricator.wikimedia.org/T390099#10726854 (10Jdforrester-WMF) [18:54:05] 06serviceops, 10MW-on-K8s, 06Release-Engineering-Team, 13Patch-For-Review: Refactor scap's kubernetes DeploymentsConfig to support selection of image kinds - https://phabricator.wikimedia.org/T389499#10727380 (10Scott_French) Thanks for the additional details on the mechanics, @BTullis! >>! In T389499#107... [21:25:15] 06serviceops: Clean up and abstract PHP_ENGINE 8.1 routing - https://phabricator.wikimedia.org/T391421#10727880 (10Scott_French) >>! In T391421#10724540, @Joe wrote: > I would suggest we make this "permanent" in lua, with some configured escape so we don't actually execute the whole lua for every request when we... [22:08:49] 06serviceops, 10Scap, 13Patch-For-Review: Migrate scap's maintenance script invocations to PHP 8.1 - https://phabricator.wikimedia.org/T390225#10728008 (10Scott_French) p:05Triage→03High The `mediawiki_runtime_image` default change is now live with scap 4.153.0, so all that remains here is to clean up th...