[00:09:08] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE, 13Patch-For-Review: Q4:rack/setup/install parsoidtest1001 - https://phabricator.wikimedia.org/T363399#9952481 (10Papaul) @Jclark-ctr i don't understand why this step was checked Update the operations/puppet repo - this should include updates to preseed.yaml, a... [00:11:28] 06serviceops, 10MW-on-K8s, 13Patch-For-Review: mwscript-k8s --attach error: TypeError: 'NoneType' object is not iterable - https://phabricator.wikimedia.org/T369175#9952482 (10RLazarus) 05Open→03Resolved This is fixed, thanks again for testing! [00:15:30] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE, 13Patch-For-Review: Q4:rack/setup/install parsoidtest1001 - https://phabricator.wikimedia.org/T363399#9952491 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by dzahn@cumin1002 for host parsoidtest1001.eqiad.wmnet with OS bullseye [00:44:11] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE, 13Patch-For-Review: Q4:rack/setup/install parsoidtest1001 - https://phabricator.wikimedia.org/T363399#9952509 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by dzahn@cumin1002 for host parsoidtest1001.eqiad.wmnet with OS bullseye completed:... [00:45:30] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE, 13Patch-For-Review: Q4:rack/setup/install parsoidtest1001 - https://phabricator.wikimedia.org/T363399#9952510 (10Dzahn) machine is now up and running with "insetup::serviceops" role. [00:45:47] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE, 13Patch-For-Review: Q4:rack/setup/install parsoidtest1001 - https://phabricator.wikimedia.org/T363399#9952511 (10Dzahn) [00:48:12] 06serviceops, 06DC-Ops, 10ops-eqiad, 06SRE, 13Patch-For-Review: Q4:rack/setup/install parsoidtest1001 - https://phabricator.wikimedia.org/T363399#9952512 (10Dzahn) @akosiaris The machine is now ready to get a production puppet role. If it's replacing `scandium`, then `role(parsoid::testing)` can be appli... [12:13:41] nemo-yiannis: do you have a task number? [12:14:04] https://phabricator.wikimedia.org/T367418 [12:14:58] alright, we'll see who can help, cheers! [12:15:07] Just for clarification, i am not that worried about removing the no-cache requests to parsoid but replacing the parsoid resource_change with mediawiki events [12:15:53] nemo-yiannis: can you please add this as a comment? [12:15:56] in the patch [12:16:21] there should be one already [12:22:08] 06serviceops, 06SRE, 10Data Products (Data Products Sprint 16), 13Patch-For-Review, 07Service-deployment-requests: Commons Impact Metrics AQS 2.0 Deployment to Staging and Production - https://phabricator.wikimedia.org/T361835#9953674 (10SGupta-WMF) @Scott_French We have the new image here - https://gitl... [13:41:52] 06serviceops, 10MW-on-K8s, 10MediaWiki-Platform-Team (Radar), 13Patch-For-Review: mcrouter daemonset on mw-on-k8s - https://phabricator.wikimedia.org/T346690#9953893 (10jijiki) This is where we are: * codfw -> full on * eqiad -> mw-parsoid, mw-api-ext, mw-api-int So far so good, no notable errors on logs... [14:00:46] 06serviceops, 06Infrastructure-Foundations, 13Patch-For-Review, 07Security: Upgrade K8s docker images to running in production on Buster with either Bullseye or Bookworm - https://phabricator.wikimedia.org/T368366#9954040 (10jijiki) From the mcrouter side of things, we hope to have T346690 sorted soon, whi... [14:05:23] 06serviceops, 10MW-on-K8s, 06SRE, 06Traffic, and 2 others: Spin down api_appserver and appserver clusters - https://phabricator.wikimedia.org/T367949#9954050 (10jijiki) [14:05:38] 06serviceops, 10MW-on-K8s, 06SRE, 06Traffic, and 2 others: Spin down api_appserver and appserver clusters - https://phabricator.wikimedia.org/T367949#9954055 (10jijiki) [14:10:56] 06serviceops, 10MW-on-K8s, 06SRE, 06Traffic, and 2 others: Spin down api_appserver and appserver clusters - https://phabricator.wikimedia.org/T367949#9954061 (10Clement_Goubert) [14:31:18] 06serviceops, 10MW-on-K8s, 06SRE, 06Traffic, and 2 others: Spin down api_appserver and appserver clusters - https://phabricator.wikimedia.org/T367949#9954181 (10Clement_Goubert) [14:35:15] folks I just discovered something a little strange, namely that in thumbor we have: [14:35:18] image: "docker-registry.wikimedia.org/haproxy:latest" [14:36:12] is there a reason? Otherwise I'll pin it to 2.4.18-2-20240630 [14:36:25] I just build 2.8 and it is available on the registry [14:41:26] https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1052125 [14:49:40] <_joe_> elukey: that should be the default in the chart [14:49:44] <_joe_> but not in the deployment [14:50:07] yep but afaics we are missing the pin in helmfile [14:50:44] <_joe_> yeah that's not great [14:52:17] 06serviceops, 06Data-Platform-SRE, 06Discovery-Search, 10Wikidata, and 2 others: Use Envoy instead of LVS to route internal federation traffic for WDQS - https://phabricator.wikimedia.org/T368972#9954239 (10JMeybohm) [14:54:11] claime: <3 [15:08:35] thumbor in staging with haproxy 2.8 is up and running [15:09:15] just a matter of testing it and then it should be ready for prime time [15:09:53] 06serviceops, 06Infrastructure-Foundations, 13Patch-For-Review: Upgrade thumbor Docker images - https://phabricator.wikimedia.org/T369144#9954305 (10elukey) Next steps: * Test Thumbor in staging to validate that everything works, and then rollout to prod. [15:10:24] 06serviceops, 06Infrastructure-Foundations, 13Patch-For-Review, 07Security: Upgrade K8s docker images to running in production on Buster with either Bullseye or Bookworm - https://phabricator.wikimedia.org/T368366#9954308 (10elukey)