[03:56:40] 06serviceops, 10MW-on-K8s, 10wikitech.wikimedia.org: Migrate Wikitech to Kubernetes - https://phabricator.wikimedia.org/T292707#9734065 (10Andrew) [06:42:09] 06serviceops, 06collaboration-services, 06Infrastructure-Foundations, 10Puppet-Core, and 5 others: Migrate roles to puppet7 - https://phabricator.wikimedia.org/T349619#9734108 (10MoritzMuehlenhoff) [08:28:49] parse1002 is down, including the management interface [08:29:02] T363086 [08:30:07] is there a way to exclude it from the scap dsh list used for image pulling? the list seems to be generated from puppetdb directly, and I don't think we want to remove it from there entirely [08:33:19] scap image pulling? [08:36:12] 06serviceops, 10MW-on-K8s, 06Release-Engineering-Team, 10Scap: Find a way to stage updated OS packages on wikikube - https://phabricator.wikimedia.org/T362628#9734407 (10MoritzMuehlenhoff) [08:36:50] the image pre-pulling functionality in https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/refs/heads/production/modules/profile/manifests/kubernetes/mediawiki_runner.pp#10 [08:37:54] which scap runs on all hosts in the list generated with https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/refs/heads/production/hieradata/common/scap/dsh.yaml#18 before running the helm commands to prevent timeouts [08:38:58] sigh, we have that mess. I wanna delete that awful thing from my memory [08:43:08] no, there isn't any easy way to exclude it btw [08:43:29] we can deactivate it in puppetdb. That might work [08:44:30] and we need to finish the migration, get rid of multiversion images (and do the equivalent some other way) and ditch this hack [08:45:12] we could just swap the role in puppet to insetup::serviceops while it's fixed [08:45:46] if it's matching for the wikikube role [08:48:12] that would require the host being not dead at the moment [08:55:47] I 'll deactivate it in a few [08:55:55] I 'll deactivate it in a few [09:14:00] done and double checked that indeed it's removed [09:29:44] 06serviceops: Provide nodejs20 base images for production - https://phabricator.wikimedia.org/T362681#9734606 (10akosiaris) >>! In T362681#9729331, @MoritzMuehlenhoff wrote: > That's not problem. We should just use the nodesource packages for this, we've been doing the same for "intermediate LTSes" before (e.g.... [09:52:04] 06serviceops, 10MoveComms-Support, 10MW-on-K8s, 06SRE, and 2 others: Move 100% of external traffic to Kubernetes (excluding Votewiki and Commons) - https://phabricator.wikimedia.org/T362323#9734722 (10Clement_Goubert) [10:16:19] 06serviceops, 06Data-Platform-SRE, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migrate charts to Calico Network Policies - https://phabricator.wikimedia.org/T359423#9734880 (10JMeybohm) [10:37:00] 06serviceops, 06collaboration-services, 06Infrastructure-Foundations, 10Puppet-Core, and 5 others: Migrate roles to puppet7 - https://phabricator.wikimedia.org/T349619#9734938 (10MoritzMuehlenhoff) [10:53:48] 06serviceops, 10MW-on-K8s, 13Patch-For-Review: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074#9734974 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1002 for host mw1414.eqiad.wmnet with OS bullseye [10:54:10] 06serviceops, 10MW-on-K8s, 13Patch-For-Review: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074#9734976 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1002 for host mw1415.eqiad.wmnet with OS bullseye [10:54:35] 06serviceops, 10MW-on-K8s, 13Patch-For-Review: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074#9734979 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1002 for host mw1416.eqiad.wmnet with OS bullseye [10:55:01] 06serviceops, 10MW-on-K8s, 13Patch-For-Review: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074#9734985 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1002 for host mw1448.eqiad.wmnet with OS bullseye [10:55:27] 06serviceops, 10MW-on-K8s, 13Patch-For-Review: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074#9734993 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1002 for host mw1449.eqiad.wmnet with OS bullseye [11:28:33] 06serviceops, 10MW-on-K8s, 13Patch-For-Review: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074#9735099 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1002 for host mw1414.eqiad.wmnet with OS bullseye completed: - mw14... [11:30:39] 06serviceops, 10MW-on-K8s, 13Patch-For-Review: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074#9735105 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1002 for host mw1449.eqiad.wmnet with OS bullseye completed: - mw14... [11:34:08] 06serviceops, 10MW-on-K8s, 13Patch-For-Review: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074#9735112 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1002 for host mw1416.eqiad.wmnet with OS bullseye completed: - mw14... [11:35:34] 06serviceops, 10MW-on-K8s, 13Patch-For-Review: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074#9735113 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1002 for host mw1448.eqiad.wmnet with OS bullseye completed: - mw14... [11:38:56] 06serviceops, 10MW-on-K8s, 13Patch-For-Review: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074#9735117 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1002 for host mw1415.eqiad.wmnet with OS bullseye completed: - mw14... [11:48:35] 06serviceops, 10Observability-Logging, 13Patch-For-Review: Logs from containers sometimes not visible in logstash - https://phabricator.wikimedia.org/T357616#9735153 (10JMeybohm) No event logs from the past 24 of either wikikube eqiad or codfw are available in logstash. ` root@deploy1002:~# kubectl -n kube... [12:50:54] 06serviceops, 10MoveComms-Support, 10MW-on-K8s, 06SRE, and 2 others: Move 100% of external traffic to Kubernetes (excluding Votewiki and Commons) - https://phabricator.wikimedia.org/T362323#9735410 (10Trizek-WMF) Checking after #MoveComms-Support was added to this task: what kind of support do you need, if... [14:07:01] 06serviceops, 10[DEPRECATED] wdwb-tech, 10Citoid, 06Content-Transform-Team-WIP, and 9 others: Upgrade mobileapps to node 18 - https://phabricator.wikimedia.org/T363168 (10Jgiannelos) 03NEW [14:07:16] 06serviceops, 10[DEPRECATED] wdwb-tech, 10Citoid, 06Content-Transform-Team-WIP, and 9 others: Upgrade mobileapps to node 18 - https://phabricator.wikimedia.org/T363168#9735740 (10Jgiannelos) a:03Jgiannelos [14:08:06] 06serviceops, 10[DEPRECATED] wdwb-tech, 10Citoid, 06Content-Transform-Team-WIP, and 9 others: Upgrade mobileapps to node 18 - https://phabricator.wikimedia.org/T363168#9735744 (10Jgiannelos) [14:15:48] 06serviceops: Package latest version of prometheus-memcached-exporter (v0.14.2) - https://phabricator.wikimedia.org/T350807#9735790 (10jijiki) 05Open→03Resolved Uploaded new package with the binary named as `prometheus-memcached-exporter` [15:37:26] 06serviceops: Package latest version of prometheus-memcached-exporter (v0.14.2) - https://phabricator.wikimedia.org/T350807#9736164 (10Andrew) Everything seems happy now. Thanks! [16:17:28] 06serviceops, 10Sustainability (Incident Followup): Cache mw-mcrouter service ClusterIP in apcu cache - https://phabricator.wikimedia.org/T363186 (10jijiki) 03NEW [16:19:50] 06serviceops, 06MediaWiki-Engineering, 10Sustainability (Incident Followup): Cache mw-mcrouter service ClusterIP in apcu cache - https://phabricator.wikimedia.org/T363186#9736387 (10Krinkle) [19:19:50] 06serviceops, 06DC-Ops: Q#:rack/setup/install X - https://phabricator.wikimedia.org/T363209 (10RobH) 03NEW [19:21:08] 06serviceops: serviceops kafka-main200[6789] and kafka-main2010 implementation tracking - https://phabricator.wikimedia.org/T363210 (10RobH) 03NEW [19:21:31] 06serviceops, 06DC-Ops: Q#:rack/setup/install kafka-main200[6789] & kafka-main2010 - https://phabricator.wikimedia.org/T363209#9737310 (10RobH) [19:21:53] 06serviceops, 06DC-Ops, 10ops-codfw: Q#:rack/setup/install kafka-main200[6789] & kafka-main2010 - https://phabricator.wikimedia.org/T363209#9737312 (10RobH) [19:22:11] 06serviceops, 06DC-Ops, 10ops-codfw: Q4:rack/setup/install kafka-main200[6789] & kafka-main2010 - https://phabricator.wikimedia.org/T363209#9737314 (10RobH) [19:22:43] 06serviceops, 06DC-Ops, 10ops-codfw: Q4:rack/setup/install kafka-main200[6789] & kafka-main2010 - https://phabricator.wikimedia.org/T363209#9737317 (10RobH) [19:30:48] 06serviceops, 06DC-Ops, 10ops-eqiad: Q4:rack/setup/install kafka-main100[6789] and kafka-main1010 - https://phabricator.wikimedia.org/T363212 (10RobH) 03NEW [19:31:19] 06serviceops, 06DC-Ops, 10ops-eqiad: Q4:rack/setup/install kafka-main100[6789] and kafka-main1010 - https://phabricator.wikimedia.org/T363212#9737376 (10RobH) [19:32:42] 06serviceops: kafka-main100[6789] and kafka-main1010 implementation tracking - https://phabricator.wikimedia.org/T363214 (10RobH) 03NEW