[07:02:36] brett: work on moving the control planes to hardware (which should mitigate the increased startup latency of the apiservers causing this) should be unblocked this week. So we should be able to move them in next week [09:25:10] 06serviceops, 06MW-Interfaces-Team, 06Traffic: map the /api/ prefix to /w/rest.php - https://phabricator.wikimedia.org/T364400#9779996 (10hnowlan) This seems like a pretty reasonable idea to me. Could we implement this remapping at the ATS layer rather than the Apache one, in a manner that would mean that wh... [10:17:27] 06serviceops, 10MW-on-K8s, 13Patch-For-Review: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074#9780139 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by hnowlan@cumin1002 for host mw2396.codfw.wmnet with OS bullseye [10:17:29] 06serviceops, 10MW-on-K8s, 13Patch-For-Review: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074#9780140 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by hnowlan@cumin1002 for host mw2397.codfw.wmnet with OS bullseye [10:17:30] 06serviceops, 10MW-on-K8s, 13Patch-For-Review: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074#9780141 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by hnowlan@cumin1002 for host mw2401.codfw.wmnet with OS bullseye [10:17:31] 06serviceops, 10MW-on-K8s, 13Patch-For-Review: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074#9780142 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by hnowlan@cumin1002 for host mw2398.codfw.wmnet with OS bullseye [10:17:32] 06serviceops, 10MW-on-K8s, 13Patch-For-Review: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074#9780143 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by hnowlan@cumin1002 for host mw2402.codfw.wmnet with OS bullseye [10:17:34] 06serviceops, 10MW-on-K8s, 13Patch-For-Review: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074#9780144 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by hnowlan@cumin1002 for host mw2399.codfw.wmnet with OS bullseye [10:55:29] 06serviceops, 10MW-on-K8s, 13Patch-For-Review: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074#9780245 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by hnowlan@cumin1002 for host mw2396.codfw.wmnet with OS bullseye completed: - mw239... [10:57:55] 06serviceops, 10MW-on-K8s, 13Patch-For-Review: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074#9780261 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by hnowlan@cumin1002 for host mw2401.codfw.wmnet with OS bullseye completed: - mw240... [11:00:47] 06serviceops, 10MW-on-K8s, 13Patch-For-Review: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074#9780275 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by hnowlan@cumin1002 for host mw2398.codfw.wmnet with OS bullseye completed: - mw239... [11:02:47] 06serviceops, 10MW-on-K8s, 13Patch-For-Review: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074#9780280 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by hnowlan@cumin1002 for host mw2402.codfw.wmnet with OS bullseye completed: - mw240... [11:05:57] 06serviceops, 10MW-on-K8s, 13Patch-For-Review: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074#9780284 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by hnowlan@cumin1002 for host mw2399.codfw.wmnet with OS bullseye completed: - mw239... [11:10:04] 06serviceops, 10MW-on-K8s, 13Patch-For-Review: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074#9780309 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by hnowlan@cumin1002 for host mw2397.codfw.wmnet with OS bullseye completed: - mw239... [12:06:09] hi serviceops - during the hackathon last weekend I was asked to review/deploy some old puppet patches (the 3 patches ending at https://gerrit.wikimedia.org/r/c/operations/puppet/+/527915) that all add redirects for the standard language codes to wikis that use a non-standard code for some reason. they all seem fine to me on a quick glance, I'm [12:06:10] happy to deploy them unless you have any concerns or prefer to do that yourself [12:48:25] 06serviceops, 06MW-Interfaces-Team, 06Traffic: map the /api/ prefix to /w/rest.php - https://phabricator.wikimedia.org/T364400#9780622 (10BBlack) >>! In T364400#9779996, @hnowlan wrote: > Could we implement this remapping at the ATS layer rather than the Apache one, in a manner that would mean that when we n... [13:19:20] 06serviceops, 06MW-Interfaces-Team, 06Traffic: map the /api/ prefix to /w/rest.php - https://phabricator.wikimedia.org/T364400#9780805 (10daniel) >>! In T364400#9779996, @hnowlan wrote: > This seems like a pretty reasonable idea to me. Could we implement this remapping at the ATS layer rather than the Apache... [13:55:52] 06serviceops, 06collaboration-services, 06Infrastructure-Foundations, 10Puppet-Core, and 5 others: Migrate roles to puppet7 - https://phabricator.wikimedia.org/T349619#9780908 (10MoritzMuehlenhoff) [13:57:30] 06serviceops, 06Content-Transform-Team, 07Essential-Work, 07Wikimedia-Incident: Maps Unavailability due to thanos-swift cfssl rollout (14 Aug 2023) - https://phabricator.wikimedia.org/T344324#9780912 (10elukey) >>! In T344324#9507114, @jijiki wrote: > Stalled until T356412 is picked up by #data-persistenc... [14:27:53] 06serviceops, 10Prod-Kubernetes, 07Kubernetes: Allow to address Kubernets API servers from NetworkPolicy - https://phabricator.wikimedia.org/T287491#9780973 (10jijiki) [14:32:30] 06serviceops, 10Prod-Kubernetes, 07Kubernetes: Allow to address Kubernets API servers from NetworkPolicy - https://phabricator.wikimedia.org/T287491#9780996 (10jijiki) [14:53:51] jayme: Okay, thanks for the response! Would it be worth the effort to loosen the alerting for this for the next few weeks? [14:56:17] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q4:rack/setup/install kafka-main200[6789] & kafka-main2010 - https://phabricator.wikimedia.org/T363209#9781043 (10Jhancock.wm) [14:59:06] 06serviceops: Provide nodejs20 base images for production - https://phabricator.wikimedia.org/T362681#9781064 (10MoritzMuehlenhoff) I kicked off a build of the node20 image, it should hopefully appear in the registry soon. [15:48:11] brett: not sure. I'm hoping to complete the transition to hardware next week, so maybe that's enough [16:23:04] 06serviceops, 10CirrusSearch, 03Discovery-Search (Current work), 13Patch-For-Review: Implement global ratelimiting in our service mesh - https://phabricator.wikimedia.org/T362310#9781316 (10JMeybohm) a:03JMeybohm [17:18:15] 06serviceops, 06Data-Platform-SRE, 06Traffic: Investigate why pools.json does not match https://config-master.wikimedia.org/pybal/${datacenter}/${service} T363702 - https://phabricator.wikimedia.org/T364037#9781567 (10bking) 05Open→03Invalid Recording @BBlack 's explanation of why this is from #wikim...