[08:41:47] 10serviceops, 10Content-Transform-Team-WIP, 10Mobile-Content-Service, 10RESTbase Sunsetting, and 3 others: Setup allowed list for MCS decom - https://phabricator.wikimedia.org/T340036 (10akosiaris) >>! In T340036#9062519, @vadim-kovalenko wrote: > Hi there! I'm responsible for Kiwix migration to another AP... [08:46:08] 10serviceops, 10Content-Transform-Team-WIP, 10Mobile-Content-Service, 10RESTbase Sunsetting, and 3 others: Setup allowed list for MCS decom - https://phabricator.wikimedia.org/T340036 (10vadim-kovalenko) @akosiaris , I've updated regexp, and now it works, thank you! [09:08:15] 10serviceops, 10MW-on-K8s, 10SRE, 10Traffic, and 2 others: Direct 5% of all traffic to mw-on-k8s - https://phabricator.wikimedia.org/T341780 (10Clement_Goubert) Pending more hardware, we will move on to 2% first. [09:16:02] 10serviceops, 10MW-on-K8s: mw-on-k8s app container CPU throttling at low average load - https://phabricator.wikimedia.org/T342748 (10Clement_Goubert) Everything looking ok, we will see how it copes with doubling the incoming traffic from {T341780} (only going to 2% for now) and resolve afterwards if everything... [11:26:06] 10serviceops, 10SRE, 10ops-eqiad: Move eqiad thumbor hosts to kubernetes cluster - https://phabricator.wikimedia.org/T343993 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by hnowlan@cumin1001 for host kubernetes1057.eqiad.wmnet with OS bullseye [11:29:41] 10serviceops, 10SRE, 10ops-eqiad: Move eqiad thumbor hosts to kubernetes cluster - https://phabricator.wikimedia.org/T343993 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by hnowlan@cumin1001 for host kubernetes1058.eqiad.wmnet with OS bullseye [11:36:08] 10serviceops, 10SRE, 10ops-codfw: Move codfw thumbor hosts to kubernetes cluster - https://phabricator.wikimedia.org/T343996 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by hnowlan@cumin2002 for host kubernetes2055.codfw.wmnet with OS bullseye [11:40:09] 10serviceops, 10SRE, 10ops-codfw: Move codfw thumbor hosts to kubernetes cluster - https://phabricator.wikimedia.org/T343996 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by hnowlan@cumin2002 for host kubernetes2056.codfw.wmnet with OS bullseye [11:42:32] 10serviceops, 10Data Products, 10RESTbase Sunsetting, 10Code-Health-Objective, 10Patch-For-Review: Route to new AQS Knowledge Gaps endpoint - https://phabricator.wikimedia.org/T342213 (10hnowlan) >>! In T342213#9106475, @fkaelin wrote: > Thanks for the clarification @hnowlan > > I see the deployment ha... [12:22:12] 10serviceops, 10SRE, 10ops-eqiad: Move eqiad thumbor hosts to kubernetes cluster - https://phabricator.wikimedia.org/T343993 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by hnowlan@cumin1001 for host kubernetes1058.eqiad.wmnet with OS bullseye executed with errors: - kubernetes1058 (**... [12:37:00] _joe_: good afternoon, I have updated my docker-pkg change to recognize a per user config file https://gerrit.wikimedia.org/r/c/operations/docker-images/docker-pkg/+/935991 :) [12:39:27] 10serviceops, 10SRE, 10ops-eqiad: Move eqiad thumbor hosts to kubernetes cluster - https://phabricator.wikimedia.org/T343993 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by hnowlan@cumin1001 for host kubernetes1057.eqiad.wmnet with OS bullseye executed with errors: - kubernetes1057 (**... [12:40:07] <_joe_> hashar: ack, I'll take a look [12:54:21] 10serviceops, 10SRE, 10ops-codfw: Move codfw thumbor hosts to kubernetes cluster - https://phabricator.wikimedia.org/T343996 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by hnowlan@cumin2002 for host kubernetes2055.codfw.wmnet with OS bullseye executed with errors: - kubernetes2055 (**... [12:57:00] 10serviceops, 10SRE, 10ops-codfw: Move codfw thumbor hosts to kubernetes cluster - https://phabricator.wikimedia.org/T343996 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by hnowlan@cumin2002 for host kubernetes2056.codfw.wmnet with OS bullseye executed with errors: - kubernetes2056 (**... [14:18:25] 10serviceops, 10SRE, 10ops-eqiad: Move eqiad thumbor hosts to kubernetes cluster - https://phabricator.wikimedia.org/T343993 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by hnowlan@cumin1001 for host kubernetes1057.eqiad.wmnet with OS bullseye [14:19:00] 10serviceops, 10SRE, 10ops-eqiad: Move eqiad thumbor hosts to kubernetes cluster - https://phabricator.wikimedia.org/T343993 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by hnowlan@cumin1001 for host kubernetes1058.eqiad.wmnet with OS bullseye [14:22:27] 10serviceops, 10SRE, 10ops-codfw: Move codfw thumbor hosts to kubernetes cluster - https://phabricator.wikimedia.org/T343996 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by hnowlan@cumin2002 for host kubernetes2055.codfw.wmnet with OS bullseye [14:23:55] 10serviceops, 10collaboration-services, 10GitLab (CI & Job Runners): Standardize Debian package builds on GitLab CI - https://phabricator.wikimedia.org/T304491 (10MatthewVernon) OK, I have enough progress to warrant an update here, I think. Short version: you can now arrange for CI to check Debian for updat... [14:25:25] 10serviceops, 10SRE, 10ops-codfw: Move codfw thumbor hosts to kubernetes cluster - https://phabricator.wikimedia.org/T343996 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by hnowlan@cumin2002 for host kubernetes2056.codfw.wmnet with OS bullseye [14:50:33] 10serviceops, 10SRE, 10ops-eqiad: Move eqiad thumbor hosts to kubernetes cluster - https://phabricator.wikimedia.org/T343993 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by hnowlan@cumin1001 for host kubernetes1057.eqiad.wmnet with OS bullseye completed: - kubernetes1057 (**WARN**) -... [14:53:12] 10serviceops, 10SRE, 10ops-eqiad: Move eqiad thumbor hosts to kubernetes cluster - https://phabricator.wikimedia.org/T343993 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by hnowlan@cumin1001 for host kubernetes1058.eqiad.wmnet with OS bullseye completed: - kubernetes1058 (**PASS**) -... [15:04:29] 10serviceops, 10SRE, 10ops-codfw, 10Patch-For-Review: Move codfw thumbor hosts to kubernetes cluster - https://phabricator.wikimedia.org/T343996 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by hnowlan@cumin2002 for host kubernetes2055.codfw.wmnet with OS bullseye completed: - kuberne... [15:07:28] 10serviceops, 10SRE, 10ops-codfw, 10Patch-For-Review: Move codfw thumbor hosts to kubernetes cluster - https://phabricator.wikimedia.org/T343996 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by hnowlan@cumin2002 for host kubernetes2056.codfw.wmnet with OS bullseye completed: - kuberne... [15:42:35] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q1:rack/setup/install kubernetes20[25-54] - https://phabricator.wikimedia.org/T342534 (10Jhancock.wm) [16:40:24] 10serviceops, 10Content-Transform-Team-WIP, 10Maintenance-Worktype, 10Wikimedia-Incident: Maps Unavailability due to thanos-swift cfssl rollout (14 Aug 2023) - https://phabricator.wikimedia.org/T344324 (10jijiki) [16:41:51] 10serviceops, 10Content-Transform-Team-WIP, 10Maintenance-Worktype, 10Wikimedia-Incident: Maps Unavailability due to thanos-swift cfssl rollout (14 Aug 2023) - https://phabricator.wikimedia.org/T344324 (10jijiki) From the 18th Aug rollout, tcpdumps can be found in my home directory on kubernetes2010 and t... [16:46:20] 10serviceops, 10Content-Transform-Team-WIP, 10Maintenance-Worktype, 10Wikimedia-Incident: Maps Unavailability due to thanos-swift cfssl rollout (14 Aug 2023) - https://phabricator.wikimedia.org/T344324 (10jijiki) [16:47:22] 10serviceops, 10Content-Transform-Team-WIP, 10Maintenance-Worktype, 10Wikimedia-Incident: Maps Unavailability due to thanos-swift cfssl rollout (14 Aug 2023) - https://phabricator.wikimedia.org/T344324 (10jijiki) [16:47:46] 10serviceops, 10Content-Transform-Team-WIP, 10Maintenance-Worktype, 10Wikimedia-Incident: Maps Unavailability due to thanos-swift cfssl rollout (14 Aug 2023) - https://phabricator.wikimedia.org/T344324 (10jijiki) 05Open→03In progress p:05Triage→03High [16:55:56] 10serviceops, 10SRE, 10ops-codfw: Move codfw thumbor hosts to kubernetes cluster - https://phabricator.wikimedia.org/T343996 (10hnowlan) 05Open→03Resolved a:03hnowlan [16:56:03] 10serviceops, 10SRE, 10Thumbor, 10Thumbor Migration, 10Platform Team Workboards (Platform Engineering Reliability): Final steps for fully-Kubernetes Thumbor - https://phabricator.wikimedia.org/T334488 (10hnowlan) [16:56:15] 10serviceops, 10SRE, 10ops-eqiad: Move eqiad thumbor hosts to kubernetes cluster - https://phabricator.wikimedia.org/T343993 (10hnowlan) [17:03:09] 10serviceops, 10MediaWiki-General, 10MediaWiki-libs-Stats, 10SRE, and 5 others: MediaWiki Prometheus support - https://phabricator.wikimedia.org/T240685 (10colewhite) [17:09:48] 10serviceops, 10SRE, 10Thumbor, 10Thumbor Migration, and 2 others: Final steps for fully-Kubernetes Thumbor - https://phabricator.wikimedia.org/T334488 (10hnowlan) 05Open→03Resolved [17:09:59] 10serviceops, 10SRE, 10Thumbor, 10Thumbor Migration, and 2 others: Migrate thumbor to Kubernetes - https://phabricator.wikimedia.org/T233196 (10hnowlan) [17:35:33] 10serviceops, 10Content-Transform-Team-WIP, 10Maintenance-Worktype, 10Wikimedia-Incident: Maps Unavailability due to thanos-swift cfssl rollout (14 Aug 2023) - https://phabricator.wikimedia.org/T344324 (10Krinkle) [17:48:35] 10serviceops, 10MediaWiki-General, 10MediaWiki-libs-Stats, 10SRE, and 5 others: MediaWiki Prometheus support - https://phabricator.wikimedia.org/T240685 (10colewhite) [19:14:32] 10serviceops, 10AQS2.0, 10Cassandra, 10SRE, 10Service-deployment-requests: AQS 2.0 differentially private pageviews deploy API - https://phabricator.wikimedia.org/T343855 (10Htriedman) Hi all! It's been a few weeks without activity, so I'm following up on this request. It seems to me there are two remai...