[06:16:29] 10serviceops: Set up kubernetes10[27-56] - https://phabricator.wikimedia.org/T346714 (10Joe) [06:26:05] 10serviceops: Set up kubernetes10[27-56] - https://phabricator.wikimedia.org/T346714 (10Joe) p:05Triage→03High [07:18:20] hello, I have a set of changes for the deployment server script: `fix-staging-perms` which is used by non root to fix permissions on /srv/mediawiki-staging [07:18:43] I don't know who would be best suited to review / deploy them, then given that is the deplokyment servers I am guessing serviceops is a good bet. The series start at https://gerrit.wikimedia.org/r/c/operations/puppet/+/927674/ [07:18:54] they are noop in the pcc as far as I can tell (which is expected) [07:19:09] ahmon reviewed them [07:22:50] <_joe_> hashar: you will have to be patient, this is switchover week, we all have our hands full [07:23:13] ah [07:24:56] <_joe_> and yes please wait for our review [07:25:25] is there anyone specifically I can add as reviewers? [07:25:29] <_joe_> me [07:25:31] <_joe_> :) [07:25:36] NOOOOO :D [07:26:34] they are not urgent, those patches follow up an issue I had back in June and it only happens once in a while [07:27:01] I have added you all on all three changes. Happy switch over! [08:23:34] 10serviceops, 10MediaWiki-Platform-Team: Migrate WMF production from PHP 7.4 to PHP 8.1 - https://phabricator.wikimedia.org/T319432 (10Tacsipacsi) >>! In T319432#8338918, @Krinkle wrote: > We choose PHP 8.1, based on PHP 8.2 not being ready yet In the meantime, 8.2 has been out for over nine months, and [[htt... [09:03:41] 10serviceops, 10Foundational Technology Requests, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: Kubernetes v1.23 use PKI for service-account signing (instead of cergen) - https://phabricator.wikimedia.org/T329826 (10JMeybohm) [09:17:55] 10serviceops, 10RESTBase Sunsetting, 10Code-Health-Objective, 10Data Products (Sprint 01), 10Patch-For-Review: Route to new AQS Knowledge Gaps endpoint - https://phabricator.wikimedia.org/T342213 (10Sfaci) I'm sorry but I cannot say much more than you have said about these headers. I don't know the detai... [09:59:29] 10serviceops, 10Foundational Technology Requests, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: Kubernetes v1.23 use PKI for service-account signing (instead of cergen) - https://phabricator.wikimedia.org/T329826 (10JMeybohm) [10:01:50] 10serviceops, 10Foundational Technology Requests, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: Post Kubernetes v1.23 cleanup - https://phabricator.wikimedia.org/T328291 (10JMeybohm) [10:01:53] 10serviceops, 10Foundational Technology Requests, 10Prod-Kubernetes, 10Shared-Data-Infrastructure, and 2 others: Update Kubernetes clusters to v1.23 - https://phabricator.wikimedia.org/T307943 (10JMeybohm) [10:02:04] 10serviceops, 10Foundational Technology Requests, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: Kubernetes v1.23 use PKI for service-account signing (instead of cergen) - https://phabricator.wikimedia.org/T329826 (10JMeybohm) 05Open→03Resolved Removed all the certs with `[puppet-private] (23d9... [10:02:08] 10serviceops, 10Foundational Technology Requests, 10Prod-Kubernetes, 10Shared-Data-Infrastructure, and 2 others: Update Kubernetes clusters to v1.23 - https://phabricator.wikimedia.org/T307943 (10JMeybohm) [10:02:18] 10serviceops, 10Foundational Technology Requests, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: Post Kubernetes v1.23 cleanup - https://phabricator.wikimedia.org/T328291 (10JMeybohm) 05Open→03Resolved [10:02:36] 🎉 - finally [10:17:41] 10serviceops, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: Use cert-manager for service-proxy certificate creation - https://phabricator.wikimedia.org/T300033 (10JMeybohm) [11:03:16] 10serviceops, 10MW-on-K8s: mcrouter daemonset of mw-on-k8s - https://phabricator.wikimedia.org/T346690 (10Aklapper) [11:11:07] jayme: niceee [11:42:22] 10serviceops, 10SRE, 10Datacenter-Switchover, 10Patch-For-Review: Sept 2023 Switchover Checklist: Services & Traffic - https://phabricator.wikimedia.org/T346330 (10kamila) [12:20:22] 10serviceops, 10SRE, 10Datacenter-Switchover, 10Patch-For-Review: Sept 2023 Switchover Checklist: Services & Traffic - https://phabricator.wikimedia.org/T346330 (10kamila) [12:20:37] 10serviceops, 10RESTBase Sunsetting, 10Code-Health-Objective, 10Data Products (Sprint 01), 10Patch-For-Review: Route to new AQS Knowledge Gaps endpoint - https://phabricator.wikimedia.org/T342213 (10SGupta-WMF) a:05Milimetric→03BTullis [12:21:51] 10serviceops, 10SRE, 10Datacenter-Switchover, 10Patch-For-Review: Sept 2023 Switchover Checklist: MediaWiki - https://phabricator.wikimedia.org/T346474 (10kamila) [14:01:31] 10serviceops, 10SRE, 10Datacenter-Switchover, 10Patch-For-Review: Sept 2023 Switchover Checklist: Services & Traffic - https://phabricator.wikimedia.org/T346330 (10ops-monitoring-bot) kamila@cumin1001 - Cookbook cookbooks.sre.discovery.datacenter depool all services in eqiad: Datacenter Switchover: Service... [14:01:51] 10serviceops, 10Cloud-VPS, 10Infrastructure-Foundations, 10Packaging, 10cloud-services-team (FY2023/2024-Q1): Package mcrouter for Debian Bookworm - https://phabricator.wikimedia.org/T346762 (10fnegri) [14:09:03] 10serviceops, 10Cloud-VPS, 10Infrastructure-Foundations, 10Packaging, 10cloud-services-team (FY2023/2024-Q1): Package mcrouter for Debian Bookworm - https://phabricator.wikimedia.org/T346762 (10fnegri) 05Open→03In progress p:05Triage→03High [14:28:58] 10serviceops, 10SRE, 10Datacenter-Switchover, 10Patch-For-Review: Sept 2023 Switchover Checklist: Services & Traffic - https://phabricator.wikimedia.org/T346330 (10ops-monitoring-bot) kamila@cumin1001 - Cookbook cookbooks.sre.discovery.datacenter depool all services in eqiad: Datacenter Switchover: Service... [14:38:57] 10serviceops, 10Machine-Learning-Team, 10Patch-For-Review: Rename the envoy's uses_ingress option to sets_sni - https://phabricator.wikimedia.org/T346638 (10calbon) a:03elukey [14:39:15] 10serviceops, 10SRE, 10Datacenter-Switchover, 10Patch-For-Review: Sept 2023 Switchover Checklist: Services & Traffic - https://phabricator.wikimedia.org/T346330 (10kamila) [14:41:18] akosiaris: the thumbor change happened in https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/914737 [14:41:32] not sure why only on codfw [14:41:39] *eqiad - sorry [14:42:00] probably debugging and trying to fix the issues [14:42:14] we should be able to mirror it in codfw, but doublechecking [14:43:17] the problem is usually not with normal load but during mw deployments as those eat quite a bit into the resource budget [14:43:36] so having the capacity rn is not a good metric [14:44:22] <_joe_> I agree [14:44:26] and AIUI we had issues deploying mw just today [14:44:34] (because of capacity) [14:44:38] yes we just did [14:44:49] I'd say we leave it alone until the new nodes have been added [14:50:56] 10serviceops, 10RESTBase Sunsetting, 10Code-Health-Objective, 10Data Products (Sprint 01), 10Patch-For-Review: Route to new AQS Knowledge Gaps endpoint - https://phabricator.wikimedia.org/T342213 (10Milimetric) a:05BTullis→03Milimetric Ok, I hear Ben's concerns but kind of decided to risk updating ev... [15:02:53] 10serviceops, 10observability, 10Patch-For-Review, 10SRE Observability (FY2023/2024-Q1): Hardcode the SLO time windows in Grafana dashboards generated via Grizzly - https://phabricator.wikimedia.org/T346144 (10klausman) I am very in favor of this scheme. As for implementation, maybe one option would be to... [15:07:29] 10serviceops, 10SRE, 10Datacenter-Switchover, 10Patch-For-Review: Sept 2023 Switchover Checklist: Services & Traffic - https://phabricator.wikimedia.org/T346330 (10kamila) [16:28:33] 10serviceops, 10SRE, 10Datacenter-Switchover: Sept 2023 Switchover Checklist: Services & Traffic - https://phabricator.wikimedia.org/T346330 (10kamila) [16:31:34] 10serviceops, 10SRE, 10Datacenter-Switchover: Sept 2023 Switchover Checklist: Services & Traffic - https://phabricator.wikimedia.org/T346330 (10kamila) 05Open→03Resolved While there are some outstanding issues due to lack of capacity in codfw, overall we're done here :-) [16:31:40] 10serviceops, 10Data-Persistence, 10Performance-Team, 10SRE, 10Datacenter-Switchover: September 2023 Datacenter Switchover - https://phabricator.wikimedia.org/T345263 (10kamila) [19:17:22] 10serviceops, 10Data-Engineering, 10Discovery-Search (Current work), 10Event-Platform: Improve the flink-app chart to provide more useful defaults - https://phabricator.wikimedia.org/T346315 (10RKemper) [23:27:22] 10serviceops, 10AQS2.0, 10Cassandra, 10SRE, 10Service-deployment-requests: AQS 2.0 differentially private pageviews deploy API - https://phabricator.wikimedia.org/T343855 (10Eevans) Apologies for the amount of time that has passed, I only just noticed this ticket. I took a quick scan of the repo and hav...