[00:36:43] 06serviceops, 07Datacenter-Switchover: imagecatalog_record.service fails due to read-only sqlite database - https://phabricator.wikimedia.org/T360652#9652540 (10RLazarus) Curious: As @Clement_Goubert and I discussed, both the directory (via puppet `file`) and the database file (via puppet `exec` of `imagecatal... [07:07:29] <_joe_> mutante: that's not an hiccup, even [07:07:40] <_joe_> as in WONTFIX [07:16:15] 06serviceops, 06Data Products: Service Ops Review of Metrics Platform Configuration Management UI - https://phabricator.wikimedia.org/T358577#9653060 (10akosiaris) Hi @MShilova_WMF. This is on my list for today, it might spill into early next week though. I 've started the review but I don't see to have access... [08:44:45] 06serviceops, 06collaboration-services, 06Infrastructure-Foundations, 10Puppet-Core, and 4 others: Migrate roles to puppet7 - https://phabricator.wikimedia.org/T349619#9653145 (10Gehel) [08:45:34] 06serviceops, 10Prod-Kubernetes, 10Data-Platform-SRE (2024.03.25 - 2024.04.14), 07Kubernetes, 13Patch-For-Review: Create the required namespaces within each Kubernetes cluster - https://phabricator.wikimedia.org/T360508#9653147 (10Gehel) [08:47:00] 06serviceops, 10Prod-Kubernetes, 10Data-Platform-SRE (2024.03.25 - 2024.04.14), 07Kubernetes, 13Patch-For-Review: Migrate an example chart to the Calico network policies template - https://phabricator.wikimedia.org/T359411#9653161 (10Gehel) [08:48:09] 06serviceops, 10Prod-Kubernetes, 10Data-Platform-SRE (2024.03.25 - 2024.04.14), 07Kubernetes, 13Patch-For-Review: Improve how we address outside k8s infrastructure from within charts (e.g. network policies) - https://phabricator.wikimedia.org/T331894#9653169 (10Gehel) [08:48:53] 06serviceops, 10Prod-Kubernetes, 10Data-Platform-SRE (2024.03.25 - 2024.04.14), 07Kubernetes: Add redis (rdb) instances to external-services - https://phabricator.wikimedia.org/T360612#9653211 (10Gehel) [08:50:46] 06serviceops, 10CirrusSearch, 06Discovery-Search, 10Data-Platform-SRE (2024.03.25 - 2024.04.14): Requesting permission to enable kafka log compaction for page_rerender on kafka-main - https://phabricator.wikimedia.org/T354794#9653228 (10Gehel) [09:00:00] 06serviceops, 06Data-Engineering, 06Data-Platform-SRE, 06SRE, 10Event-Platform: DRY kafka broker declaration in helmfiles - https://phabricator.wikimedia.org/T253058#9653302 (10Gehel) [09:00:12] 06serviceops, 10Prod-Kubernetes, 10Data-Platform-SRE (2024.03.25 - 2024.04.14), 07Kubernetes, 13Patch-For-Review: Improve how we address outside k8s infrastructure from within charts (e.g. network policies) - https://phabricator.wikimedia.org/T331894#9653303 (10Gehel) [10:18:21] 06serviceops, 10Prod-Kubernetes, 10Data-Platform-SRE (2024.03.25 - 2024.04.14), 07Kubernetes, 13Patch-For-Review: Improve how we address outside k8s infrastructure from within charts (e.g. network policies) - https://phabricator.wikimedia.org/T331894#9653470 (10brouberol) [10:18:43] 06serviceops, 10Prod-Kubernetes, 10Data-Platform-SRE (2024.03.25 - 2024.04.14), 07Kubernetes, 13Patch-For-Review: 14Create the required namespaces within each Kubernetes cluster - 14https://phabricator.wikimedia.org/T360508#9653468 (10brouberol) 05Open→03Resolved 14` root@deploy1002:/srv/deploym... [10:29:01] 06serviceops, 10ChangeProp, 10MW-on-K8s, 06SRE, and 2 others: Alter changeprop chart to use the service mesh - https://phabricator.wikimedia.org/T360625#9653492 (10Clement_Goubert) That makes sense. I don't necessarily have a problem with it not using the service mesh (except for the lack of telemetry), ex... [10:52:17] 06serviceops, 07Datacenter-Switchover: imagecatalog_record.service fails due to read-only sqlite database - https://phabricator.wikimedia.org/T360652#9653555 (10Clement_Goubert) Checking on `deploy2002` (which we moved away from with this switchover), the `catalog.sqlite` files stays in place after a switchove... [11:21:54] 06serviceops, 10MW-on-K8s, 06SRE, 06Traffic, 10Release-Engineering-Team (Seen): Move 70% of mediawiki external requests to mw on k8s - https://phabricator.wikimedia.org/T360763 (10Clement_Goubert) 03NEW [11:22:11] 06serviceops, 10MW-on-K8s, 06SRE, 06Traffic, 10Release-Engineering-Team (Seen): Serve production traffic via Kubernetes - https://phabricator.wikimedia.org/T290536#9653618 (10Clement_Goubert) [11:23:35] 06serviceops, 10MW-on-K8s, 06SRE, 06Traffic, 10Release-Engineering-Team (Seen): Move 70% of mediawiki external requests to mw on k8s - https://phabricator.wikimedia.org/T360763#9653616 (10Clement_Goubert) 05Open→03In progress p:05Triage→03High [11:25:00] 06serviceops, 10MW-on-K8s, 06SRE, 06Traffic, 10Release-Engineering-Team (Seen): Move 70% of mediawiki external requests to mw on k8s - https://phabricator.wikimedia.org/T360763#9653621 (10Clement_Goubert) Waiting on `codfw` repool as part of {T357547} before moving forward with this increase. [12:52:01] 06serviceops, 10MW-on-K8s, 06SRE, 06Traffic, 10Release-Engineering-Team (Seen): Serve production traffic via Kubernetes - https://phabricator.wikimedia.org/T290536#9653629 (10Clement_Goubert) [12:57:07] 06serviceops, 10MW-on-K8s, 06SRE, 06Traffic, and 2 others: Migrate internal traffic to k8s - https://phabricator.wikimedia.org/T333120#9653700 (10Clement_Goubert) [12:57:47] 06serviceops, 10ChangeProp, 10MW-on-K8s, 06SRE, and 2 others: 14Alter changeprop chart to use the service mesh - 14https://phabricator.wikimedia.org/T360625#9653698 (10Clement_Goubert) 05Open→03Declined 14Abandoned because the internals of changeprop make it unadvisable to add another layer. I'll... [12:58:40] 06serviceops, 10MW-on-K8s, 06SRE, 06Traffic, 10Release-Engineering-Team (Seen): Migrate changeprop to mw-api-int - https://phabricator.wikimedia.org/T360767 (10Clement_Goubert) 03NEW [12:59:00] 06serviceops, 10MW-on-K8s, 06SRE, 06Traffic, 10Release-Engineering-Team (Seen): Migrate changeprop to mw-api-int - https://phabricator.wikimedia.org/T360767#9653739 (10Clement_Goubert) p:05Triage→03High [13:02:36] 06serviceops, 06Data Products: Service Ops Review of Metrics Platform Configuration Management UI - https://phabricator.wikimedia.org/T358577#9653832 (10akosiaris) Hi, I 've already left various comments on the 2 docs. I am still going through the Miro board, but I can summarize the following: This proposal... [13:10:52] 06serviceops, 10MW-on-K8s, 06SRE, 06Traffic, and 2 others: Migrate internal traffic to k8s - https://phabricator.wikimedia.org/T333120#9653741 (10Clement_Goubert) [13:11:04] 06serviceops, 10MW-on-K8s, 06SRE, 06Traffic, and 2 others: Migrate internal traffic to k8s - https://phabricator.wikimedia.org/T333120#9653763 (10Clement_Goubert) [13:12:32] 06serviceops, 10MW-on-K8s, 06SRE, 06Traffic, 10Release-Engineering-Team (Seen): Move 70% of mediawiki external requests to mw on k8s - https://phabricator.wikimedia.org/T360763#9653845 (10Clement_Goubert) Given we have increased `mw-web` and `mw-api-ext` by respectively 53 and 10 replicas to cope with ha... [13:34:21] 06serviceops, 10Prod-Kubernetes, 10Data-Platform-SRE (2024.03.25 - 2024.04.14), 07Kubernetes, 13Patch-For-Review: 14Create a networkpolicy template allowing charts to define a Calico Network policy to external services - 14https://phabricator.wikimedia.org/T359334#9654009 (10brouberol) 05Open→03Re... [13:41:51] 06serviceops, 10Prod-Kubernetes, 10Data-Platform-SRE (2024.03.25 - 2024.04.14), 07Kubernetes, 13Patch-For-Review: Improve how we address outside k8s infrastructure from within charts (e.g. network policies) - https://phabricator.wikimedia.org/T331894#9654016 (10brouberol) @JMeybohm and myself have deploy... [13:44:35] 06serviceops, 10Prod-Kubernetes, 10Data-Platform-SRE (2024.03.25 - 2024.04.14), 07Kubernetes, 13Patch-For-Review: Improve how we address outside k8s infrastructure from within charts (e.g. network policies) - https://phabricator.wikimedia.org/T331894#9654039 (10BTullis) Nice! [14:14:15] 06serviceops, 10iPoid-Service, 10Observability-Logging, 13Patch-For-Review: Logs from containers sometimes not visible in logstash - https://phabricator.wikimedia.org/T357616#9654087 (10JMeybohm) Unfortunately we did not gain any insides from the new metrics (dashboard at https://grafana-rw.wikimedia.org/d... [14:16:24] 06serviceops, 06Data-Engineering, 06Data-Platform-SRE, 06SRE, 10Event-Platform: DRY kafka broker declaration in helmfiles - https://phabricator.wikimedia.org/T253058#9654089 (10brouberol) Starting today (at least for the `staging-codfw` and `dse-k8s-eqiad` clusters), apps running in Kubernetes we can use...