[08:20:34] 06serviceops: Allow dash-suffixes for chart versions - https://phabricator.wikimedia.org/T411411#11422668 (10daniel) >>! In T411411#11421223, @RLazarus wrote: > mmm I agree with you about the problem but I'm not sure about the solution > I think you can still get into confusing versioning semantics w... [10:26:02] 06serviceops, 10observability: Create a visual representation of where each service is active from, any given time - https://phabricator.wikimedia.org/T327663#11422939 (10MLechvien-WMF) a:03MLechvien-WMF [10:32:02] 06serviceops, 10MW-on-K8s: Allow running one-off scripts manually - https://phabricator.wikimedia.org/T341553#11422966 (10Dreamy_Jazz) Not sure if it has already been mentioned, but `mwscript-k8s` doesn't seem to support running maintenance scripts that are not defined in public code. Specifically I have the n... [11:51:35] 06serviceops, 10observability: Create a visual representation of where each service is active from, any given time - https://phabricator.wikimedia.org/T327663#11423199 (10Clement_Goubert) >>! In T327663#11416513, @Raine wrote: > Is anything still needed beyond the functionality in `sudo cookbook -d sre.discove... [12:23:00] 06serviceops, 10MW-on-K8s: Allow running one-off scripts manually - https://phabricator.wikimedia.org/T341553#11423316 (10Tgr) >>! In T341553#11422966, @Dreamy_Jazz wrote: > Not sure if it has already been mentioned, but `mwscript-k8s` doesn't seem to support running maintenance scripts that are not defined in... [13:05:17] 06serviceops, 06Data-Engineering, 06Data-Engineering-Radar, 10Observability-Logging, and 2 others: Fix Kafka replicas skew - https://phabricator.wikimedia.org/T407185#11423504 (10brouberol) ` brouberol@kafka-main2008:~/T407185$ kafka reassign-partitions kafka reassign-partitions --reassignment-json-file ./... [14:19:34] 06serviceops, 10MW-on-K8s: Allow running one-off scripts manually - https://phabricator.wikimedia.org/T341553#11423856 (10Dreamy_Jazz) >>! In T341553#11423316, @Tgr wrote: >>>! In T341553#11422966, @Dreamy_Jazz wrote: >> Not sure if it has already been mentioned, but `mwscript-k8s` doesn't seem to support runn... [14:23:46] 06serviceops, 06Data-Engineering, 06Data-Engineering-Radar, 10Observability-Logging, and 2 others: Fix Kafka replicas skew - https://phabricator.wikimedia.org/T407185#11423873 (10brouberol) One thing to note: we're seeing a lot of traffic already coming out of kafka-main2010, and the rebalancing will add n... [14:36:30] 06serviceops, 10MW-on-K8s: Allow running one-off scripts manually - https://phabricator.wikimedia.org/T341553#11423916 (10Clement_Goubert) >>! In T341553#11423856, @Dreamy_Jazz wrote: >>>! In T341553#11423316, @Tgr wrote: >>>>! In T341553#11422966, @Dreamy_Jazz wrote: >>> Not sure if it has already been mentio... [14:59:19] 06serviceops, 06Connection-Team, 13Patch-For-Review: MediaWiki periodic job campaignevents-aggregateanswers-metawiki failed - https://phabricator.wikimedia.org/T411417#11424035 (10Clement_Goubert) Hmm I'd like to be able to actually see a failed run, so I'll change the job definition to keep runs for longer... [15:13:08] 06serviceops, 06Connection-Team, 13Patch-For-Review: MediaWiki periodic job campaignevents-aggregateanswers-metawiki failed - https://phabricator.wikimedia.org/T411417#11424128 (10Daimona) >>! In T411417#11424035, @Clement_Goubert wrote: > Hmm I'd like to be able to actually see a failed run, so I'll change... [15:31:40] 06serviceops, 06Infrastructure-Foundations, 10netops, 07Puppet: network::constants::mw_appserver_networks is out of date (or named poorly?) - https://phabricator.wikimedia.org/T411508 (10taavi) 03NEW [15:41:24] 06serviceops, 07Puppet: network::constants::mw_appserver_networks is out of date (or named poorly?) - https://phabricator.wikimedia.org/T411508#11424250 (10taavi) [15:49:49] 06serviceops, 07Puppet: network::constants::mw_appserver_networks is out of date (or named poorly?) - https://phabricator.wikimedia.org/T411508#11424318 (10Clement_Goubert) p:05Triage→03Low Probably a good task to pair on with @Blake or @jasmine_ on tracking down what's used where in puppet. [15:58:56] 06serviceops, 10MW-on-K8s: Allow running one-off scripts manually - https://phabricator.wikimedia.org/T341553#11424364 (10Urbanecm_WMF) @Clement_Goubert It works...sometimes. This works: ` [urbanecm@deploy2002 ~]$ mwscript-k8s --file=test.php --follow -- /data/test.php --wiki=testwiki ⏳ Starting /data/test.... [16:00:54] 06serviceops, 10MW-on-K8s: Allow running one-off scripts manually - https://phabricator.wikimedia.org/T341553#11424373 (10Urbanecm_WMF) Hmm...run.php also has problem with this script if it is in maintenance. Okay, so it just works. I'll fill another task to investigate run.php behaviour... [16:12:43] 06serviceops, 10RESTBase Sunsetting, 07Essential-Work, 06MW-Interfaces-Team (MWI-Sprint-24 (2025-12-02 to 2025-12-24)), 13Patch-For-Review: Reroute /api/rest_v1 documentation to REST Sandbox - https://phabricator.wikimedia.org/T396807#11424438 (10HCoplin-WMF) [16:13:40] 06serviceops, 06Data-Engineering, 06Data-Engineering-Radar, 10Observability-Logging, and 2 others: Fix Kafka replicas skew - https://phabricator.wikimedia.org/T407185#11424460 (10brouberol) {F70821449} Storage has been rebalanced on kafka-main. I'll let traffic settle, and will see whether we need to adjus... [16:33:59] 06serviceops, 06Data-Engineering, 06Data-Engineering-Radar, 10Observability-Logging, and 2 others: Fix Kafka replicas skew - https://phabricator.wikimedia.org/T407185#11424574 (10brouberol) I've switched leadership for the `codfw.mediawiki.job.htmlCacheUpdate` topic. ` brouberol@kafka-main2008:~/T407185$... [16:34:06] 06serviceops, 06Data-Engineering, 06Data-Engineering-Radar, 10Observability-Logging, and 2 others: Fix Kafka replicas skew - https://phabricator.wikimedia.org/T407185#11424576 (10brouberol) 05In progress→03Resolved [16:39:08] 06serviceops, 06Data-Engineering, 06Data-Engineering-Radar, 10Observability-Logging, and 2 others: Fix Kafka replicas skew - https://phabricator.wikimedia.org/T407185#11424636 (10Clement_Goubert) Thanks for all the support @brouberol <3 [16:46:54] 06serviceops, 06Data-Engineering, 06Data-Engineering-Radar, 10Observability-Logging, and 2 others: Fix Kafka replicas skew - https://phabricator.wikimedia.org/T407185#11424674 (10brouberol) Anytime! [17:13:09] 06serviceops, 10DiscussionTools, 10MediaWiki-REST-API, 10RESTBase Sunsetting, and 2 others: Reroute RESTBase /page/lint/ endpoints to MediaWiki REST endpoints - https://phabricator.wikimedia.org/T384216#11424936 (10aaron) 05Open→03Resolved [17:42:17] 06serviceops, 06Infrastructure-Foundations, 10SRE-tools, 13Patch-For-Review: Add a --rack flag to sre.k8s.pool-depool-node - https://phabricator.wikimedia.org/T410537#11425151 (10MLechvien-WMF) 05Open→03Resolved [18:20:56] 06serviceops, 10observability: Create a visual representation of where each service is active from, any given time - https://phabricator.wikimedia.org/T327663#11425393 (10MLechvien-WMF) Thanks a lot @Clement_Goubert . The new dashboard is here: https://grafana.wikimedia.org/goto/AyPMrxZDR?orgId=1 It disp... [18:21:05] 06serviceops, 10observability: Create a visual representation of where each service is active from, any given time - https://phabricator.wikimedia.org/T327663#11425394 (10MLechvien-WMF) 05Open→03In progress [18:46:34] 06serviceops, 07Epic, 06MediaWiki-Platform-Team (Kanban Board): Migrate Wikimedia production from PHP 8.1 to PHP 8.3 - https://phabricator.wikimedia.org/T360995#11425537 (10Krinkle) //Progress report for 27 Oct - 24 Nov (four weeks) on WE6.4.8 Support PHP 8.3 upgrade ([m:FY2025-2026#Q2](https://meta.wikimedi... [19:11:36] 06serviceops, 13Patch-For-Review: Migrate the etcd main cluster to cfssl-based PKI - https://phabricator.wikimedia.org/T352245#11425640 (10Scott_French) Alright, that should be everything. All configcluster hosts have had etcd migrated to use cfssl-based PKI, which should unblock migration to Puppet 7. I've a... [19:13:36] 06serviceops, 13Patch-For-Review: Migrate the etcd main cluster to cfssl-based PKI - https://phabricator.wikimedia.org/T352245#11425644 (10Scott_French)