[09:19:57] 06serviceops, 10LPL Essential (LPL Essential 2025 Feb-Mar): Migrate language_and_product_localization jobs to mw-cron - https://phabricator.wikimedia.org/T388539#10631834 (10PWaigi-WMF) [09:46:57] 06serviceops, 07Datacenter-Switchover, 07User-notice: MoveComms support for March 2025 Datacentre switchover - https://phabricator.wikimedia.org/T387444#10631896 (10Trizek-WMF) Suggested by @tacsipacsi on the message talk page - change the title of the message sent to communities from > Your wiki will be in... [10:20:28] 06serviceops, 10Data-Engineering-Roadmap, 06Data-Platform-SRE, 10Dumps-Generation, and 3 others: WE 5.4 KR - Hypothesis 5.4.4 - Q3 FY24/25 - Migrate current-generation dumps to run on kubernetes - https://phabricator.wikimedia.org/T352650#10631977 (10BTullis) [12:24:12] 06serviceops, 06Language and Product Localization: Migrate language_and_product_localization jobs to mw-cron - https://phabricator.wikimedia.org/T388539#10632308 (10Nikerabbit) `mediawiki_job_updatetranslationstats.timer` seems okay for removal. `mediawiki_job_purge_old_cx_drafts.timer` has had some issues in... [12:28:57] 06serviceops, 06Language and Product Localization: Migrate language_and_product_localization jobs to mw-cron - https://phabricator.wikimedia.org/T388539#10632342 (10Clement_Goubert) >>! In T388539#10632308, @Nikerabbit wrote: > `mediawiki_job_updatetranslationstats.timer` seems okay for removal. `mediawiki_job... [12:29:17] 06serviceops, 06Language and Product Localization: Migrate language_and_product_localization jobs to mw-cron - https://phabricator.wikimedia.org/T388539#10632344 (10Clement_Goubert) [12:33:27] 06serviceops, 07sre-alert-triage: Alert in need of triage: Postgres Replication Lag (instance maps-test2002) - https://phabricator.wikimedia.org/T388782 (10LSobanski) 03NEW [12:57:16] 06serviceops, 13Patch-For-Review: MediaWiki on PHP 8.1 production traffic ramp-up - https://phabricator.wikimedia.org/T383845#10632529 (10matmarex) [13:05:59] 06serviceops, 10Citoid, 06Editing-team, 10RESTBase Sunsetting, and 2 others: Switchover plan from restbase to api gateway for Citoid - https://phabricator.wikimedia.org/T361576#10632561 (10hnowlan) Citoid is now fully routed via the rest gateway for all wikis. [14:03:01] 06serviceops, 10MW-on-K8s, 10Observability-Logging, 07Kubernetes: Move rsyslog-generated mediawiki logs within k8s to their own kafka topics - https://phabricator.wikimedia.org/T384335#10632917 (10fgiunchedi) @JMeybohm re: the above, what `.Values` could I use in `charts/mediawiki/templates/rsyslog/configm... [14:51:14] 06serviceops: php-wmerrors rsyslog rule selects on php7 only - https://phabricator.wikimedia.org/T388799 (10fgiunchedi) 03NEW [15:10:35] 06serviceops, 07Datacenter-Switchover, 07User-notice: MoveComms support for March 2025 Datacentre switchover - https://phabricator.wikimedia.org/T387444#10633295 (10Trizek-WMF) [15:20:34] 06serviceops: Update api-gateway ratelimit version - https://phabricator.wikimedia.org/T388804 (10hnowlan) 03NEW [15:32:21] 06serviceops, 07Datacenter-Switchover, 07User-notice: MoveComms support for March 2025 Datacentre switchover - https://phabricator.wikimedia.org/T387444#10633442 (10Trizek-WMF) I checked all the times on the translations. The message can be sent to communities. [15:33:11] 06serviceops, 07Datacenter-Switchover, 07User-notice: MoveComms support for March 2025 Datacentre switchover - https://phabricator.wikimedia.org/T387444#10633452 (10Trizek-WMF) @hnowlan, I let you announce the read-only time to all staff (see task description)? [15:33:33] 06serviceops, 10Page Content Service, 10Content-Transform-Team (Work In Progress), 13Patch-For-Review: Pregeneration rules don't pregenerate caches for the same cases restbase did - https://phabricator.wikimedia.org/T388214#10633455 (10Jgiannelos) So far I've tested: * Editing a page -> edit shows up in m... [16:40:56] 06serviceops, 13Patch-For-Review: php-wmerrors rsyslog rule selects on php7 only - https://phabricator.wikimedia.org/T388799#10633863 (10Scott_French) Thanks for catching that, @fgiunchedi! Yes, that was, alas, an oversight. [16:45:28] 06serviceops, 06collaboration-services, 06Data-Platform-SRE, 10Prod-Kubernetes, 07Kubernetes: Ensure all required kubectl versions are installed on deploy hosts - https://phabricator.wikimedia.org/T388388#10633887 (10JMeybohm) Running puppet agent with --debug --trace suggests that `apt-cache madison kub... [16:55:54] 06serviceops, 06Infrastructure-Foundations, 10Maps (Kartotherian): Scale up Kartotherian on Wikikube and move live traffic to it - https://phabricator.wikimedia.org/T386926#10633932 (10elukey) It seems that mapnik.Image does indeed allocate native/external (to nodejs) memory that is not reclaimable by the GC... [16:57:55] 06serviceops, 06collaboration-services, 06Data-Platform-SRE, 10Prod-Kubernetes, 07Kubernetes: Ensure all required kubectl versions are installed on deploy hosts - https://phabricator.wikimedia.org/T388388#10633960 (10JMeybohm) I think what happens is that in some cases 'apt-get update' does not run befor... [17:00:57] hey folks, added some thoughts about the memory leak in kartotherian - https://phabricator.wikimedia.org/T386926#10633932 [17:01:32] TL;DR is that node-mapnik, that we use to render the images from tiles, allocates memory via C++ objects that nodejs doesn't recall [17:02:26] the immediate plan is to let k8s to restart pods that OOM (sigh, even if it takes days) and the medium/long term plan is to add "pools" of mapnik.Image objects to the kartotherian code [17:02:32] (But it will take a bit) [17:02:43] lemme know if you like or not the idea, I am open to something different [17:14:10] 06serviceops, 07Datacenter-Switchover, 07User-notice: MoveComms support for March 2025 Datacentre switchover - https://phabricator.wikimedia.org/T387444#10633996 (10hnowlan) >>! In T387444#10633452, @Trizek-WMF wrote: > @hnowlan, I let you announce the read-only time to all staff (see task description)? Don... [17:14:23] 06serviceops, 07Datacenter-Switchover, 07User-notice: MoveComms support for March 2025 Datacentre switchover - https://phabricator.wikimedia.org/T387444#10633998 (10hnowlan) [17:53:32] 06serviceops: Update api-gateway ratelimit version - https://phabricator.wikimedia.org/T388804#10634167 (10Kappakayala) a:03Jasmine [19:57:55] 06serviceops: php-wmerrors rsyslog rule selects on php7 only - https://phabricator.wikimedia.org/T388799#10634564 (10Scott_French) 05Open→03In progress p:05Triage→03Low Alright, the rsyslog config will how match on both the 'php7.' and 'php8.' prefix, and I've added a TODO to consider whether that can be... [22:13:14] 06serviceops: Update api-gateway ratelimit version - https://phabricator.wikimedia.org/T388804#10634949 (10jasmine_) a:05Jasmine→03jasmine_ [23:14:54] 06serviceops, 13Patch-For-Review: MediaWiki on PHP 8.1 production traffic ramp-up - https://phabricator.wikimedia.org/T383845#10635101 (10Scott_French)