[01:39:37] 06serviceops, 10MW-on-K8s, 13Patch-For-Review: Support passing env variables to maintenance scripts in mwscript-k8s - https://phabricator.wikimedia.org/T380925#10814604 (10RLazarus) 05Open→03Resolved [09:09:19] 06serviceops, 13Patch-For-Review: Migrate etcd::tlsproxy Nginx certs and etcd itself to PKI - https://phabricator.wikimedia.org/T352245#10815389 (10elukey) @Scott_French ack! If you need any review/help/etc.. I'll be available :) Quick question to be sure - are the various clients like confd/pybal/liberica..... [09:23:06] 06serviceops, 13Patch-For-Review: Migrate etcd::tlsproxy Nginx certs and etcd itself to PKI - https://phabricator.wikimedia.org/T352245#10815424 (10Vgutierrez) liberica uses go.etcd.io/etcd that uses stdlib http client to perform HTTP connections against the etcd endpoints so TLS certificates are currently bei... [09:30:50] 06serviceops, 13Patch-For-Review: Migrate etcd::tlsproxy Nginx certs and etcd itself to PKI - https://phabricator.wikimedia.org/T352245#10815436 (10elukey) >>! In T352245#10815424, @Vgutierrez wrote: > liberica uses go.etcd.io/etcd that uses stdlib http client to perform HTTP connections against the etcd endpo... [09:34:08] 06serviceops, 13Patch-For-Review: Migrate etcd::tlsproxy Nginx certs and etcd itself to PKI - https://phabricator.wikimedia.org/T352245#10815441 (10Vgutierrez) >>! In T352245#10815436, @elukey wrote: >>>! In T352245#10815424, @Vgutierrez wrote: >> liberica uses go.etcd.io/etcd that uses stdlib http client to p... [09:36:01] 06serviceops, 13Patch-For-Review: Migrate etcd::tlsproxy Nginx certs and etcd itself to PKI - https://phabricator.wikimedia.org/T352245#10815444 (10elukey) >>! In T352245#10815441, @Vgutierrez wrote: >>>! In T352245#10815436, @elukey wrote: >>>>! In T352245#10815424, @Vgutierrez wrote: >>> liberica uses go.etc... [10:14:20] 06serviceops, 06Content-Transform-Team: Bump memory of testreduce1002 - https://phabricator.wikimedia.org/T393904#10815553 (10Clement_Goubert) p:05Triage→03High It would seem you're right about memory pressure being an issue: {F59932991} The biggest RAM consumer is by far MariaDB with spikes up to almost 7... [10:36:20] 06serviceops, 10RESTBase, 10RESTBase Sunsetting, 06Traffic, and 2 others: Block external traffic to RESTBase /page/data-parsoid endpoint and investigate internal usage - https://phabricator.wikimedia.org/T393557#10815665 (10MSantos) [10:39:50] 06serviceops, 13Patch-For-Review: sre.discovery cookbooks: refactor use of resolve_with_client_ip - https://phabricator.wikimedia.org/T393600#10815676 (10JMeybohm) a:03JMeybohm [11:03:20] 06serviceops, 10MW-on-K8s: Functional replacement for importImages.php on Kubernetes - https://phabricator.wikimedia.org/T377497#10815766 (10MatthewVernon) [A brief aside: data-persistence sometimes need to use this script for restoring images we had to fish out of backups (or one of the ms clusters if an imag... [11:23:07] 06serviceops, 06Growth-Team, 10GrowthExperiments, 10MW-on-K8s, 13Patch-For-Review: Migrate GrowthExperiments maintenance jobs to mw-cron - https://phabricator.wikimedia.org/T385782#10815802 (10Michael) Mh, something might have gone awry when migrating that `listTaskCounts` maintenance script. I'm noticin... [11:35:17] 06serviceops, 10MW-on-K8s: Create mw-cron dashboards - https://phabricator.wikimedia.org/T393680#10815833 (10jijiki) 05Open→03Resolved p:05Triage→03Low [12:10:33] 06serviceops, 06collaboration-services, 06Data-Platform-SRE, 10Prod-Kubernetes, 07Kubernetes: Fix alternatives entries in helm and kubernetes-client packages - https://phabricator.wikimedia.org/T387548#10815983 (10Jelto) This is fixed for helm. kubernetes-client needs an updated alternative entry (for th... [12:14:29] 06serviceops, 10Data-Engineering-Roadmap, 06Data-Platform-SRE, 10Dumps-Generation, and 4 others: WE 5.4 KR - Hypothesis 5.4.4 - Q3 FY24/25 - Migrate current-generation dumps to run on kubernetes - https://phabricator.wikimedia.org/T352650#10816009 (10BTullis) [13:06:48] 06serviceops, 13Patch-For-Review: Migrate etcd::tlsproxy Nginx certs and etcd itself to PKI - https://phabricator.wikimedia.org/T352245#10816239 (10Vgutierrez) >>! In T352245#10815444, @elukey wrote: >>>! In T352245#10815441, @Vgutierrez wrote: >>>>! In T352245#10815436, @elukey wrote: >>>>>! In T352245#108154... [13:20:16] 06serviceops, 13Patch-For-Review: Migrate etcd::tlsproxy Nginx certs and etcd itself to PKI - https://phabricator.wikimedia.org/T352245#10816311 (10elukey) Thanks a lot for checking, sigh. I guess that we should only check confd now (if Scott hasn't already done it). [13:50:01] hnowlan, Raine o/ - if you have a min later on https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1144460/ [13:50:43] elukey: oops, will give it a look now [13:52:23] oops, sorry and ty [13:52:37] thanks! [13:53:36] lgtm! [13:53:39] hello, I noticed that mc-misc2001 seems to be unreachable but I don't see phab tasks related to it. Is that known? [14:07:00] 06serviceops, 06Trust and Safety Product Team, 13Patch-For-Review: Migrate trust_and_safety_product_team jobs to mw-cron - https://phabricator.wikimedia.org/T388542#10816540 (10kamila) [14:27:27] 06serviceops, 10Page Content Service, 10Content-Transform-Team (Work In Progress), 13Patch-For-Review: Rollout more wikis: week 4 - https://phabricator.wikimedia.org/T393591#10816714 (10Jgiannelos) [14:56:35] 06serviceops, 10MW-on-K8s, 13Patch-For-Review: Implement periodic maintenance scripts for mw-on-k8s - https://phabricator.wikimedia.org/T341555#10816893 (10Clement_Goubert) >>! In T341555#10760093, @A_smart_kitten wrote: > Thanks for the work that people are doing on this! Hi, thanks for the feedback. > I... [15:05:08] 06serviceops, 13Patch-For-Review: Migrate etcd::tlsproxy Nginx certs and etcd itself to PKI - https://phabricator.wikimedia.org/T352245#10816965 (10Scott_French) Thank you both! In short, confd uses go.etcd.io/etcd just like Liberica, and thus will pick up the WMF root PKI CA cert from `/etc/ssl/certs` without... [15:10:16] 06serviceops, 10function-orchestrator, 10Abstract Wikipedia team (25Q4 (Apr–Jun)), 07OKR-Work: Enable memcached in the orchestrator - https://phabricator.wikimedia.org/T391986#10816997 (10Jdforrester-WMF) [15:10:20] 06serviceops, 13Patch-For-Review: Implement KeyModifyRoute in mcrouter configuration - https://phabricator.wikimedia.org/T393281#10816998 (10Jdforrester-WMF) [15:45:00] 06serviceops, 06Content-Transform-Team: Bump memory of testreduce1002 - https://phabricator.wikimedia.org/T393904#10817140 (10ssastry) Anytime today or tomorrow works. We'll hold off running rt-testing till the reboot happens. [15:54:45] 06serviceops, 10MW-on-K8s, 13Patch-For-Review: Allow members of restricted to run maintenance scripts - https://phabricator.wikimedia.org/T378429#10817215 (10JMeybohm) 05Resolved→03Open I've reverted https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1127087 because it triggers a race condit... [15:56:56] 06serviceops, 06collaboration-services, 06Data-Platform-SRE, 10Prod-Kubernetes, 07Kubernetes: Check/update grafana dashboards for k8s 1.31 - https://phabricator.wikimedia.org/T389084#10817238 (10JMeybohm) 05Open→03Resolved a:03JMeybohm Easy win this time. Only one relevant change: > Replace th... [15:57:48] 06serviceops, 06Content-Transform-Team: Bump memory of testreduce1002 - https://phabricator.wikimedia.org/T393904#10817242 (10ops-monitoring-bot) VM testreduce1002.eqiad.wmnet rebooted by cgoubert@cumin1002 with reason: Pick up new 10GB ram [16:02:32] 06serviceops, 06Content-Transform-Team: Bump memory of testreduce1002 - https://phabricator.wikimedia.org/T393904#10817288 (10Clement_Goubert) ` cgoubert@testreduce1002:~$ free -m total used free shared buff/cache available Mem: 9944 2211 7646... [16:07:58] 06serviceops, 06Content-Transform-Team: Bump memory of testreduce1002 - https://phabricator.wikimedia.org/T393904#10817308 (10ssastry) Thanks! [17:45:42] 06serviceops, 10MediaWiki-Page-derived-data: Migrate MediaWiki-Page-derived-data jobs to mw-cron - https://phabricator.wikimedia.org/T388530#10817930 (10Scott_French) [17:48:15] 06serviceops, 10MediaWiki-Page-derived-data: Migrate MediaWiki-Page-derived-data jobs to mw-cron - https://phabricator.wikimedia.org/T388530#10817942 (10Scott_French) 05Open→03Resolved a:03Scott_French The remaining shards of the (renamed) job have been migrated. Given what we saw with the pilot on s... [17:53:44] 06serviceops, 10FlaggedRevs, 13Patch-For-Review: Migrate flaggedrevs jobs to mw-cron - https://phabricator.wikimedia.org/T388535#10817967 (10Scott_French) [17:55:12] 06serviceops, 10FlaggedRevs, 13Patch-For-Review: Migrate flaggedrevs jobs to mw-cron - https://phabricator.wikimedia.org/T388535#10817973 (10Scott_French) a:03Scott_French The update-flaggedrev-stats job has been migrated. I'll hold onto this until I'm able to confirm the first run is successful later today. [17:56:08] 06serviceops, 10MediaWiki-extensions-CentralAuth, 10MW-on-K8s, 10MediaWiki-Platform-Team (Radar), 13Patch-For-Review: Migrate CentralAuth maintenance jobs to mw-cron - https://phabricator.wikimedia.org/T385866#10817979 (10Scott_French) [18:03:46] 06serviceops, 10MediaWiki-extensions-CentralAuth, 10MW-on-K8s, 10MediaWiki-Platform-Team (Radar), 13Patch-For-Review: Migrate CentralAuth maintenance jobs to mw-cron - https://phabricator.wikimedia.org/T385866#10818015 (10Scott_French) Updates: * The first run of purge-temporary-accounts appears to have... [23:10:29] 06serviceops, 10MediaWiki-extensions-ReadingLists, 06MW-Interfaces-Team, 10RESTBase Sunsetting, 13Patch-For-Review: Switchover plan from RESTbase to REST Gateway for Reading Lists endpoints - https://phabricator.wikimedia.org/T384891#10818906 (10HCoplin-WMF) Awesome. @Seddon confirmed that the apps team...