[05:39:46] 06serviceops, 06collaboration-services, 06Infrastructure-Foundations, 06SRE, and 2 others: Create a cookbook to automate gerrit's switchover - https://phabricator.wikimedia.org/T260666#10851232 (10ABran-WMF) [08:01:11] 06serviceops, 10Discovery-Search (2025.05.02 - 2025.05.23), 13Patch-For-Review: Migrate discovery-search jobs to mw-cron - https://phabricator.wikimedia.org/T388538#10851395 (10Clement_Goubert) ` cgoubert@deploy1003:~$ kubectl logs cirrus-build-completion-indices-codfw-s8-29132790-wc6dx mediawiki-main-app e... [08:17:53] 06serviceops, 10Discovery-Search (2025.05.02 - 2025.05.23), 13Patch-For-Review: Migrate discovery-search jobs to mw-cron - https://phabricator.wikimedia.org/T388538#10851415 (10dcausse) @Clement_Goubert thanks! >>! In T388538#10851395, @Clement_Goubert wrote: > ` > cgoubert@deploy1003:~$ kubectl logs cirrus... [08:28:37] 06serviceops, 10Discovery-Search (2025.05.02 - 2025.05.23), 13Patch-For-Review: Migrate discovery-search jobs to mw-cron - https://phabricator.wikimedia.org/T388538#10851450 (10Clement_Goubert) >>! In T388538#10851415, @dcausse wrote: > @Clement_Goubert thanks! > >>>! In T388538#10851395, @Clement_Goubert w... [08:37:56] 06serviceops, 10Discovery-Search (2025.05.02 - 2025.05.23), 13Patch-For-Review: Migrate discovery-search jobs to mw-cron - https://phabricator.wikimedia.org/T388538#10851467 (10Clement_Goubert) By the way, here are the completion times for the different shards: ` cirrus-build-completion-indices-codfw-s1-2913... [08:47:40] 06serviceops, 10Prod-Kubernetes, 06Traffic, 07Kubernetes, 13Patch-For-Review: Handling inbound IPIP traffic on low traffic LVS k8s based realservers - https://phabricator.wikimedia.org/T352956#10851490 (10akosiaris) After having to deal a bit with a staging-eqiad calico upgrade yesterday, I did find 1 th... [09:00:17] 06serviceops: Service mesh envoy does not treat incoming connections as local - https://phabricator.wikimedia.org/T354853#10851546 (10Gehel) [09:00:18] 06serviceops, 06Discovery-Search, 07Analytics-Data-Problem, 10Data-Platform-SRE (2025.05.02 - 2025.05.23): Search Update Pipeline requests to Action API are logged as coming from 127.0.0.1 - https://phabricator.wikimedia.org/T388855#10851544 (10Gehel) →14Duplicate dup:03T354853 [09:04:33] 06serviceops, 10Discovery-Search (2025.05.02 - 2025.05.23), 13Patch-For-Review: Migrate discovery-search jobs to mw-cron - https://phabricator.wikimedia.org/T388538#10851569 (10dcausse) >>! In T388538#10851450, @Clement_Goubert wrote: > > It's when trying to run on s8, so wikidata, yes. I could also just re... [09:29:19] 06serviceops, 10Wikidata, 10Discovery-Search (2025.05.02 - 2025.05.23), 13Patch-For-Review: Migrate discovery-search jobs to mw-cron - https://phabricator.wikimedia.org/T388538#10851630 (10dcausse) pinging @hoo & #wikidata for visibility on the work on `mediawiki_job_wikidata-updateQueryServiceLag.timer` [09:36:39] 06serviceops, 10Wikidata, 10Discovery-Search (2025.05.02 - 2025.05.23), 13Patch-For-Review: Migrate discovery-search jobs to mw-cron - https://phabricator.wikimedia.org/T388538#10851661 (10Clement_Goubert) >>! In T388538#10851630, @dcausse wrote: > pinging @hoo & #wikidata for visibility on the work on `me... [09:40:47] 06serviceops, 10Wikidata, 10Discovery-Search (2025.05.02 - 2025.05.23), 13Patch-For-Review: Migrate discovery-search jobs to mw-cron - https://phabricator.wikimedia.org/T388538#10851664 (10dcausse) >>! In T388538#10851661, @Clement_Goubert wrote: >>>! In T388538#10851630, @dcausse wrote: >> pinging @hoo &... [10:10:32] 06serviceops, 10Wikidata, 10Discovery-Search (2025.05.02 - 2025.05.23), 13Patch-For-Review: Migrate discovery-search jobs to mw-cron - https://phabricator.wikimedia.org/T388538#10851732 (10Clement_Goubert) `wikidata-updatequeryservicelag` looks to be working fine ` cgoubert@deploy1003:/srv/deployment-char... [10:10:45] 06serviceops, 10Wikidata, 10Discovery-Search (2025.05.02 - 2025.05.23), 13Patch-For-Review: Migrate discovery-search jobs to mw-cron - https://phabricator.wikimedia.org/T388538#10851733 (10Clement_Goubert) [10:11:53] 06serviceops, 10MediaWiki-extensions-ReadingLists, 06MW-Interfaces-Team, 10RESTBase Sunsetting, 13Patch-For-Review: Switchover plan from RESTbase to REST Gateway for Reading Lists endpoints - https://phabricator.wikimedia.org/T384891#10851739 (10hnowlan) Changes are live for moving the API to something t... [10:28:49] 06serviceops, 06Trust and Safety Product Team, 13Patch-For-Review: Migrate trust_and_safety_product_team jobs to mw-cron - https://phabricator.wikimedia.org/T388542#10851761 (10Clement_Goubert) @kostajh @Dreamy_Jazz The `purge_securepoll` job failed due to running on all wikis instead of only the ones th... [10:39:41] 06serviceops, 06Language and Product Localization: Migrate language_and_product_localization jobs to mw-cron - https://phabricator.wikimedia.org/T388539#10851803 (10hnowlan) 05In progress→03Resolved [10:48:59] 06serviceops, 06Trust and Safety Product Team, 13Patch-For-Review: Migrate trust_and_safety_product_team jobs to mw-cron - https://phabricator.wikimedia.org/T388542#10851824 (10Clement_Goubert) Manual run seems to have worked correctly. [10:53:26] 06serviceops, 06Security-Team, 07SecTeam-Processed: Migrate Security-Team jobs to mw-cron - https://phabricator.wikimedia.org/T388531#10851842 (10Clement_Goubert) There may be a third way of working around this, possibly quicker than both the above options, which would be to either add functionality to `Gene... [10:57:03] 06serviceops, 06collaboration-services, 06Data-Platform-SRE, 10Prod-Kubernetes, and 2 others: Update Kubernetes clusters to 1.31 - https://phabricator.wikimedia.org/T341984#10851847 (10ops-monitoring-bot) depool host kubestage2004.codfw.wmnet by jayme@cumin1002 with reason: test cookbook [10:57:51] 06serviceops, 06collaboration-services, 06Data-Platform-SRE, 10Prod-Kubernetes, and 2 others: Update Kubernetes clusters to 1.31 - https://phabricator.wikimedia.org/T341984#10851848 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by jayme@cumin1002 depool for host kubestage2004.c... [11:04:01] 06serviceops, 06collaboration-services, 06Data-Platform-SRE, 10Prod-Kubernetes, and 2 others: Update Kubernetes clusters to 1.31 - https://phabricator.wikimedia.org/T341984#10851855 (10ops-monitoring-bot) pool host kubestage2004.codfw.wmnet by jayme@cumin1002 with reason: test cookbook [11:04:04] 06serviceops, 06collaboration-services, 06Data-Platform-SRE, 10Prod-Kubernetes, and 2 others: Update Kubernetes clusters to 1.31 - https://phabricator.wikimedia.org/T341984#10851856 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by jayme@cumin1002 pool for host kubestage2004.cod... [11:06:43] 06serviceops, 06collaboration-services, 06Data-Platform-SRE, 10Prod-Kubernetes, and 2 others: Update Kubernetes clusters to 1.31 - https://phabricator.wikimedia.org/T341984#10851858 (10ops-monitoring-bot) depool host kubestage2004.codfw.wmnet by jayme@cumin1002 with reason: test cookbook [11:06:55] 06serviceops, 06collaboration-services, 06Data-Platform-SRE, 10Prod-Kubernetes, and 2 others: Update Kubernetes clusters to 1.31 - https://phabricator.wikimedia.org/T341984#10851859 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by jayme@cumin1002 depool for host kubestage2004.c... [11:07:27] 06serviceops, 06collaboration-services, 06Data-Platform-SRE, 10Prod-Kubernetes, and 2 others: Update Kubernetes clusters to 1.31 - https://phabricator.wikimedia.org/T341984#10851860 (10ops-monitoring-bot) pool host kubestage2004.codfw.wmnet by jayme@cumin1002 with reason: test cookbook [11:07:28] 06serviceops, 06collaboration-services, 06Data-Platform-SRE, 10Prod-Kubernetes, and 2 others: Update Kubernetes clusters to 1.31 - https://phabricator.wikimedia.org/T341984#10851861 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.pool-depool-node started by jayme@cumin1002 pool for host kubestage2004.cod... [12:13:27] 06serviceops, 07Analytics-Data-Problem, 10Data-Platform-SRE (2025.05.02 - 2025.05.23), 10Discovery-Search (2025.05.02 - 2025.05.23): Search Update Pipeline requests to Action API are logged as coming from 127.0.0.1 - https://phabricator.wikimedia.org/T388855#10852059 (10Gehel) [13:01:29] 06serviceops, 10Wikidata, 10Discovery-Search (2025.05.24 - 2025.06.13), 13Patch-For-Review: Migrate discovery-search jobs to mw-cron - https://phabricator.wikimedia.org/T388538#10852208 (10Gehel) [13:59:34] 06serviceops, 10Prod-Kubernetes, 06Traffic, 07Kubernetes, 13Patch-For-Review: Handling inbound IPIP traffic on low traffic LVS k8s based realservers - https://phabricator.wikimedia.org/T352956#10852605 (10cmooney) Hmm ok. Yes the big problem with pmtud is that it relies on ICMP. This is why it regularl... [14:05:57] 06serviceops, 10envoy, 10Observability-Metrics, 13Patch-For-Review, 10SRE Observability (FY2024/2025-Q4): Revisit default envoy histogram buckets - https://phabricator.wikimedia.org/T391333#10852614 (10akosiaris) I 've spent a good deal of time today to do what I assumed to be easy, that is perform t... [14:55:04] 06serviceops, 10ChangeProp: Exclude 404s from changeprop/changeprop-jobqueue logs - https://phabricator.wikimedia.org/T395132 (10hnowlan) 03NEW [15:42:22] 06serviceops, 10Ceph, 06Data-Persistence, 10SRE-swift-storage: Onboard the Docker Registry to apus - https://phabricator.wikimedia.org/T394476#10852861 (10MatthewVernon) A few TB of quota shouldn't be a problem; how many objects per bucket are you looking at? We get better performance out of fewer larger o... [16:38:25] 06serviceops, 06Trust and Safety Product Team: Migrate trust_and_safety_product_team jobs to mw-cron - https://phabricator.wikimedia.org/T388542#10853042 (10Dreamy_Jazz) >>! In T388542#10851761, @Clement_Goubert wrote: > @kostajh @Dreamy_Jazz The `purge_securepoll` job failed due to running on all wikis in... [16:44:37] 06serviceops, 06Trust and Safety Product Team: Migrate trust_and_safety_product_team jobs to mw-cron - https://phabricator.wikimedia.org/T388542#10853126 (10Dreamy_Jazz) I've left a comment on https://gerrit.wikimedia.org/r/c/operations/puppet/+/1149629 to indicate that it might be the wrong db list. It se... [16:45:56] 06serviceops, 10MW-on-K8s, 06Trust and Safety Product Team, 10MediaModeration (MediaModeration 2.1): Migrate MediaModeration jobs to mw-cron - https://phabricator.wikimedia.org/T385799#10853129 (10Dreamy_Jazz) Thanks for doing the migration! [18:07:25] 06serviceops, 10envoy, 10Observability-Metrics, 13Patch-For-Review, 10SRE Observability (FY2024/2025-Q4): Revisit default envoy histogram buckets - https://phabricator.wikimedia.org/T391333#10853430 (10elukey) >>! In T391333#10852614, @akosiaris wrote: > I 've spent a good deal of time today to do wh...