[08:32:15] 06serviceops, 06MediaWiki-Platform-Team: Migrate "startupregistrystats" maintenance script to k8s-mw-cron (mediawiki-platform-team) - https://phabricator.wikimedia.org/T388540#10666626 (10Krinkle) [08:42:40] 06serviceops, 06MediaWiki-Platform-Team: Migrate "startupregistrystats" maintenance script to k8s-mw-cron (mediawiki-platform-team) - https://phabricator.wikimedia.org/T388540#10666646 (10Krinkle) a:05Krinkle→03None > * jobs that should be watched more: Yes, the main one of the three. Details below. > *... [10:45:28] 06serviceops, 10MW-on-K8s, 06Release-Engineering-Team: Refactor scap's kubernetes DeploymentsConfig to support selection of image kinds - https://phabricator.wikimedia.org/T389499#10667103 (10BTullis) I'd just like to pick up on something that was mentioned in the description: > cli (used by mw-cron and mw-s... [11:10:03] 06serviceops, 10Shellbox, 10SyntaxHighlight, 13Patch-For-Review, 07Wikimedia-production-error: Shellbox bubbles GuzzleHttp\Exception\ConnectException when it should probably wrap it in a ShellboxError? - https://phabricator.wikimedia.org/T374117#10667244 (10hashar) Shellbox now documents it throws `Psr\H... [11:56:10] 06serviceops, 10Deployments, 10Shellbox, 10Wikibase-Quality-Constraints, and 5 others: Burst of GuzzleHttp Exception for http://localhost:6025/call/constraint-regex-checker - https://phabricator.wikimedia.org/T371633#10667501 (10ArthurTaylor) [11:56:18] 06serviceops, 10Deployments, 10Shellbox, 10Wikibase-Quality-Constraints, and 5 others: Burst of GuzzleHttp Exception for http://localhost:6025/call/constraint-regex-checker - https://phabricator.wikimedia.org/T371633#10667503 (10ArthurTaylor) a:03ArthurTaylor [12:31:28] 06serviceops, 10MW-on-K8s, 06SRE, 06Traffic, 10Release-Engineering-Team (Seen): Deploy mediawiki kubernetes services - https://phabricator.wikimedia.org/T321786#10667658 (10Clement_Goubert) 05Open→03In progress [12:43:28] 06serviceops, 06Trust and Safety Product Team: Migrate trust_and_safety_product_team jobs to mw-cron - https://phabricator.wikimedia.org/T388542#10667714 (10Clement_Goubert) I was looking at setting up {T385709}, and realized you've already been onboarded to alertmanager and critical alerts for your team are s... [12:43:34] 06serviceops, 06Growth-Team, 10GrowthExperiments, 10MW-on-K8s: Migrate GrowthExperiments maintenance jobs to mw-cron - https://phabricator.wikimedia.org/T385782#10667718 (10Clement_Goubert) I was looking at setting up {T385709}, and realized you've already been onboarded to alertmanager and critical alerts... [12:49:23] 06serviceops, 06MediaWiki-Platform-Team: Migrate "startupregistrystats" maintenance script to k8s-mw-cron (mediawiki-platform-team) - https://phabricator.wikimedia.org/T388540#10667732 (10Clement_Goubert) I was looking at setting up {T385709}, and realized you've already been onboarded to alertmanager and aler... [13:04:34] 06serviceops, 10Deployments, 10Shellbox, 10Wikibase-Quality-Constraints, and 5 others: Burst of GuzzleHttp Exception for http://localhost:6025/call/constraint-regex-checker - https://phabricator.wikimedia.org/T371633#10667800 (10ArthurTaylor) a:05ArthurTaylor→03None [13:18:32] 06serviceops, 06Trust and Safety Product Team: Migrate trust_and_safety_product_team jobs to mw-cron - https://phabricator.wikimedia.org/T388542#10667863 (10Dreamy_Jazz) >>! In T388542#10667714, @Clement_Goubert wrote: > I was looking at setting up {T385709}, and realized you've already been onboarded to alert... [13:47:33] 06serviceops, 06Wikimedia Enterprise, 07affects-Kiwix-and-openZIM, 07Wikimedia-Performance-recommendation: large amount of traffic to the action=parse API from MWOffliner - https://phabricator.wikimedia.org/T324866#10668072 (10Benoit74) For the record, we are in the process of implementing @cscott suggesti... [13:48:27] 06serviceops, 06Wikimedia Enterprise, 07affects-Kiwix-and-openZIM, 07Wikimedia-Performance-recommendation: large amount of traffic to the action=parse API from MWOffliner - https://phabricator.wikimedia.org/T324866#10668097 (10Benoit74) We are also still discussing with Wikimedia Enterprise, but next steps... [14:43:32] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q3:rack/setup/install wikikube-worker2248-2331, wikikube-ctrl2004-2005 - https://phabricator.wikimedia.org/T384970#10668543 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host wikikube-worker2254.codfw.wmnet with... [14:57:37] 06serviceops, 10Page Content Service, 10RESTBase Sunsetting, 10Content-Transform-Team (Work In Progress), and 2 others: Pregeneration rules don't pregenerate caches for the same cases restbase did - https://phabricator.wikimedia.org/T388214#10668697 (10MSantos) [14:57:43] 06serviceops, 10Page Content Service, 10RESTBase Sunsetting, 10Content-Transform-Team (Work In Progress), 07Essential-Work: Change changeprops rules to pre-generate/invalidate cache directly to PCS rather than in restbase - https://phabricator.wikimedia.org/T348996#10668698 (10MSantos) [15:10:50] 06serviceops, 10Wikidata, 06Wikidata Dev Team, 10Wikidata Integration in Wikimedia projects: Migrate wikidata_dev_team jobs to mw-cron - https://phabricator.wikimedia.org/T388543#10668796 (10Clement_Goubert) I was looking at setting up {T385709}, and realized you've already been onboarded to alertmanager a... [15:15:10] 06serviceops, 10Image-Suggestions, 10Structured Data Engineering, 06Structured-Data-Backlog: Migrate data-engineering jobs to mw-cron - https://phabricator.wikimedia.org/T388537#10668836 (10Clement_Goubert) I was looking at setting up {T385709}, and realized you've already been onboarded to alertmanager an... [15:27:01] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q3:rack/setup/install wikikube-worker2248-2331, wikikube-ctrl2004-2005 - https://phabricator.wikimedia.org/T384970#10668925 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host wikikube-ctrl2005.codfw.wmnet with O... [15:39:57] 06serviceops, 10Excimer, 10MediaWiki-Platform-Team (Radar), 13Patch-For-Review: Update Excimer to 1.2.3 in production - https://phabricator.wikimedia.org/T389243#10669073 (10pmiazga) [15:58:24] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q3:rack/setup/install wikikube-worker2248-2331, wikikube-ctrl2004-2005 - https://phabricator.wikimedia.org/T384970#10669175 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host wikikube-ctrl2005.codfw.wmnet with OS bo... [16:03:52] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q3:rack/setup/install wikikube-worker2248-2331, wikikube-ctrl2004-2005 - https://phabricator.wikimedia.org/T384970#10669258 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host wikikube-worker2254.codfw.wmnet with... [16:03:54] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q3:rack/setup/install wikikube-worker2248-2331, wikikube-ctrl2004-2005 - https://phabricator.wikimedia.org/T384970#10669259 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host wikikube-worker2255.codfw.wmnet with... [16:03:55] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q3:rack/setup/install wikikube-worker2248-2331, wikikube-ctrl2004-2005 - https://phabricator.wikimedia.org/T384970#10669260 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host wikikube-worker2256.codfw.wmnet with... [16:10:53] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q3:rack/setup/install wikikube-worker2248-2331, wikikube-ctrl2004-2005 - https://phabricator.wikimedia.org/T384970#10669309 (10Jhancock.wm) [16:26:54] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q3:rack/setup/install wikikube-worker2248-2331, wikikube-ctrl2004-2005 - https://phabricator.wikimedia.org/T384970#10669510 (10Jhancock.wm) [16:38:41] Hey! I am trying to debug something on PCS. Can somebody help me figure out when changeprop helm chart with version 0.16.4 and 0.16.5 where deployed on k8s ? (i don't think i have the rights to see history) [16:41:11] nemo-yiannis: changeprop or changeprop-jobqueue? [16:41:18] changeprop [16:42:13] 42 Mon Mar 10 16:49:54 2025 superseded changeprop-0.16.3 Upgrade complete [16:42:15] 43 Mon Mar 10 17:48:53 2025 superseded changeprop-0.16.4 Upgrade complete [16:42:17] 44 Mon Mar 17 12:52:38 2025 deployed changeprop-0.16.5 Upgrade complete [16:42:49] 06serviceops, 06Discovery-Search, 06SRE, 10Wikimedia-Apache-configuration, and 3 others: www.wikipedia.org: prefilling the search box with the "search" URL parameter does not work - https://phabricator.wikimedia.org/T318285#10669644 (10Gehel) [16:43:20] 06serviceops, 06Discovery-Search, 06SRE, 10Wikimedia-Apache-configuration, and 3 others: www.wikipedia.org: prefilling the search box with the "search" URL parameter does not work - https://phabricator.wikimedia.org/T318285#10669647 (10Gehel) We're not expecting the Search Platform team to do any work on t... [16:43:58] thanks claime [16:45:30] 06serviceops, 06Discovery-Search, 06SRE, 10Wikimedia-Apache-configuration, and 4 others: www.wikipedia.org: prefilling the search box with the "search" URL parameter does not work - https://phabricator.wikimedia.org/T318285#10669663 (10simon04) Let's try to finish this on the #wikimedia-hackathon-2025 then... [16:52:13] 06serviceops, 10Discovery-Search (2025.03.22 - 2025.04.11): Migrate discovery-search jobs to mw-cron - https://phabricator.wikimedia.org/T388538#10669786 (10Gehel) [16:53:12] 06serviceops, 06Discovery-Search, 06SRE, 10Wikimedia-Apache-configuration, and 4 others: www.wikipedia.org: prefilling the search box with the "search" URL parameter does not work - https://phabricator.wikimedia.org/T318285#10669816 (10Clement_Goubert) >>! In T318285#10669663, @simon04 wrote: > Let's try t... [16:54:32] 06serviceops, 06Discovery-Search, 06SRE, 10Wikimedia-Apache-configuration, and 3 others: www.wikipedia.org: prefilling the search box with the "search" URL parameter does not work - https://phabricator.wikimedia.org/T318285#10669822 (10simon04) Even better, thank you! Sorry for the noise! [17:02:29] 06serviceops, 06Discovery-Search, 06SRE, 10Wikimedia-Apache-configuration, and 3 others: www.wikipedia.org: prefilling the search box with the "search" URL parameter does not work - https://phabricator.wikimedia.org/T318285#10669902 (10Clement_Goubert) We may do it at the end of that window or right after... [17:35:09] 06serviceops, 10Excimer, 10MediaWiki-Platform-Team (Radar): Update Excimer to 1.2.3 in production - https://phabricator.wikimedia.org/T389243#10670180 (10Scott_French) Alright, the new excimer 1.2.3 package should now be live. [18:01:51] 06serviceops, 06Growth-Team, 10GrowthExperiments, 10MW-on-K8s: Migrate GrowthExperiments maintenance jobs to mw-cron - https://phabricator.wikimedia.org/T385782#10670353 (10Michael) >>! In T385782#10667718, @Clement_Goubert wrote: > I was looking at setting up {T385709}, and realized you've already been on... [18:14:26] 06serviceops, 06Growth-Team, 10GrowthExperiments, 10MW-on-K8s: Migrate GrowthExperiments maintenance jobs to mw-cron - https://phabricator.wikimedia.org/T385782#10670401 (10Clement_Goubert) What I'm envisioning is alerting when the kubernetes `Job` fails, i.e. the process exits with a code other than `0`.... [18:22:27] 06serviceops, 10MW-on-K8s, 06Release-Engineering-Team, 13Patch-For-Review: Refactor scap's kubernetes DeploymentsConfig to support selection of image kinds - https://phabricator.wikimedia.org/T389499#10670449 (10BTullis) As mentioned, this is our effort to integrate scap with our cluster/namespace {T389786} [19:26:35] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q3:rack/setup/install wikikube-worker2248-2331, wikikube-ctrl2004-2005 - https://phabricator.wikimedia.org/T384970#10671069 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host wikikube-worker2254.codfw.wmnet with... [19:31:41] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q3:rack/setup/install wikikube-worker2248-2331, wikikube-ctrl2004-2005 - https://phabricator.wikimedia.org/T384970#10671094 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host wikikube-worker2255.codfw.wmnet with... [19:33:54] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q3:rack/setup/install wikikube-worker2248-2331, wikikube-ctrl2004-2005 - https://phabricator.wikimedia.org/T384970#10671098 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host wikikube-worker2255.codfw.wmnet with OS... [19:39:49] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q3:rack/setup/install wikikube-worker2248-2331, wikikube-ctrl2004-2005 - https://phabricator.wikimedia.org/T384970#10671130 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host wikikube-worker2256.codfw.wmnet with... [19:46:16] 06serviceops, 10Citoid, 06Editing-team, 10RESTBase Sunsetting, and 2 others: Switchover plan from restbase to api gateway for Citoid - https://phabricator.wikimedia.org/T361576#10671140 (10Mvolz) Should we resolve this? Or is there more work to do? [19:50:50] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q3:rack/setup/install wikikube-worker2248-2331, wikikube-ctrl2004-2005 - https://phabricator.wikimedia.org/T384970#10671174 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host wikikube-worker2257.codfw.wmnet with... [19:57:21] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q3:rack/setup/install wikikube-worker2248-2331, wikikube-ctrl2004-2005 - https://phabricator.wikimedia.org/T384970#10671186 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host wikikube-worker2254.codfw.wmnet with OS... [19:59:28] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q3:rack/setup/install wikikube-worker2248-2331, wikikube-ctrl2004-2005 - https://phabricator.wikimedia.org/T384970#10671194 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host wikikube-worker2258.codfw.wmnet with... [20:17:42] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q3:rack/setup/install wikikube-worker2248-2331, wikikube-ctrl2004-2005 - https://phabricator.wikimedia.org/T384970#10671280 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host wikikube-worker2256.codfw.wmnet with OS... [20:21:19] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q3:rack/setup/install wikikube-worker2248-2331, wikikube-ctrl2004-2005 - https://phabricator.wikimedia.org/T384970#10671293 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host wikikube-worker2257.codfw.wmnet with OS... [20:36:05] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q3:rack/setup/install wikikube-worker2248-2331, wikikube-ctrl2004-2005 - https://phabricator.wikimedia.org/T384970#10671358 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host wikikube-worker2258.codfw.wmnet with OS... [20:36:26] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q3:rack/setup/install wikikube-worker2248-2331, wikikube-ctrl2004-2005 - https://phabricator.wikimedia.org/T384970#10671359 (10Jhancock.wm) [20:40:32] 06serviceops, 07Wikimedia-production-error: Error for wmf.14 (while wmf.21 should be live) Uncaught MediaWiki\Config\ConfigException: Failed to load configuration from etcd: (curl error: 1) Unsupported protocol in /srv/mediawiki/php-1.44.0-wmf.14/includes/config/EtcdCo... - https://phabricator.wikimedia.org/T389877 [20:55:11] 06serviceops, 10Excimer, 10MediaWiki-Platform-Team (Radar): Update Excimer to 1.2.3 in production - https://phabricator.wikimedia.org/T389243#10671426 (10Scott_French) 05Open→03Resolved Spot checking https://performance.wikimedia.org/arclamp/svgs/daily/2025-03-24.excimer-k8s-php8.load.svgz after a co... [22:59:38] 06serviceops, 10MW-on-K8s, 06Release-Engineering-Team, 13Patch-For-Review: Refactor scap's kubernetes DeploymentsConfig to support selection of image kinds - https://phabricator.wikimedia.org/T389499#10671841 (10Scott_French) Thanks, @BTullis - I was aware of Joe's recent scap change to support `cluster_di... [23:00:45] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q3:rack/setup/install wikikube-worker2248-2331, wikikube-ctrl2004-2005 - https://phabricator.wikimedia.org/T384970#10671844 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host wikikube-worker2259.codfw.wmnet with... [23:00:53] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q3:rack/setup/install wikikube-worker2248-2331, wikikube-ctrl2004-2005 - https://phabricator.wikimedia.org/T384970#10671845 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host wikikube-worker2260.codfw.wmnet with... [23:01:06] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q3:rack/setup/install wikikube-worker2248-2331, wikikube-ctrl2004-2005 - https://phabricator.wikimedia.org/T384970#10671846 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host wikikube-worker2261.codfw.wmnet with... [23:38:19] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q3:rack/setup/install wikikube-worker2248-2331, wikikube-ctrl2004-2005 - https://phabricator.wikimedia.org/T384970#10671944 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host wikikube-worker2261.codfw.wmnet with OS... [23:38:21] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q3:rack/setup/install wikikube-worker2248-2331, wikikube-ctrl2004-2005 - https://phabricator.wikimedia.org/T384970#10671945 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host wikikube-worker2260.codfw.wmnet with OS... [23:41:15] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q3:rack/setup/install wikikube-worker2248-2331, wikikube-ctrl2004-2005 - https://phabricator.wikimedia.org/T384970#10671947 (10Jhancock.wm)