[09:23:36] <wikibugs>	 06serviceops, 06Growth-Team, 10MW-on-K8s, 10Notifications (Echo): Migrate Notifications (Echo) periodic jobs - https://phabricator.wikimedia.org/T394471#10838161 (10Clement_Goubert) 05Open→03Resolved ` echo-mail-batch-29128320                                   1/1           7m16s      9h `  echo-ma...
[09:38:14] <wikibugs>	 06serviceops, 10MediaWiki-extensions-ReadingLists, 06MW-Interfaces-Team, 10RESTBase Sunsetting: Switchover plan from RESTbase to REST Gateway for Reading Lists endpoints - https://phabricator.wikimedia.org/T384891#10838203 (10hnowlan) Internal testing can now be run against the REST gateway using URLs of t...
[10:12:50] <wikibugs>	 06serviceops, 06DBA, 10Editing-team (Tracking), 10MW-1.45-notes (1.45.0-wmf.1; 2025-05-13), and 2 others: Fatal exception of type "Wikimedia\Rdbms\DBUnexpectedError: Database servers in extension1 are overloaded. In order to protect application servers, t... - https://phabricator.wikimedia.org/T393513#10838276
[10:39:00] <wikibugs>	 06serviceops, 10MW-on-K8s: Convert characterEditStatsTranslate to kubernetes CronJob - https://phabricator.wikimedia.org/T380170#10838381 (10Clement_Goubert) 05Open→03Invalid Deleted in {T388539}
[10:39:27] <wikibugs>	 06serviceops, 10MW-on-K8s, 13Patch-For-Review: Identify low-criticality maintenance job to move to mwcron - https://phabricator.wikimedia.org/T377963#10838388 (10Clement_Goubert) 05Open→03Resolved
[11:15:54] <wikibugs>	 06serviceops, 06Trust and Safety Product Team: Migrate trust_and_safety_product_team jobs to mw-cron - https://phabricator.wikimedia.org/T388542#10838533 (10kamila)
[12:56:41] <wikibugs>	 06serviceops, 10Prod-Kubernetes, 06Traffic, 07Kubernetes, 13Patch-For-Review: Handling inbound IPIP traffic on low traffic LVS k8s based realservers - https://phabricator.wikimedia.org/T352956#10839001 (10akosiaris) Unfortunately the 2 patches above didn't work. For ml-staging-codfw, just because it's st...
[13:29:39] <wikibugs>	 06serviceops, 10MW-on-K8s, 06SRE, 07Python3-Porting: mwgrep cannot be used from a deployment host - https://phabricator.wikimedia.org/T384764#10839219 (10Jdforrester-WMF) >>! In T384764#10629639, @gerritbot wrote: > Change #1127075 had a related patch set uploaded (by Reedy; author: Reedy): > %%%[operation...
[13:30:08] <wikibugs>	 06serviceops, 10MW-on-K8s, 06SRE, 07Python3-Porting: mwgrep cannot be used from a deployment host - https://phabricator.wikimedia.org/T384764#10839226 (10Reedy) 05Open→03Resolved a:03Reedy Nope, all good.
[13:59:13] <wikibugs>	 06serviceops, 10Prod-Kubernetes, 06Traffic, 07Kubernetes, 13Patch-For-Review: Handling inbound IPIP traffic on low traffic LVS k8s based realservers - https://phabricator.wikimedia.org/T352956#10839348 (10akosiaris) `lang=bash nobody@wmfdebug:/$ ip link 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noque...
[15:18:33] <wikibugs>	 06serviceops, 10Discovery-Search (2025.05.02 - 2025.05.23): Migrate discovery-search jobs to mw-cron - https://phabricator.wikimedia.org/T388538#10839708 (10Clement_Goubert) 05Open→03In progress
[15:25:14] <wikibugs>	 06serviceops, 10Discovery-Search (2025.05.02 - 2025.05.23), 13Patch-For-Review: Migrate discovery-search jobs to mw-cron - https://phabricator.wikimedia.org/T388538#10839770 (10Clement_Goubert) >>! In T388538#10695049, @EBernhardson wrote: > I ran a test invocation to see how it would work and it seems to ha...
[15:26:33] <wikibugs>	 06serviceops, 06Growth-Team, 10GrowthExperiments, 10MW-on-K8s, 13Patch-For-Review: Migrate GrowthExperiments maintenance jobs to mw-cron - https://phabricator.wikimedia.org/T385782#10839778 (10hnowlan) Now that we've largely unblocked the continuously running jobs, the refreshLinkRecommendations shards a...
[15:27:49] <wikibugs>	 06serviceops, 06Growth-Team, 10GrowthExperiments, 10MW-on-K8s, 13Patch-For-Review: Migrate GrowthExperiments maintenance jobs to mw-cron - https://phabricator.wikimedia.org/T385782#10839784 (10Clement_Goubert) 05Open→03In progress p:05Triage→03High
[15:29:42] <wikibugs>	 06serviceops, 06Data-Persistence, 10MW-on-K8s, 13Patch-For-Review: Migrate ParserCachePurging jobs to mw-cron - https://phabricator.wikimedia.org/T385800#10839790 (10Clement_Goubert) 05Open→03In progress p:05Triage→03High
[15:30:36] <wikibugs>	 06serviceops, 06Commons, 10MediaWiki-File-management, 10MW-on-K8s, 13Patch-For-Review: Migrate cleanupUploadStash job to mw-cron - https://phabricator.wikimedia.org/T385868#10839797 (10Clement_Goubert) 05Open→03In progress p:05Triage→03High
[15:30:43] <wikibugs>	 06serviceops, 06Security-Team, 07SecTeam-Processed: Migrate Security-Team jobs to mw-cron - https://phabricator.wikimedia.org/T388531#10839800 (10Clement_Goubert) p:05Triage→03High
[15:31:12] <wikibugs>	 06serviceops, 10MediaWiki-Categories: Migrate recount_categories job to mw-cron - https://phabricator.wikimedia.org/T388533#10839803 (10Clement_Goubert) 05Open→03Resolved
[15:31:34] <wikibugs>	 06serviceops, 06Language and Product Localization: Migrate language_and_product_localization jobs to mw-cron - https://phabricator.wikimedia.org/T388539#10839806 (10Clement_Goubert) p:05Triage→03High
[15:31:45] <wikibugs>	 06serviceops, 06Trust and Safety Product Team: Migrate trust_and_safety_product_team jobs to mw-cron - https://phabricator.wikimedia.org/T388542#10839813 (10Clement_Goubert) 05Open→03In progress p:05Triage→03High
[15:32:02] <wikibugs>	 06serviceops, 10Wikidata, 10Wikidata Integration in Wikimedia projects: Migrate wikidata_dev_team jobs to mw-cron - https://phabricator.wikimedia.org/T388543#10839817 (10Clement_Goubert) p:05Triage→03High
[15:36:06] <wikibugs>	 06serviceops, 10MW-on-K8s, 06Trust and Safety Product Team, 10MediaModeration (MediaModeration 2.1), 13Patch-For-Review: Migrate MediaModeration jobs to mw-cron - https://phabricator.wikimedia.org/T385799#10839853 (10Clement_Goubert) 05Open→03In progress
[15:39:36] <wikibugs>	 06serviceops, 10MW-on-K8s, 06Trust and Safety Product Team, 10MediaModeration (MediaModeration 2.1), 13Patch-For-Review: Migrate MediaModeration jobs to mw-cron - https://phabricator.wikimedia.org/T385799#10839895 (10Clement_Goubert) >>! In T385799#10787866, @Dreamy_Jazz wrote: > I'm not sure if it is re...
[15:39:51] <wikibugs>	 06serviceops, 10MW-on-K8s, 06Trust and Safety Product Team, 10MediaModeration (MediaModeration 2.1), 13Patch-For-Review: Migrate MediaModeration jobs to mw-cron - https://phabricator.wikimedia.org/T385799#10839898 (10Clement_Goubert) p:05Medium→03High
[15:40:15] <wikibugs>	 06serviceops, 10Discovery-Search (2025.05.02 - 2025.05.23), 13Patch-For-Review: Migrate discovery-search jobs to mw-cron - https://phabricator.wikimedia.org/T388538#10839901 (10Clement_Goubert) p:05Triage→03High
[15:40:32] <wikibugs>	 06serviceops, 10MediaWiki-Special-pages, 13Patch-For-Review: Migrate updatequerypages/update_special_pages/initsitestats jobs to mw-cron - https://phabricator.wikimedia.org/T388534#10839912 (10Clement_Goubert) p:05Medium→03High
[15:45:57] <wikibugs>	 06serviceops, 06Growth-Team, 10GrowthExperiments, 10MW-on-K8s, 13Patch-For-Review: Migrate GrowthExperiments maintenance jobs to mw-cron - https://phabricator.wikimedia.org/T385782#10839937 (10Michael) >>! In T385782#10839778, @hnowlan wrote: > Now that we've largely unblocked the continuously running jo...
[16:01:16] <wikibugs>	 06serviceops, 10Wikidata, 10Wikidata Integration in Wikimedia projects: Migrate wikidata_dev_team jobs to mw-cron - https://phabricator.wikimedia.org/T388543#10839985 (10Clement_Goubert) 05Open→03Resolved
[16:49:42] <wikibugs>	 06serviceops, 10Wikidata, 10Wikidata Integration in Wikimedia projects: Migrate wikidata_dev_team jobs to mw-cron - https://phabricator.wikimedia.org/T388543#10840249 (10Clement_Goubert) ` kubectl logs wikidata-resubmit-changes-for-dispatch-29129319-q8psf mediawiki-main-app  extensions/Wikibase/repo/main...
[17:11:54] <wikibugs>	 06serviceops, 10MediaWiki-Special-pages, 13Patch-For-Review: Migrate updatequerypages/update_special_pages/initsitestats jobs to mw-cron - https://phabricator.wikimedia.org/T388534#10840442 (10hnowlan)
[17:16:25] <wikibugs>	 06serviceops, 10MW-on-K8s, 06Trust and Safety Product Team, 10MediaModeration (MediaModeration 2.1), 13Patch-For-Review: Migrate MediaModeration jobs to mw-cron - https://phabricator.wikimedia.org/T385799#10840508 (10Dreamy_Jazz) >>! In T385799#10839895, @Clement_Goubert wrote: > On that note, do you thi...
[17:59:58] <wikibugs>	 06serviceops, 06DBA, 10Editing-team (Tracking), 10MW-1.45-notes (1.45.0-wmf.1; 2025-05-13), and 2 others: Fatal exception of type "DBUnexpectedError: Database servers in extension1 are overloaded." affecting page views - https://phabricator.wikimedia.org/T393513#10840676 (10Krinkle)
[18:45:24] <wikibugs>	 06serviceops, 10Discovery-Search (2025.05.02 - 2025.05.23), 13Patch-For-Review: Migrate discovery-search jobs to mw-cron - https://phabricator.wikimedia.org/T388538#10840891 (10EBernhardson) >>! In T388538#10839770, @Clement_Goubert wrote: > Coming back to this, we could also shard that script by database se...
[19:04:26] <wikibugs>	 06serviceops, 06DBA, 10Editing-team (Tracking), 10MW-1.45-notes (1.45.0-wmf.1; 2025-05-13), and 2 others: Fatal exception of type "DBUnexpectedError: Database servers in extension1 are overloaded." affecting page views - https://phabricator.wikimedia.org/T393513#10841029 (10jijiki) p:05Triage→03High
[20:33:53] <wikibugs>	 06serviceops: Investigate performance degradation at high concurrencies in php-fpm - https://phabricator.wikimedia.org/T293630#10841469 (10Krinkle)
[21:10:15] <wikibugs>	 06serviceops, 06DBA, 10Editing-team (Tracking), 10MW-1.45-notes (1.45.0-wmf.1; 2025-05-13), and 2 others: Fatal exception of type "DBUnexpectedError: Database servers in extension1 are overloaded." affecting page views - https://phabricator.wikimedia.org/T393513#10841658 (10jijiki) Capturing number of esta...
[21:43:23] <wikibugs>	 06serviceops, 10DNS, 06SRE, 06Traffic: Create redirect from tj.*.org to tg.*.org - https://phabricator.wikimedia.org/T393803#10841877 (10Dzahn)
[21:48:34] <wikibugs>	 06serviceops, 10DNS, 06SRE, 06Traffic: Create redirect from tj.*.org to tg.*.org - https://phabricator.wikimedia.org/T393803#10841893 (10Dzahn) Added serviceops because this would have to be added in Apache/appserver redirects (as opposed to redirecting entire domains in ncredir service, which would be tra...