[00:49:46] (03update) 10bd808: main.py: Improve wording of retry_continue_exit exit case [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1048 [00:50:55] (03update) 10bd808: main.py: Improve wording of retry_continue_exit exit case [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1048 [00:51:21] (03update) 10bd808: main.py: Improve wording of retry_continue_exit exit case [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1048 [03:08:03] 06Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 05Release, 05Train Deployments: 1.46.0-wmf.5 deployment blockers - https://phabricator.wikimedia.org/T408275#11422332 (10cscott) The above patch [Add post-processing cache, disabled by default (1205194)](https://gerrit.wikimedia.org/r/c/med... [03:30:13] Project mediawiki-core-doxygen build #15689: 04FAILURE in 12 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen/15689/ [04:28:20] Yippee, build fixed! [04:28:20] Project mediawiki-core-doxygen build #15690: 09FIXED in 10 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen/15690/ [09:30:30] Project mediawiki-core-doxygen build #15698: 04FAILURE in 12 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen/15698/ [10:00:10] Hi all! I created a new project on GitHub under mediawiki/services, but now I can't push to it: https://gitlab.wikimedia.org/repos/mediawiki/services/smoke-alarm# [10:00:54] It seems to me like I'm hitting the issue that basically anyone can create a project, but not everyone can create a branch. That seems... odd? Shouldn't an empty project come with a main branch? I see this has been an issue for some time... https://gitlab.com/gitlab-org/gitlab-foss/-/issues/51688 [10:02:33] Can someone please make me the owner of that new repo? Shouldn't the user who creates the project automatically be the owner?... [10:28:17] @jeena @kostajh looks like you are "owners" of that repo now, can you make me an owner or maintainer? Or set up a main branch I can push to? [10:31:40] Yippee, build fixed! [10:31:40] Project mediawiki-core-doxygen build #15699: 09FIXED in 13 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen/15699/ [10:45:21] 10Deployments, 06Release-Engineering-Team (Radar), 10observability, 10Observability-Metrics, 07Grafana: Graphana no more shows "MW deploy" "Train deployments" annotations - https://phabricator.wikimedia.org/T411474 (10hashar) 03NEW [11:20:10] (03CR) 10Hashar: [C:03+2] Adding extension:WP2EasterEggs to "in-wikimedia-production" job template [integration/config] - 10https://gerrit.wikimedia.org/r/1213565 (https://phabricator.wikimedia.org/T411408) (owner: 10Jdrewniak) [11:22:50] (03Merged) 10jenkins-bot: Adding extension:WP2EasterEggs to "in-wikimedia-production" job template [integration/config] - 10https://gerrit.wikimedia.org/r/1213565 (https://phabricator.wikimedia.org/T411408) (owner: 10Jdrewniak) [11:46:00] (03PS1) 10Hashar: Zuul: [mediawiki/extensions/WP25EasterEggs] Add CommunityConfiguration phan dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1214001 (https://phabricator.wikimedia.org/T411408) [11:46:12] (03CR) 10Hashar: [C:03+2] Zuul: [mediawiki/extensions/WP25EasterEggs] Add CommunityConfiguration phan dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1214001 (https://phabricator.wikimedia.org/T411408) (owner: 10Hashar) [11:46:42] (03PS2) 10Hashar: Zuul: [WP25EasterEggs] Add CommunityConfiguration phan dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1214001 (https://phabricator.wikimedia.org/T411408) [11:46:57] (03CR) 10Hashar: [C:03+2] Zuul: [WP25EasterEggs] Add CommunityConfiguration phan dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1214001 (https://phabricator.wikimedia.org/T411408) (owner: 10Hashar) [11:48:25] (03Merged) 10jenkins-bot: Zuul: [WP25EasterEggs] Add CommunityConfiguration phan dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1214001 (https://phabricator.wikimedia.org/T411408) (owner: 10Hashar) [11:54:23] 10Continuous-Integration-Config, 10MediaWiki-extensions-WP25EasterEggs, 13Patch-For-Review, 05PES1.3.3 WP25 Easter Eggs: Deploy extension:WP25EasterEggs to beta cluster - https://phabricator.wikimedia.org/T411408#11423208 (10hashar) [12:02:59] 10Continuous-Integration-Infrastructure, 10Cite, 10DiscussionTools, 07ci-test-error, and 2 others: CI fails in REL1_43 due to CI infrastructure change to not recursively expand dependencies - https://phabricator.wikimedia.org/T411339#11423245 (10hashar) [13:03:22] 06Project-Admins, 06Wiki-Mentor-Africa: Create project tag for WMA-HackChallenge-2025 - https://phabricator.wikimedia.org/T411491 (10Eugene233) 03NEW [13:07:58] 10Continuous-Integration-Infrastructure, 10Cite, 10DiscussionTools, 07ci-test-error, and 2 others: CI fails in REL1_43 due to CI infrastructure change to not recursively expand dependencies - https://phabricator.wikimedia.org/T411339#11423510 (10Tacsipacsi) [13:13:20] 06Project-Admins, 06Wiki-Mentor-Africa: Create project tag for WMA-HackChallenge-2025 - https://phabricator.wikimedia.org/T411491#11423554 (10Eugene233) [13:28:20] duesen: confusing gitlab and github in the same line? :P [13:31:39] 06Project-Admins, 06Release-Engineering-Team (Doing 😎), 06Wiki-Mentor-Africa: Create project tag for WMA-HackChallenge-2025 - https://phabricator.wikimedia.org/T411491#11423648 (10Aklapper) 05Open→03Resolved a:03Aklapper Requested public project #wma-hackchallenge-2025 has been created: https://pha... [14:11:37] 10Gerrit, 06collaboration-services, 13Patch-For-Review: gerrit2003 is trying to backup incrementally 3.5 million files every hour, clogging backus and filling in available disk space - https://phabricator.wikimedia.org/T406762#11423824 (10Jelto) >>! In T406762#11409848, @hashar wrote: > This task was for ger... [14:14:47] 10Gerrit, 06collaboration-services, 13Patch-For-Review: gerrit2003 is trying to backup incrementally 3.5 million files every hour, clogging backus and filling in available disk space - https://phabricator.wikimedia.org/T406762#11423843 (10jcrespo) The reason I ask is because the other hosts' backups are very... [14:36:21] Reedy: eh, sorry :D [14:42:15] 10GitLab, 06Release-Engineering-Team, 06collaboration-services: gitlab: Make @daniel an owner of repos/mediawiki/services/smoke-alarm - https://phabricator.wikimedia.org/T411502 (10daniel) 03NEW [15:46:05] (03approved) 10dancy: main.py: Improve wording of retry_continue_exit exit case [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1048 (owner: 10bd808) [15:47:36] 10Gerrit, 06collaboration-services, 13Patch-For-Review: gerrit2003 is trying to backup incrementally 3.5 million files every hour, clogging backus and filling in available disk space - https://phabricator.wikimedia.org/T406762#11424303 (10hashar) 05Open→03Resolved a:03hashar >>! In T406762#11410132... [15:48:23] so hmm having to maintain REL1_39 + REL1_43 + REL1_44 + REL1_45 is ... interesting [15:48:34] and I am forgetting fundraising/REL branch and the wmfbranches [15:48:43] I had enough backport madness for today [15:48:46] train looks quiet [15:49:01] I'll resume tomorrow ! [15:49:59] REL1_39 is EOL this month at least hashar :P [15:50:12] hopefully ! :) [15:50:23] Reedy: there will be one last release and after that is closed isn't it? [15:50:36] Basically [15:52:31] some repo might be failling in CI cause some I have disabled recursion for adding extension dependencies [15:53:58] I am also wondering whether we could reduce the overlap between version. Aka drop 1.44 support one 1.45 is released instead of the six months overlap [15:54:18] (same for LTS versions which have a 1 year long overlap) [15:54:29] anyway. I am off ;-] [16:00:03] maintenance-disconnect-full-disks build 759763 integration-agent-docker-1056 (/: 36%, /srv: 97%, /var/lib/docker: 38%): OFFLINE due to disk space [16:05:03] maintenance-disconnect-full-disks build 759764 integration-agent-docker-1056 (/: 36%, /srv: 66%, /var/lib/docker: 38%): RECOVERY disk space OK [16:05:38] 10Deployments, 06Release-Engineering-Team (Radar), 10observability, 10Observability-Metrics, 07Grafana: Grafana "MW deploy" "Train deployments" annotations broken on some dashboards - https://phabricator.wikimedia.org/T411474#11424397 (10colewhite) [16:11:15] CI is mostly a disaster because of a few unfortunate failures and a high stack of Wikidata patches, it seems. :-( [16:12:29] :( [16:12:47] do we have statistics on which tests are flaking the most? [16:13:11] browser tests I'm guessing [16:13:19] but I think releng might have some more stats [16:13:29] yeah, but I'm interested in which specific individual test cases are flaking [16:13:32] 10Diffusion, 10Phabricator, 06collaboration-services, 10Wikidata, and 3 others: WikibaseLexeme submodule is not available - https://phabricator.wikimedia.org/T409519#11424456 (10dancy) I'm currently blocked from updating my train-dev development environment because I can't clone https://phabricator.wikimed... [16:13:59] there are some bugs filed. but it's many individual test cases [16:14:09] (i can find these bugs in a minute) [16:14:10] And the more tests we add... [16:14:28] but there are clearly some underlying issues that cause things to fail seemingly at random [16:17:03] taavi: https://phabricator.wikimedia.org/T411278 is a very common failure recently [16:18:21] like, I'm guessing most of the random failures are from a very small amount of tests. so ihmo it'd be more productive to disable those tests if they're not getting fixed instead of continuing to waste hours of developer time from those flakes [16:20:00] we don't have statistics but my impression is that you'd have to disable everything with "selenium" or "cypress" in the name to achieve a semblance of reliability, and while that sounds good to me, some people like having those tests [16:21:30] there are many examples filed under #ci-test-error https://phabricator.wikimedia.org/maniphest/?project=PHID-PROJ-qiqvttgmlqcvfbpysisx&statuses=open()&group=none&order=newest#R [16:21:40] (03open) 10dancy: build-images.py: Update reference to train-dev registry [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/222 [16:21:43] (03update) 10dancy: build-images.py: Update reference to train-dev registry [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/222 [16:23:14] (03merge) 10dancy: build-images.py: Update reference to train-dev registry [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/222 [16:29:09] (03open) 10dancy: scap.cfg: Remove unused php_version variable [repos/releng/train-dev] - 10https://gitlab.wikimedia.org/repos/releng/train-dev/-/merge_requests/172 (https://phabricator.wikimedia.org/T396166) [16:29:09] (03update) 10dancy: scap.cfg: Remove unused php_version variable [repos/releng/train-dev] - 10https://gitlab.wikimedia.org/repos/releng/train-dev/-/merge_requests/172 (https://phabricator.wikimedia.org/T396166) [16:30:15] (03merge) 10dancy: scap.cfg: Remove unused php_version variable [repos/releng/train-dev] - 10https://gitlab.wikimedia.org/repos/releng/train-dev/-/merge_requests/172 (https://phabricator.wikimedia.org/T396166) [16:31:46] 10Scap, 10MediaWiki-Internationalization: Add ability to ignore missing extensions in mergeMessageFileList's `--list-file` input - https://phabricator.wikimedia.org/T411516 (10bd808) 03NEW [16:32:08] 10Scap, 10MediaWiki-Configuration: Scap reliance on extension-list in turn requires extensions to be deployed to WMF production before being deployed to the Beta Cluster - https://phabricator.wikimedia.org/T125678#11424564 (10bd808) [16:32:10] 10Scap, 10MediaWiki-Internationalization: Add ability to ignore missing extensions in mergeMessageFileList's `--list-file` input - https://phabricator.wikimedia.org/T411516#11424565 (10bd808) [16:34:20] 10Scap, 10MediaWiki-Internationalization: Add ability to ignore missing extensions in mergeMessageFileList's `--list-file` input - https://phabricator.wikimedia.org/T411516#11424579 (10bd808) I have recently been talking with @Catrope and @thcipriani about the underlying problem here and would like their feedb... [16:35:15] hey all, I'm about to reboot a bunch of gitlab runners (as per https://lists.wikimedia.org/hyperkitty/list/cloud-announce@lists.wikimedia.org/thread/2RV6C3TKSILUX6BGZZY4MFLLIJ6IEVDE/ ) -- this is your last chance to advise about how/when to do that [16:37:26] dancy, hashar, thcipriani ^ [16:43:20] I think the worst case is that jobs that are running there will fail and then they will come back. dancy may have smarter thoughts. [16:43:45] Nod. Just reboot. [16:44:10] ok! thank you, will start that now [16:44:48] !log Pausing WMCS Gitlab runners [16:44:53] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:45:01] I found a button in https://gitlab.wikimedia.org/admin/runners/1507 [16:56:55] dancy: all done [16:57:16] Awesome. Thanks! [16:57:39] !log Resumed WMCS Gitlab runners [16:57:40] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:02:44] (03open) 10dancy: web/package-lock.json: Bump js-yaml to 4.1.1 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1050 [17:02:48] (03update) 10dancy: web/package-lock.json: Bump js-yaml to 4.1.1 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1050 [17:09:27] 10Gerrit, 06collaboration-services, 13Patch-For-Review: gerrit2003 is trying to backup incrementally 3.5 million files every hour, clogging backus and filling in available disk space - https://phabricator.wikimedia.org/T406762#11424916 (10Dzahn) The task was to open a new task. [17:18:02] 06Release-Engineering-Team (Doing 😎), 10Catalyst (Luka Moli Telo), 07Essential-Work, 13Patch-For-Review: Migration to MariaDB operator: Catalyst API DB - https://phabricator.wikimedia.org/T408116#11424994 (10jnuche) 05Open→03Resolved DB migrated successfully in production [17:18:56] 06Project-Admins: Create project tag for - https://phabricator.wikimedia.org/T411529 (10Tbodt) 03NEW [17:22:05] 06Project-Admins: Create project tag for MediaWiki-extensions-MultiTitle - https://phabricator.wikimedia.org/T411529#11425037 (10Aklapper) [17:23:46] 06Project-Admins, 06Release-Engineering-Team (Doing 😎): Create project tag for MediaWiki-extensions-MultiTitle - https://phabricator.wikimedia.org/T411529#11425039 (10Aklapper) 05Open→03Resolved a:03Aklapper Requested public project #mediawiki-extensions-multititle has been created: https://phabricat... [17:42:16] (03PS1) 10Dillon: Move Thanks to extras for PersonalDashboard [integration/config] - 10https://gerrit.wikimedia.org/r/1214128 (https://phabricator.wikimedia.org/T402783) [17:47:48] (03CR) 10Dillon: Update dependencies for PersonalDashboard (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/1213555 (https://phabricator.wikimedia.org/T402783) (owner: 10Dillon) [18:12:52] (03update) 10dancy: web/package-lock.json: Bump js-yaml to 4.1.1 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1050 [18:14:05] (03merge) 10dancy: web/package-lock.json: Bump js-yaml to 4.1.1 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1050 [18:21:51] (03open) 10dancy: Revise job card appearance in job history list [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1051 (https://phabricator.wikimedia.org/T393128) [18:21:53] (03update) 10dancy: Revise job card appearance in job history list [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1051 (https://phabricator.wikimedia.org/T393128) [18:23:07] (03update) 10dancy: Revise job card appearance in job history list [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1051 (https://phabricator.wikimedia.org/T393128) [18:36:17] 06Release-Engineering-Team (Doing 😎), 10Catalyst (Luka Moli Telo), 07Essential-Work: Patchdemo wiki creation fails due to PHP 8.1 not supported by MW - https://phabricator.wikimedia.org/T411277#11425464 (10Krinkle) [18:45:35] (03PS1) 10Techwizzie: Add dependency for WandaScore extension [integration/config] - 10https://gerrit.wikimedia.org/r/1214139 [18:47:24] (03PS1) 10Techwizzie: Add dependency for WandaScribe extension [integration/config] - 10https://gerrit.wikimedia.org/r/1214140 [18:49:31] 10Deployments, 06Release-Engineering-Team (Radar), 07Grafana: Grafana "MW deploy" "Train deployments" annotations broken on some dashboards - https://phabricator.wikimedia.org/T411474#11425557 (10colewhite) The replacement for this annotation tool is to use the `Public Logs` datasource in Grafana which is ba... [18:53:13] 10Continuous-Integration-Config, 06Growth-Team, 06Test Kitchen, 13Patch-For-Review: Add MetricsPlatform to gated extensions - https://phabricator.wikimedia.org/T404058#11425569 (10JVanderhoop-WMF) [18:54:21] 06Release-Engineering-Team (Radar), 10Scap, 07Grafana: Grafana "MW deploy" "Train deployments" annotations broken on some dashboards - https://phabricator.wikimedia.org/T411474#11425576 (10hashar) Scap has: ` name=scap/main.py,lang=python def increment_stat(self, stat, all_stat=True, value=1): ""... [19:03:55] 06Release-Engineering-Team (Radar), 10Scap, 07Grafana: Grafana "MW deploy" "Train deployments" annotations broken on some dashboards - https://phabricator.wikimedia.org/T411474#11425617 (10hashar) Note from @colewhite, the scap logs are available at (requires NDA login) https://grafana.wikimedia.org/goto/XO7... [21:23:38] 10Beta-Cluster-Infrastructure, 06Traffic, 06MediaWiki-Platform-Team (Radar): Make "Requests from your IP have been blocked" more visible in Beta cluster error page - https://phabricator.wikimedia.org/T411552 (10Krinkle) 03NEW [21:29:22] 10Beta-Cluster-Infrastructure, 06Traffic, 06MediaWiki-Platform-Team (Radar), 13Patch-For-Review: Make "Requests from your IP have been blocked" more visible in Beta cluster error page - https://phabricator.wikimedia.org/T411552#11426103 (10taavi) Duplicate of {T401489}? [21:40:14] 10Beta-Cluster-Infrastructure, 06Traffic, 06MediaWiki-Platform-Team (Radar), 13Patch-For-Review: Make "Requests from your IP have been blocked" more visible in Beta cluster error page - https://phabricator.wikimedia.org/T411552#11426155 (10Krinkle) 05Open→03Invalid Thanks. [21:45:38] 10Beta-Cluster-Infrastructure, 06Traffic, 13Patch-For-Review: Separate error templates for 5xx (server errors) and 4xx (IP blocks) or parametrize Varnish messaging - https://phabricator.wikimedia.org/T401489#11426182 (10bd808) [21:45:46] 10Beta-Cluster-Infrastructure, 06Traffic, 06MediaWiki-Platform-Team (Radar), 13Patch-For-Review: Make "Requests from your IP have been blocked" more visible in Beta cluster error page - https://phabricator.wikimedia.org/T411552#11426185 (10bd808) →14Duplicate dup:03T401489 [21:58:24] (03update) 10bd808: main.py: Improve wording of retry_continue_exit exit case [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1048 [23:27:09] (03merge) 10bd808: main.py: Improve wording of retry_continue_exit exit case [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1048 [23:41:17] (03approved) 10jhuneidi: Revise job card appearance in job history list [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1051 (https://phabricator.wikimedia.org/T393128) (owner: 10dancy)