[00:28:53] 10Gerrit, 06collaboration-services: Investigate Gerrit root disk usage and logging - https://phabricator.wikimedia.org/T425667#12032130 (10Dzahn) @ABran-WMF Is this alert above just telling us that there were NO 5xxs on one of the hosts? I checked with curl and while we have some 5xx on the prod machine there... [05:28:59] RECOVERY - jenkins_service_running on contint1003 is OK: PROCS OK: 1 process with regex args .*/bin/java .*-jar /usr/share/java/jenkins.war https://wikitech.wikimedia.org/wiki/Jenkins [05:31:59] PROBLEM - jenkins_service_running on contint1003 is CRITICAL: PROCS CRITICAL: 0 processes with regex args .*/bin/java .*-jar /usr/share/java/jenkins.war https://wikitech.wikimedia.org/wiki/Jenkins [07:17:27] (03PS1) 10Hslater: Zuul: [mediawiki/extensions/PreviewLinks] Re-enable master jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1303906 [07:21:47] (03CR) 10Hashar: [C:03+2] "Awesome! I have updated the Jenkins job 🎉" [integration/config] - 10https://gerrit.wikimedia.org/r/1303491 (https://phabricator.wikimedia.org/T429547) (owner: 10Pwangai) [07:23:29] (03Merged) 10jenkins-bot: jjb: [maven-java] Authenticate sonar via SONAR_TOKEN [integration/config] - 10https://gerrit.wikimedia.org/r/1303491 (https://phabricator.wikimedia.org/T429547) (owner: 10Pwangai) [07:24:25] 10Continuous-Integration-Config, 06Test Platform, 13Patch-For-Review, 06Quality-and-Test-Engineering-Team (SonarCloud Admin): Migrate sonar analysis from deprecated sonar.login to sonar.token - https://phabricator.wikimedia.org/T429547#12032415 (10hashar) 05Open→03In progress a:03pwangai [07:41:31] (03PS1) 10Hslater: Zuul: [mediawiki/extensions/PageCheckout] Re-enable master jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1303917 [07:46:35] (03CR) 10Hashar: [C:03+2] "@mail@lucaswerkmeister.de yes that is the tricky part :-] We shall see!" [integration/config] - 10https://gerrit.wikimedia.org/r/1303351 (https://phabricator.wikimedia.org/T428975) (owner: 10DCausse) [07:47:57] (03PS1) 10Hslater: Zuul: [mediawiki/extensions/StandardDialogs] Re-enable master jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1303921 [07:48:07] 10Continuous-Integration-Config, 06Test Platform, 13Patch-For-Review, 06Quality-and-Test-Engineering-Team (SonarCloud Admin): Migrate sonar analysis from deprecated sonar.login to sonar.token - https://phabricator.wikimedia.org/T429547#12032443 (10JAllemandou) I confirm the original `refinery-source` patch... [07:48:16] (03Merged) 10jenkins-bot: Zuul: [CirrusSearch] add WikibaseLexemeCirrusSearch [integration/config] - 10https://gerrit.wikimedia.org/r/1303351 (https://phabricator.wikimedia.org/T428975) (owner: 10DCausse) [08:12:32] 10Continuous-Integration-Config, 10CirrusSearch, 10Wikidata, 10Wikidata Lexicographical data, and 5 others: LexemeFullTextQueryBuilderTest::testSearchElastic CI test errors - https://phabricator.wikimedia.org/T428975#12032536 (10dcausse) a:03hashar @hashar thanks!! [08:12:34] 10Continuous-Integration-Config, 10CirrusSearch, 10Wikidata, 10Wikidata Lexicographical data, and 5 others: LexemeFullTextQueryBuilderTest::testSearchElastic CI test errors - https://phabricator.wikimedia.org/T428975#12032541 (10hashar) [08:16:56] (03CR) 10Hashar: [C:03+2] Zuul: [mediawiki/extensions/PageCheckout] Re-enable master jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1303917 (owner: 10Hslater) [08:17:14] (03CR) 10Hashar: [C:03+2] Zuul: [mediawiki/extensions/StandardDialogs] Re-enable master jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1303921 (owner: 10Hslater) [08:17:27] (03CR) 10Hashar: [C:03+2] Zuul: [mediawiki/extensions/PreviewLinks] Re-enable master jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1303906 (owner: 10Hslater) [08:18:58] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/PageCheckout] Re-enable master jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1303917 (owner: 10Hslater) [08:19:06] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/StandardDialogs] Re-enable master jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1303921 (owner: 10Hslater) [08:21:23] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/PreviewLinks] Re-enable master jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1303906 (owner: 10Hslater) [08:22:28] (03CR) 10Hashar: [C:03+2] "Deployed!" [integration/config] - 10https://gerrit.wikimedia.org/r/1303906 (owner: 10Hslater) [08:22:31] (03CR) 10Hashar: [C:03+2] "Deployed!" [integration/config] - 10https://gerrit.wikimedia.org/r/1303917 (owner: 10Hslater) [08:22:33] (03CR) 10Hashar: [C:03+2] "Deployed!" [integration/config] - 10https://gerrit.wikimedia.org/r/1303921 (owner: 10Hslater) [08:39:39] (03PS2) 10Hashar: Zuul: […/FlaggedRevs] Drop GoogleNewsSitemap dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1284724 (https://phabricator.wikimedia.org/T421798) (owner: 10Jforrester) [08:42:46] (03CR) 10Hashar: "Hooks got removed by I5af7f01371f0348606dcfd2cf96b504e414850d3 as part of decommissioning GoogleNewsSitemap (T421798)." [integration/config] - 10https://gerrit.wikimedia.org/r/1284724 (https://phabricator.wikimedia.org/T421798) (owner: 10Jforrester) [08:54:51] (03CR) 10SomeRandomDeveloper: "> That'll break Phan on the release branches which are still receiving localization updates :-\" [integration/config] - 10https://gerrit.wikimedia.org/r/1284724 (https://phabricator.wikimedia.org/T421798) (owner: 10Jforrester) [09:08:22] 10Continuous-Integration-Config, 06Release-Engineering-Team: Introduce yamlllint to the integration/config repository - https://phabricator.wikimedia.org/T429576 (10hashar) 03NEW [09:10:54] (03PS1) 10Hashar: tox: introduce yamlllint [integration/config] - 10https://gerrit.wikimedia.org/r/1303970 (https://phabricator.wikimedia.org/T429576) [09:12:33] (03CR) 10CI reject: [V:04-1] tox: introduce yamlllint [integration/config] - 10https://gerrit.wikimedia.org/r/1303970 (https://phabricator.wikimedia.org/T429576) (owner: 10Hashar) [09:16:20] (03PS2) 10Hashar: tox: introduce yamlllint [integration/config] - 10https://gerrit.wikimedia.org/r/1303970 (https://phabricator.wikimedia.org/T429576) [09:39:48] 10GitLab (CI & Job Runners), 06Release-Engineering-Team, 06collaboration-services: Standardize Debian package builds on GitLab CI - https://phabricator.wikimedia.org/T304491#12032875 (10MatthewVernon) In general, you don't want the same version in two suites - while it might work, different e.g. C library ve... [10:23:19] 06Release-Engineering-Team (Priority Backlog 📥), 05Release, 05Train Deployments: 1.47.0-wmf.7 deployment blockers - https://phabricator.wikimedia.org/T423916#12033011 (10Dreamy_Jazz) [10:28:10] (03PS3) 10Hashar: tox: introduce yamlllint [integration/config] - 10https://gerrit.wikimedia.org/r/1303970 (https://phabricator.wikimedia.org/T429576) [10:32:38] 06Release-Engineering-Team (Priority Backlog 📥), 05Release, 05Train Deployments: 1.47.0-wmf.7 deployment blockers - https://phabricator.wikimedia.org/T423916#12033069 (10Dreamy_Jazz) [11:01:57] 10GitLab: Gitlab requires logging in (at least) every day - https://phabricator.wikimedia.org/T429587 (10zeljkofilipin) 03NEW [11:06:43] 10GitLab: Gitlab requires logging in (at least) every day - https://phabricator.wikimedia.org/T429587#12033186 (10zeljkofilipin) Workaround provided by @hashar > Gitlab sets a _gitlab_session cookie which is set to expire at the end of the session (ie when closing the web browser). > > The fix is to edit the... [11:09:59] lets FORCE_COLOR / NO_COLOR everything! https://github.com/adrienverge/yamllint/pull/813 [11:48:41] 10Continuous-Integration-Config, 06Test Platform, 10Tool-wmf-openapi-linter, 03[MWI] FY2025-26 Q4, and 2 others: [5.2.5b Epic] Prepare the linter for use in CI - https://phabricator.wikimedia.org/T422918#12033296 (10hashar) [12:08:32] FIRING: InstanceDown: Project deployment-prep instance deployment-mwmaint03 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [12:08:39] 10Beta-Cluster-Infrastructure: Project deployment-prep instance deployment-mwmaint03 is down - https://phabricator.wikimedia.org/T429592 (10wmcs-alerts) 03NEW [12:13:32] RESOLVED: InstanceDown: Project deployment-prep instance deployment-mwmaint03 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [12:44:41] 10GitLab (Infrastructure), 06collaboration-services: Explore apt pining / hold for gitlab-ce package - https://phabricator.wikimedia.org/T429595 (10Jelto) 03NEW [13:09:34] 10Beta-Cluster-Infrastructure, 13Patch-For-Review: Eliminate sudo rm -rf /var/lib/puppet/ssl step in new deployment-prep WMCS project (and others) - https://phabricator.wikimedia.org/T429413#12033569 (10Andrew) >>! In T429413#12027023, @dancy wrote: > @Andrew Do you anticipate any issues if we set `profile::pu... [13:52:40] 06Release-Engineering-Team (Doing 😎), 10Catalyst (Luka Ijo Pimeja Jan), 07Essential-Work, 13Patch-For-Review: Remove legacy localsettings from codebase - https://phabricator.wikimedia.org/T429182#12033834 (10jnuche) 05Open→03Resolved Files have now been removed and replaced with a README pointing t... [14:08:32] FIRING: InstanceDown: Project deployment-prep instance deployment-mwmaint03 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [14:18:17] 06Release-Engineering-Team (Priority Backlog 📥), 05Release, 05Train Deployments: 1.47.0-wmf.7 deployment blockers - https://phabricator.wikimedia.org/T423916#12034042 (10ihurbain) https://gerrit.wikimedia.org/r/c/mediawiki/extensions/DiscussionTools/+/1304061 should fix the DT one (T429582) and can be backpo... [14:26:43] 06Release-Engineering-Team, 10Scap, 07Developer Productivity, 06MediaWiki-Platform-Team (Radar): Consider reducing output in pre-sync phase of scap-backport - https://phabricator.wikimedia.org/T323325#12034117 (10JTweed-WMF) [14:30:04] 10Scap, 10MediaWiki-Internationalization, 10MW-on-K8s, 06MediaWiki-Platform-Team (Kanban Board), 13Patch-For-Review: Investigate unexpected l10n cache rebuilds - https://phabricator.wikimedia.org/T316714#12034138 (10JTweed-WMF) [14:43:32] RESOLVED: InstanceDown: Project deployment-prep instance deployment-mwmaint03 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [14:46:59] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 10QS-Test-Automation, 13Patch-For-Review, 06Quality-and-Test-Engineering-Team (Quality Engineering): Better prevent changes in MW core from breaking WMF production extensions - https://phabricator.wikimedia.org/T404403#12034288 (10SLong-... [15:11:14] 10Continuous-Integration-Config, 06Test Platform, 06Quality-and-Test-Engineering-Team (SonarCloud Admin): Migrate sonar analysis from deprecated sonar.login to sonar.token - https://phabricator.wikimedia.org/T429547#12034405 (10pwangai) p:05High→03Medium [15:15:13] !log Deleting deployment-db11 and deployment-db14 (T428910) [15:15:16] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:15:16] T428910: Beta Cluster MariaDB is still 10.6.17, MW now requires 10.11 - https://phabricator.wikimedia.org/T428910 [15:15:25] Anyone coming to the train log triage meeting or did y'all already do triage out of band? if so, any highlights? (couple of us hanging out in the gmeet) [15:17:16] apergos: I am skipping it, I got another meeting [15:17:24] (and haven't followed the train this week) [15:17:42] thanks for the info [16:45:10] (03PS1) 10Hashar: tox.ini: upgrade editorconfig-checker 2.4.0..3.6.1 [integration/config] - 10https://gerrit.wikimedia.org/r/1304116 [17:05:14] (03CR) 10Hashar: "In 2022 you had editorconfig-checker pinned ( I435e2e425f24daf0b9ef1a061c573a4bfba8c4bc ), probably because as upstream released a new ver" [integration/config] - 10https://gerrit.wikimedia.org/r/1304116 (owner: 10Hashar) [17:06:39] (03PS4) 10Jforrester: tox: introduce yamllint [integration/config] - 10https://gerrit.wikimedia.org/r/1303970 (https://phabricator.wikimedia.org/T429576) (owner: 10Hashar) [17:52:24] (03CR) 10Ahmon Dancy: [C:03+2] zuul: retry on connection resets, not only GnuTLS errors [integration/quibble] - 10https://gerrit.wikimedia.org/r/1302819 (https://phabricator.wikimedia.org/T420865) (owner: 10Arnaudb) [18:09:52] (03Merged) 10jenkins-bot: zuul: retry on connection resets, not only GnuTLS errors [integration/quibble] - 10https://gerrit.wikimedia.org/r/1302819 (https://phabricator.wikimedia.org/T420865) (owner: 10Arnaudb) [18:17:37] 10Beta-Cluster-Infrastructure, 06Data-Platform-SRE (2026-06-05 - 2026-06-26): Write lightweight OCI-image-based Puppet plans for beta cluster - https://phabricator.wikimedia.org/T425585#12035040 (10bking) [18:21:14] 10Beta-Cluster-Infrastructure, 06Growth-Team, 10GrowthExperiments-NewcomerTasks, 06Discovery-Search (2026.06.01 - 2026.07.03): [beta-cluster] Fetching task suggestions failed: cirrussearch-backend-error - https://phabricator.wikimedia.org/T427196#12035043 (10bking) Thanks everyone for your patience and sor... [18:34:36] (03PS1) 10Ahmon Dancy: Changelog: 1.18.2 release [integration/quibble] - 10https://gerrit.wikimedia.org/r/1304148 [18:41:53] (03CR) 10Ahmon Dancy: [C:03+2] Changelog: 1.18.2 release [integration/quibble] - 10https://gerrit.wikimedia.org/r/1304148 (owner: 10Ahmon Dancy) [18:45:27] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06collaboration-services, 13Patch-For-Review: puppetize setup of new zuul VMs - https://phabricator.wikimedia.org/T395938#12035113 (10Dzahn) @dduvall Currently I don't have any open patches here anymore. Got any opinions on what is missing on the puppet... [18:47:23] (03Merged) 10jenkins-bot: Changelog: 1.18.2 release [integration/quibble] - 10https://gerrit.wikimedia.org/r/1304148 (owner: 10Ahmon Dancy) [18:50:34] !log Tag Quibble 1.18.2 @ 152b497d317eaafc3cec334d5ce7e549697a2980 # T420865 [18:50:37] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:50:37] T420865: Fetches from Gerrit aborted due to: GnuTLS recv error (-54): Error in the pull function - https://phabricator.wikimedia.org/T420865 [18:52:00] (03PS1) 10Ahmon Dancy: release: Start new version cycle [integration/quibble] - 10https://gerrit.wikimedia.org/r/1304155 [18:53:56] (03CR) 10Ahmon Dancy: [C:03+2] release: Start new version cycle [integration/quibble] - 10https://gerrit.wikimedia.org/r/1304155 (owner: 10Ahmon Dancy) [19:03:55] (03Merged) 10jenkins-bot: release: Start new version cycle [integration/quibble] - 10https://gerrit.wikimedia.org/r/1304155 (owner: 10Ahmon Dancy) [19:21:52] (03PS1) 10Ahmon Dancy: utils/update-quibble: Handle dumb terminals gracefully [integration/config] - 10https://gerrit.wikimedia.org/r/1304160 [19:23:08] (03PS1) 10Ahmon Dancy: dockerfiles: update Quibble to 1.18.2 [integration/config] - 10https://gerrit.wikimedia.org/r/1304162 [19:24:04] (03PS2) 10Ahmon Dancy: dockerfiles: update Quibble to 1.18.2 [integration/config] - 10https://gerrit.wikimedia.org/r/1304162 (https://phabricator.wikimedia.org/T420865) [19:24:51] (03CR) 10Ahmon Dancy: [C:03+2] utils/update-quibble: Handle dumb terminals gracefully [integration/config] - 10https://gerrit.wikimedia.org/r/1304160 (owner: 10Ahmon Dancy) [19:26:33] (03Merged) 10jenkins-bot: utils/update-quibble: Handle dumb terminals gracefully [integration/config] - 10https://gerrit.wikimedia.org/r/1304160 (owner: 10Ahmon Dancy) [19:31:56] (03PS1) 10Ahmon Dancy: utils/update-quibble: Set DEBEMAIL and DEBFULLNAME [integration/config] - 10https://gerrit.wikimedia.org/r/1304163 [19:41:34] 06Release-Engineering-Team (Priority Backlog 📥): Allow configuration of canary and production checks based on deployment target - https://phabricator.wikimedia.org/T428971#12035294 (10dancy) Thanks @Scott_French. Your suggested layout and sample config make sense to me and look like a good place to start experi... [19:42:51] (03CR) 10Ahmon Dancy: "Antoine, what do you think?" [integration/config] - 10https://gerrit.wikimedia.org/r/1304163 (owner: 10Ahmon Dancy) [19:49:24] (03CR) 10Ahmon Dancy: [C:03+2] dockerfiles: update Quibble to 1.18.2 [integration/config] - 10https://gerrit.wikimedia.org/r/1304162 (https://phabricator.wikimedia.org/T420865) (owner: 10Ahmon Dancy) [19:51:26] (03Merged) 10jenkins-bot: dockerfiles: update Quibble to 1.18.2 [integration/config] - 10https://gerrit.wikimedia.org/r/1304162 (https://phabricator.wikimedia.org/T420865) (owner: 10Ahmon Dancy) [19:52:52] !log Building quibble 1.18.2 images on contint primary (T420865) [19:52:54] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:52:54] T420865: Fetches from Gerrit aborted due to: GnuTLS recv error (-54): Error in the pull function - https://phabricator.wikimedia.org/T420865 [19:55:18] (03open) 10dduvall: ci: Directions and defaults for building from WMF branches [repos/releng/zuul/zuul] - 10https://gitlab.wikimedia.org/repos/releng/zuul/zuul/-/merge_requests/17 [19:58:21] (03update) 10dduvall: ci: Directions and defaults for building from WMF branches [repos/releng/zuul/zuul] - 10https://gitlab.wikimedia.org/repos/releng/zuul/zuul/-/merge_requests/17 [20:33:32] FIRING: InstanceDown: Project deployment-prep instance deployment-mwmaint03 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [21:27:41] (03PS1) 10Ahmon Dancy: Upgrade Quibble jobs to 1.18.2 [integration/config] - 10https://gerrit.wikimedia.org/r/1304181 [21:32:15] (03PS2) 10Ahmon Dancy: Upgrade Quibble jobs to 1.18.2 [integration/config] - 10https://gerrit.wikimedia.org/r/1304181 (https://phabricator.wikimedia.org/T420865) [21:53:48] (03merge) 10dduvall: ci: Directions and defaults for building from WMF branches [repos/releng/zuul/zuul] - 10https://gitlab.wikimedia.org/repos/releng/zuul/zuul/-/merge_requests/17 [21:54:28] (03CR) 10Ahmon Dancy: "@hashar@free.fr Would you mind deploying this for me? I'll be offline Friday-Monday. Back on Tuesday." [integration/config] - 10https://gerrit.wikimedia.org/r/1304181 (https://phabricator.wikimedia.org/T420865) (owner: 10Ahmon Dancy) [21:58:32] RESOLVED: InstanceDown: Project deployment-prep instance deployment-mwmaint03 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [23:11:23] (03update) 10dduvall: ci: Use `CI_PROJECT_URL` to avoid embedding CI token in image provenance [repos/releng/zuul/zuul] - 10https://gitlab.wikimedia.org/repos/releng/zuul/zuul/-/merge_requests/18 [23:11:27] (03open) 10dduvall: ci: Use `CI_PROJECT_URL` to avoid embedding CI token in image provenance [repos/releng/zuul/zuul] - 10https://gitlab.wikimedia.org/repos/releng/zuul/zuul/-/merge_requests/18 [23:12:26] (03merge) 10dduvall: ci: Use `CI_PROJECT_URL` to avoid embedding CI token in image provenance [repos/releng/zuul/zuul] - 10https://gitlab.wikimedia.org/repos/releng/zuul/zuul/-/merge_requests/18