[06:25:02] 06Project-Admins, 06Release-Engineering-Team (Doing 😎): Create project tag for wikilovesmonuments.az - https://phabricator.wikimedia.org/T440150#12420197 (10Aklapper) 05Open→03Resolved a:03Aklapper Requested public project tag has been created: #wikilovesmonuments.az (In case you need to edit the pr... [07:31:09] (03PS2) 10Hashar: jjb: Enable pcov by default in coverage jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1354251 (https://phabricator.wikimedia.org/T420009) (owner: 10Daimona Eaytoy) [07:34:07] (03CR) 10Hashar: [C:03+2] "For the historical context, I think that is inherited from when we used:" [integration/config] - 10https://gerrit.wikimedia.org/r/1354250 (https://phabricator.wikimedia.org/T420009) (owner: 10Daimona Eaytoy) [07:37:19] (03CR) 10CI reject: [V:04-1] Docker: [quibble-coverage] Enable pcov by default [integration/config] - 10https://gerrit.wikimedia.org/r/1354250 (https://phabricator.wikimedia.org/T420009) (owner: 10Daimona Eaytoy) [07:46:04] (03CR) 10Hashar: [C:03+2] "usual docker-registry time out:" [integration/config] - 10https://gerrit.wikimedia.org/r/1354250 (https://phabricator.wikimedia.org/T420009) (owner: 10Daimona Eaytoy) [07:49:15] (03CR) 10CI reject: [V:04-1] Docker: [quibble-coverage] Enable pcov by default [integration/config] - 10https://gerrit.wikimedia.org/r/1354250 (https://phabricator.wikimedia.org/T420009) (owner: 10Daimona Eaytoy) [08:02:06] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team (Doing 😎), 10Release Pipeline: Upgrade integration-agent-docker instances to trixie - https://phabricator.wikimedia.org/T440670#12420417 (10hashar) This morning I had the job https://integration.wikimedia.org/ci/job/integration-config-zuul-l... [08:04:27] (03CR) 10Hashar: [C:03+2] "That is an issue with a newer Jenkins agent which is based on Trixie. I have disabled the agent and commented on the task: https://phabric" [integration/config] - 10https://gerrit.wikimedia.org/r/1354250 (https://phabricator.wikimedia.org/T420009) (owner: 10Daimona Eaytoy) [08:05:37] (03Merged) 10jenkins-bot: Docker: [quibble-coverage] Enable pcov by default [integration/config] - 10https://gerrit.wikimedia.org/r/1354250 (https://phabricator.wikimedia.org/T420009) (owner: 10Daimona Eaytoy) [08:12:02] (03CR) 10Hashar: [C:03+2] "I have amended the commit message to show the list of affected jobs which can be retrieved by running `./utils/jjb-diff.sh`." [integration/config] - 10https://gerrit.wikimedia.org/r/1354251 (https://phabricator.wikimedia.org/T420009) (owner: 10Daimona Eaytoy) [08:12:41] (03CR) 10Hashar: [C:03+2] zuul: add Mr. Starfleet Command to CI allowlist [integration/config] - 10https://gerrit.wikimedia.org/r/1353335 (https://phabricator.wikimedia.org/T436312) (owner: 10SD0001) [08:13:09] (03Merged) 10jenkins-bot: jjb: Enable pcov by default in coverage jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1354251 (https://phabricator.wikimedia.org/T420009) (owner: 10Daimona Eaytoy) [08:14:14] (03Merged) 10jenkins-bot: zuul: add Mr. Starfleet Command to CI allowlist [integration/config] - 10https://gerrit.wikimedia.org/r/1353335 (https://phabricator.wikimedia.org/T436312) (owner: 10SD0001) [08:36:11] (03CR) 10Hashar: [V:03+2 C:03+2] "I had +2ed it earlier before finding out Ahmon had made remarks. They have been taken in account." [mediawiki] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/1346021 (https://phabricator.wikimedia.org/T439548) (owner: 10Thcipriani) [08:37:13] (03CR) 10Hashar: [C:03+2] Zuul: [mediawiki/extensions/WikimediaAntiAbuse] Add CheckUser phan dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1353231 (owner: 10Kosta Harlan) [08:38:24] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/WikimediaAntiAbuse] Add CheckUser phan dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1353231 (owner: 10Kosta Harlan) [08:42:15] (03CR) 10Hashar: [C:03+2] fab: allocate pseudo-terminal for git coloring [integration/config] - 10https://gerrit.wikimedia.org/r/1352359 (owner: 10Hashar) [08:43:25] (03Merged) 10jenkins-bot: fab: allocate pseudo-terminal for git coloring [integration/config] - 10https://gerrit.wikimedia.org/r/1352359 (owner: 10Hashar) [10:00:01] (03PS1) 10Hashar: Remove image-diff plugin [software/gerrit] (wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1354387 (https://phabricator.wikimedia.org/T341291) [10:39:34] (03PS1) 10Hashar: jjb: [gerrit] capture plugins in a zip file [integration/config] - 10https://gerrit.wikimedia.org/r/1354404 [10:39:51] (03CR) 10Hashar: [C:03+2] "Tested live" [integration/config] - 10https://gerrit.wikimedia.org/r/1354404 (owner: 10Hashar) [10:41:01] (03Merged) 10jenkins-bot: jjb: [gerrit] capture plugins in a zip file [integration/config] - 10https://gerrit.wikimedia.org/r/1354404 (owner: 10Hashar) [10:41:43] (03CR) 10Hashar: "recheck after having deployed Ic4cb993cbebaaa3e5ae01efa2faed10a0faf0ff2 which makes the Jenkins job to archive all plugins in an `all_plug" [software/gerrit] (wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1354387 (https://phabricator.wikimedia.org/T341291) (owner: 10Hashar) [10:48:28] (03CR) 10CI reject: [V:04-1] Remove image-diff plugin [software/gerrit] (wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1354387 (https://phabricator.wikimedia.org/T341291) (owner: 10Hashar) [10:59:17] (03CR) 10Hashar: "recheck `//plugins/its-base:its_base_tests` is apparently flappy" [software/gerrit] (wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1354387 (https://phabricator.wikimedia.org/T341291) (owner: 10Hashar) [11:07:03] (03CR) 10Hashar: [C:03+2] Remove image-diff plugin [software/gerrit] (wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1354387 (https://phabricator.wikimedia.org/T341291) (owner: 10Hashar) [11:13:33] (03Merged) 10jenkins-bot: Remove image-diff plugin [software/gerrit] (wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1354387 (https://phabricator.wikimedia.org/T341291) (owner: 10Hashar) [11:18:12] (03PS7) 10Hashar: Merge tag 'stable-3.10-integration-issues-538883789-539032891-540877705-541287630-545820925' into wmf/stable-3.10 [software/gerrit] (wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1350527 (https://phabricator.wikimedia.org/T437620) [11:18:42] (03CR) 10Hashar: "rebased after merge of I4449a47f114ca4d9de50a0fef129c46b990920f3 (remove image-diff plugin)" [software/gerrit] (wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1350527 (https://phabricator.wikimedia.org/T437620) (owner: 10Hashar) [11:24:22] (03CR) 10CI reject: [V:04-1] Merge tag 'stable-3.10-integration-issues-538883789-539032891-540877705-541287630-545820925' into wmf/stable-3.10 [software/gerrit] (wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1350527 (https://phabricator.wikimedia.org/T437620) (owner: 10Hashar) [11:32:58] (03CR) 10Hashar: "recheck com.googlesource.gerrit.plugins.its.base.validation.ItsValidateCommentTest is flappy" [software/gerrit] (wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1350527 (https://phabricator.wikimedia.org/T437620) (owner: 10Hashar) [11:48:25] (03PS8) 10Hashar: Merge tag 'stable-3.10-integration-issues-538883789-539032891-540877705-541287630-545820925' into wmf/stable-3.10 [software/gerrit] (wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1350527 (https://phabricator.wikimedia.org/T437620) [12:15:38] (03PS1) 10Hashar: jjb: [gerrit] drop its-base.jar from all_plugins.zip [integration/config] - 10https://gerrit.wikimedia.org/r/1354439 [12:17:30] (03CR) 10Hashar: [C:03+2] "Done and doc updated at https://wikitech.wikimedia.org/w/index.php?title=Gerrit/Upgrade&diff=prev&oldid=2466432" [integration/config] - 10https://gerrit.wikimedia.org/r/1354439 (owner: 10Hashar) [12:18:00] 10Phabricator (Upstream), 07Upstream: Make it more obvious when attempting to add both a parent and subproject - https://phabricator.wikimedia.org/T416231#12421500 (10Aklapper) 05Open→03Declined I don't think this is feasible to implement, given the current architecture of Phorge. If you have strong op... [12:18:45] (03Merged) 10jenkins-bot: jjb: [gerrit] drop its-base.jar from all_plugins.zip [integration/config] - 10https://gerrit.wikimedia.org/r/1354439 (owner: 10Hashar) [12:37:16] GitLab needs a short maintenance break at 13:00 UTC [12:53:18] Hello, I think Beta cluster code updates seem stuck since ~05:18–07:35 UTC today. The VisualEditor revert https://gerrit.wikimedia.org/r/1354379 (merged 09:29) still isn't deployed. [12:53:19] Result: beta Wikidata Main Page and Wikidata:Administrators and Wikidata:Main_Page give an internal error: "Call to private method VisualEditor\Hooks::getPreferredEditor()" (from https://gerrit.wikimedia.org/r/1342335). Special:Version shows VE at 1c6e0f6f, master is 6c339dc6. [12:53:19] Could someone check the beta auto-update/scap on deployment-deploy04/06? Maybe related to the T435393 changes (not sure on that). [12:53:20] Thanks a lot! [12:53:20] T435393: Beta Cluster: Switch to PHP 8.5, includes writing Puppet changes to support PHP 8.5 on baremetal - https://phabricator.wikimedia.org/T435393 [13:10:16] GitLab maintenance done [14:05:02] (03PS1) 10Hashar: jjb: [gerrit] unshallow clone and dump version labels [integration/config] - 10https://gerrit.wikimedia.org/r/1354461 (https://phabricator.wikimedia.org/T437837) [14:09:51] (03CR) 10Hashar: [C:03+2] "`" [integration/config] - 10https://gerrit.wikimedia.org/r/1354461 (https://phabricator.wikimedia.org/T437837) (owner: 10Hashar) [14:11:00] (03Merged) 10jenkins-bot: jjb: [gerrit] unshallow clone and dump version labels [integration/config] - 10https://gerrit.wikimedia.org/r/1354461 (https://phabricator.wikimedia.org/T437837) (owner: 10Hashar) [14:11:02] (03CR) 10Hashar: "recheck after I3262bf245d1ef53b26d48d408a3c969a40f4e75f which should get us proper versions." [software/gerrit] (wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1350527 (https://phabricator.wikimedia.org/T437620) (owner: 10Hashar) [14:26:29] 10Gerrit (Gerrit 3.15): Upgrade to Gerrit 3.15 - https://phabricator.wikimedia.org/T440883 (10hashar) 03NEW [14:27:03] 10Gerrit (Gerrit 3.15): Upgrade to Gerrit 3.15 - https://phabricator.wikimedia.org/T440883#12422043 (10hashar) [14:27:33] 10Gerrit (Gerrit 3.14): Upgrade to Gerrit 3.14 - https://phabricator.wikimedia.org/T440112#12422044 (10hashar) [14:28:24] 10Gerrit (Gerrit 3.15): Upgrade to Gerrit 3.15 - https://phabricator.wikimedia.org/T440883#12422050 (10hashar) [14:42:15] (03CR) 10Daimona Eaytoy: "Merci :)" [integration/config] - 10https://gerrit.wikimedia.org/r/1354251 (https://phabricator.wikimedia.org/T420009) (owner: 10Daimona Eaytoy) [14:44:51] 06Release-Engineering-Team (Doing 😎), 10MediaWiki-Change-tagging, 10QS-Test-Automation, 10Catalyst (Luka Ijo Pimeja Jan), and 2 others: Catalyst the first edit using each tag is saved without its tag - https://phabricator.wikimedia.org/T439780#12422115 (10jnuche) I could reproduce the issue locally, `innod... [14:45:49] 10Beta-Cluster-Infrastructure, 10Wikidata, 06Wikidata-Omega: Beta cluster isn't updates stuck - https://phabricator.wikimedia.org/T440884 (10mahmoud.abdelsattar.wmde) 03NEW [14:48:24] (03PS1) 10Hashar: Gerrit v3.10.9-79-g5500b68896 and rebuild plugins [software/gerrit] (deploy/wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1354468 (https://phabricator.wikimedia.org/T437620) [14:48:52] 10Beta-Cluster-Infrastructure, 10Wikidata, 06Wikidata-Omega: Beta Cluster code is no longer auto-updating - https://phabricator.wikimedia.org/T440884#12422153 (10mahmoud.abdelsattar.wmde) [14:52:48] 10Beta-Cluster-Infrastructure, 10Wikidata, 06Wikidata-Omega: Beta Cluster code is no longer auto-updating - https://phabricator.wikimedia.org/T440884#12422168 (10awight) The [[https://beta-update.wmcloud.org/202610121450.log|status page]] is a whole lot of this: > [14:50:07] socket.gaierror: [Errno -2] Name... [15:15:10] (03CR) 10Hashar: "I have uploaded the build artifacts at I179d8dd16378bfd73396de90e3ec06918d4fbede . Will verify them and I guess re-deploy Gerrit" [software/gerrit] (wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1350527 (https://phabricator.wikimedia.org/T437620) (owner: 10Hashar) [15:21:15] (03PS2) 10Hashar: Gerrit v3.10.9-79-g5500b68896 and rebuild plugins [software/gerrit] (deploy/wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1354468 (https://phabricator.wikimedia.org/T437620) [16:47:46] 10Beta-Cluster-Infrastructure: Unable to confirm email address on beta cluster - https://phabricator.wikimedia.org/T285527#12422623 (10Nux) I think e-mails might be broken from beta. I'm not getting verification e-mails and I think it used to work (I was able to log in to beta). I think the account is on wp.pl (... [17:59:26] 10Continuous-Integration-Config: Add Mr. Starfleet Command to Zuul allow list - https://phabricator.wikimedia.org/T436312#12422811 (10SD0001) 05Open→03Resolved a:03SD0001 [19:08:53] (03PS3) 10Hashar: wm-pcc: switch commands from experimental to new puppet [software/gerrit] (deploy/wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1244816 (https://phabricator.wikimedia.org/T414621) (owner: 10Jforrester) [19:10:12] (03CR) 10Hashar: [C:03+1] "Amended to keep recognizing old changes made via `check experimental` (aka anything before If41efbb8e8dd19e8eb4ebf74d6ab42905d1f0dda has b" [software/gerrit] (deploy/wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1244816 (https://phabricator.wikimedia.org/T414621) (owner: 10Jforrester) [19:12:04] (03PS5) 10Hashar: Zuul: [operations/puppet] Drop experimental pipeline use [integration/config] - 10https://gerrit.wikimedia.org/r/1234515 (https://phabricator.wikimedia.org/T414621) (owner: 10Jforrester) [19:12:10] (03CR) 10Hashar: [C:03+1] Zuul: [operations/puppet] Drop experimental pipeline use [integration/config] - 10https://gerrit.wikimedia.org/r/1234515 (https://phabricator.wikimedia.org/T414621) (owner: 10Jforrester) [19:14:02] (03CR) 10Jforrester: "Will we drop the old code in future?" [software/gerrit] (deploy/wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1244816 (https://phabricator.wikimedia.org/T414621) (owner: 10Jforrester) [19:23:08] !log Remove /var/log/wmf-beta-update-all/syslog.log on deployment-deploy04 to resolve disk space issue, unbreaks deployments - T440884 [19:23:11] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:23:11] T440884: Beta Cluster code is no longer auto-updating - https://phabricator.wikimedia.org/T440884 [19:24:29] !log Re-apply https://gerrit.wikimedia.org/r/c/operations/puppet/+/1338305 cherry-pick on Puppetserver - T440884 [19:24:32] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:36:09] 10Beta-Cluster-Infrastructure, 10observability, 07Tracking-Neverending: Setup monitoring for Beta Cluster (tracking) - https://phabricator.wikimedia.org/T53497#12422956 (10Southparkfan) Not sure if a more-specific task exists, but monitoring the deployment server would have been useful: T440884. [19:36:56] 10Beta-Cluster-Infrastructure: Failed to update Puppet repository /srv/git/operations/puppet on instance deployment-puppetserver-1 in project deployment-prep - https://phabricator.wikimedia.org/T439024#12422960 (10Southparkfan) 05Open→03Invalid Works fine? [19:37:18] 10Beta-Cluster-Infrastructure: Project deployment-prep instance deployment-eventstreams-2 is down - https://phabricator.wikimedia.org/T439989#12422963 (10Southparkfan) 05Open→03Resolved a:03Southparkfan Resolved as part of T440085. [19:37:19] 10Beta-Cluster-Infrastructure: Project deployment-prep instance deployment-schema-3 is down - https://phabricator.wikimedia.org/T440012#12422969 (10Southparkfan) 05Open→03Resolved a:03Southparkfan Resolved as part of T440085. [19:37:48] 10Beta-Cluster-Infrastructure: Puppet agent disabled on instance deployment-mwmaint03 in project deployment-prep - https://phabricator.wikimedia.org/T438787#12422975 (10Southparkfan) 05Open→03Resolved a:03Southparkfan Host no longer exists: T413223. [19:44:18] 10Beta-Cluster-Infrastructure, 10Wikidata, 06Wikidata-Omega: Beta Cluster code is no longer auto-updating - https://phabricator.wikimedia.org/T440884#12422985 (10Southparkfan) 05Open→03Resolved p:05Triage→03High a:03Southparkfan >>! In T440884#12422168, @awight wrote: > The [[https://beta-updat... [19:44:23] ^ Beta Cluster deployments have been unblocked. [20:05:24] (03CR) 10Hashar: [C:04-1] "For DonationInterface there is already a job for that in `jjb/wm-fundraising.yaml`. Some similar logic was done for cldr/FundraisingEmail" [integration/config] - 10https://gerrit.wikimedia.org/r/1353588 (https://phabricator.wikimedia.org/T438502) (owner: 10Clare Ming) [21:46:16] PROBLEM - jenkins_service_running on contint1003 is CRITICAL: PROCS CRITICAL: 2 processes with regex args .*/bin/java .*-jar /usr/share/java/jenkins.war https://wikitech.wikimedia.org/wiki/Jenkins [21:47:15] RECOVERY - jenkins_service_running on contint1003 is OK: PROCS OK: 1 process with regex args .*/bin/java .*-jar /usr/share/java/jenkins.war https://wikitech.wikimedia.org/wiki/Jenkins