[01:20:03] Project beta-update-databases-eqiad build #61615: 04FAILURE in 3.2 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/61615/ [01:26:42] ^ me [01:54:30] Project beta-update-databases-eqiad build #61616: 04STILL FAILING in 2.9 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/61616/ [02:04:13] Yippee, build fixed! [02:04:14] Project beta-update-databases-eqiad build #61617: 09FIXED in 5 min 1 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/61617/ [02:06:55] !log created backup of all databases on deployment-db08 # T318126 [02:06:59] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [02:06:59] T318126: Migrate deployment-prep db hosts to bullseye - https://phabricator.wikimedia.org/T318126 [03:18:20] 10Project-Admins: Migrate mwbot-rs tasks to Phabricator - https://phabricator.wikimedia.org/T318069 (10Legoktm) >>! In T318069#8245043, @Aklapper wrote: > If the subprojects are disjunct this could be a parent project `mwbot-rs` with numerous subprojects. They're not currently disjunct, but I reviewed the exist... [05:56:14] 10Continuous-Integration-Infrastructure, 10Jenkins, 10Release-Engineering-Team (Radar), 10Upstream: Jobs started failing on https://releases-jenkins.wikimedia.org on 2022-08-21 - https://phabricator.wikimedia.org/T315897 (10hashar) Upstream has released a new version 4.12.1 https://github.com/jenkinsci/git... [08:08:48] !log Upgrading CI and releases Jenkins plugins notably to update the git client T315897 [08:08:50] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [08:08:50] T315897: Jobs started failing on https://releases-jenkins.wikimedia.org on 2022-08-21 - https://phabricator.wikimedia.org/T315897 [08:39:10] 10Continuous-Integration-Infrastructure, 10Jenkins, 10Release-Engineering-Team (Radar), 10Upstream: Jobs started failing on https://releases-jenkins.wikimedia.org on 2022-08-21 - https://phabricator.wikimedia.org/T315897 (10hashar) 05Stalled→03Resolved I have upgraded the git plugin on both our Jenkins. [10:20:07] 10Beta-Cluster-Infrastructure: prometheus-beta.wmcloud.org 504 timeout - https://phabricator.wikimedia.org/T315699 (10TheresNoTime) Apologies for the uninvited subscribe @taavi, but you're very knowledgable on these topics — any suggestions or thoughts would be greatly appreciated. No rush of course [10:20:32] 10Beta-Cluster-Infrastructure: prometheus-beta.wmcloud.org 504 timeout - https://phabricator.wikimedia.org/T315699 (10TheresNoTime) 05Open→03Stalled [12:17:48] (03CR) 10Jaime Nuche: Replace Lock with TimeoutLock (033 comments) [tools/scap] - 10https://gerrit.wikimedia.org/r/828075 (https://phabricator.wikimedia.org/T315531) (owner: 10Jeena Huneidi) [12:52:53] (03CR) 10Hashar: [C: 04-1] "Thanks Dan that is an excellent suggestion. I will look at rewriting this change to use before/after instead ;)" [tools/scap] - 10https://gerrit.wikimedia.org/r/831152 (owner: 10Hashar) [12:55:57] (03CR) 10Hashar: [C: 03+2] "Looks like that is working ;)" [software/gerrit] (deploy/wmf/stable-3.4) - 10https://gerrit.wikimedia.org/r/832518 (https://phabricator.wikimedia.org/T317412) (owner: 10Hashar) [12:56:26] (03Merged) 10jenkins-bot: Use gerrit-deploy for deployment on devtools [software/gerrit] (deploy/wmf/stable-3.4) - 10https://gerrit.wikimedia.org/r/832518 (https://phabricator.wikimedia.org/T317412) (owner: 10Hashar) [13:01:17] (03PS3) 10Hashar: Gerrit v3.5.2 and rebuild plugins [software/gerrit] (deploy/wmf/stable-3.5) - 10https://gerrit.wikimedia.org/r/824200 (https://phabricator.wikimedia.org/T307334) [13:15:47] (03CR) 10Jaime Nuche: [C: 03+2] "Verified in traindev" [tools/scap] - 10https://gerrit.wikimedia.org/r/832373 (owner: 10Ahmon Dancy) [13:21:54] (03Merged) 10jenkins-bot: ProgressReporter: Prefix progress indicator with a timestamp [tools/scap] - 10https://gerrit.wikimedia.org/r/832373 (owner: 10Ahmon Dancy) [13:22:37] 10Phabricator: https://phab.wmflabs.org/ Is down - https://phabricator.wikimedia.org/T318158 (10Devnull) [13:24:03] 10Phabricator, 10SRE: https://phab.wmflabs.org/ Is down - https://phabricator.wikimedia.org/T318158 (10Devnull) [13:33:03] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team: Zuul: Interpret extension dependencies as mutual - https://phabricator.wikimedia.org/T318107 (10Jdforrester-WMF) >>! In T318107#8246583, @daniel wrote: >>>>! In T318107#8246434, @daniel wrote: >>> Point in c... [14:13:11] 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-Phonos, 10Community-Tech (CommTech-Sprint-33), 10MW-1.40-notes (1.40.0-wmf.2; 2022-09-19), 10Patch-For-Review: Phonos links to unroutable domain/URL for the MP3 file - https://phabricator.wikimedia.org/T317417 (10JMcLeod_WMF) [15:29:34] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Priority Backlog đŸ“Ĩ), 10Patch-For-Review, 10User-brennen: Deploy buildkitd to trusted GitLab runners - https://phabricator.wikimedia.org/T308271 (10Dzahn) [15:29:44] 10GitLab (CI & Job Runners), 10serviceops, 10serviceops-collab: DNS/networking not working on Trusted Runners - https://phabricator.wikimedia.org/T311241 (10Dzahn) 05Open→03Resolved DNS is actually workig now. We will follow-up with a separate task to implement firewall rules and talk about egress. [15:30:40] 10GitLab (CI & Job Runners), 10serviceops, 10serviceops-collab: DNS/networking not working on Trusted Runners - https://phabricator.wikimedia.org/T311241 (10Dzahn) [15:34:00] 10GitLab (CI & Job Runners), 10Release-Engineering-Team, 10serviceops, 10serviceops-collab, and 2 others: Changes to modules/gitlab_runner/templates/config-template.toml.erb have no effect on existing runners - https://phabricator.wikimedia.org/T311746 (10LSobanski) a:05dduvall→03Jelto [15:37:08] 10GitLab (CI & Job Runners), 10serviceops, 10serviceops-collab: Standardize Debian package builds on GitLab CI - https://phabricator.wikimedia.org/T304491 (10Jelto) [15:37:41] 10GitLab (Infrastructure), 10serviceops, 10serviceops-collab: Create GitLab mirror - https://phabricator.wikimedia.org/T291322 (10Jelto) a:05Jelto→03None [15:37:53] 10Phabricator, 10Release-Engineering-Team (Bonus Level 🕹ī¸), 10DBA, 10serviceops, and 2 others: sort out mysql privileges for phab1004/phab2002 - https://phabricator.wikimedia.org/T315713 (10Dzahn) a:03Dzahn [15:50:36] 10Phabricator, 10Trust-and-Safety: Reset 2FA for Anoop - https://phabricator.wikimedia.org/T318172 (10Tgr) [15:52:16] 10GitLab (Infrastructure), 10serviceops, 10serviceops-collab: Automate setup of GitLab test instance - https://phabricator.wikimedia.org/T302976 (10Jelto) [15:52:45] 10GitLab (Infrastructure), 10serviceops, 10serviceops-collab, 10Patch-For-Review: Migrate gitlab-test instance to puppet - https://phabricator.wikimedia.org/T297411 (10Jelto) [15:52:58] 10GitLab (Infrastructure), 10serviceops, 10serviceops-collab: Reduce usage of public IPv4 addresses on GitLab hosts - https://phabricator.wikimedia.org/T310265 (10Jelto) [15:54:09] 10GitLab (Project Migration), 10Release-Engineering-Team, 10serviceops, 10serviceops-collab: Create new GitLab project group: wm-juniors-il - https://phabricator.wikimedia.org/T313750 (10Dzahn) [15:54:24] (03CR) 10Jaime Nuche: Replace Lock with TimeoutLock (032 comments) [tools/scap] - 10https://gerrit.wikimedia.org/r/828075 (https://phabricator.wikimedia.org/T315531) (owner: 10Jeena Huneidi) [15:55:59] 10GitLab (Project Migration), 10Phabricator, 10Release-Engineering-Team (Bonus Level 🕹ī¸), 10serviceops, and 4 others: Migrate active repositories in Phabricator Differential to GitLab - https://phabricator.wikimedia.org/T191182 (10Dzahn) [16:09:47] 10Phabricator, 10Trust-and-Safety: Reset Phabricator 2FA for Anoop - https://phabricator.wikimedia.org/T318172 (10Tgr) [16:57:18] 10Release-Engineering-Team, 10Scap: Perform mediawiki k8s deployments in parallel - https://phabricator.wikimedia.org/T318181 (10dancy) [16:58:24] 10Release-Engineering-Team, 10Scap: Perform mediawiki k8s deployments in parallel - https://phabricator.wikimedia.org/T318181 (10dancy) [16:58:29] 10Release-Engineering-Team (Bonus Level 🕹ī¸), 10MW-on-K8s, 10serviceops, 10Patch-For-Review: Make scap deploy to kubernetes together with the legacy systems - https://phabricator.wikimedia.org/T299648 (10dancy) [17:00:24] (03CR) 10Ahmon Dancy: "I let the existing rollback code in place since it felt silly to undo work that I spent a couple of hours on." [tools/scap] - 10https://gerrit.wikimedia.org/r/832338 (https://phabricator.wikimedia.org/T318181) (owner: 10Ahmon Dancy) [17:07:28] 10Release-Engineering-Team (Radar), 10Scap: Catalog of errors from scap (or subprocesses) related to php-fpm restart always being enabled - https://phabricator.wikimedia.org/T311181 (10dancy) 05Open→03Resolved a:03dancy php-fpm restarts have been running smoothly lately so closing this ticket. [17:33:03] 10Release-Engineering-Team (Priority Backlog đŸ“Ĩ), 10Scap: scap backport sanity check prevents using it to fix an about-to-be-live branch - https://phabricator.wikimedia.org/T317795 (10dancy) >>! In T317795#8237322, @jeena wrote: > what about having it do everything except sync if it's not deployed anywhere? Ye... [17:34:48] 10Release-Engineering-Team (Seen), 10Scap: Make "scap backport" skip syncing steps for labs-only changes - https://phabricator.wikimedia.org/T317242 (10dancy) @Zabe Is LabsServices.php the only thing that would need to be handled? [17:38:11] 10Release-Engineering-Team (Seen), 10Scap: Make "scap backport" skip syncing steps for labs-only changes - https://phabricator.wikimedia.org/T317242 (10Zabe) >>! In T317242#8248763, @dancy wrote: > @Zabe Is LabsServices.php the only thing that would need to be handled? No, here a full list. ` wmf-config/Cirr... [18:50:49] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team: Zuul: Interpret extension dependencies as mutual - https://phabricator.wikimedia.org/T318107 (10daniel) @Jdforrester-WMF My understanding is that when we add a new extension A that depends on existing extens... [20:04:49] !log rsynced dump from deployment-db08 to deployment-db09 # T318126 [20:04:51] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:04:51] T318126: Migrate deployment-prep db hosts to bullseye - https://phabricator.wikimedia.org/T318126 [20:06:58] !log deployment-db09: import dump into mariadb # T318126 [20:07:00] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:18:51] (03CR) 10Ahmon Dancy: Replace Lock with TimeoutLock (031 comment) [tools/scap] - 10https://gerrit.wikimedia.org/r/828075 (https://phabricator.wikimedia.org/T315531) (owner: 10Jeena Huneidi) [20:55:26] (03PS4) 10Hashar: Support checks running before a stage [tools/scap] - 10https://gerrit.wikimedia.org/r/831152 [20:57:22] 10Phabricator: H408 doesn't seem to work with milestone board - https://phabricator.wikimedia.org/T318015 (10ldelench_wmf) 05Open→03Resolved Looks like it's working as expected now; thank you so much @Aklapper ! [20:59:32] (03CR) 10CI reject: [V: 04-1] Support checks running before a stage [tools/scap] - 10https://gerrit.wikimedia.org/r/831152 (owner: 10Hashar) [21:14:14] (03PS1) 10Ahmon Dancy: Make "scap backport" skip sync for labs-only changes [tools/scap] - 10https://gerrit.wikimedia.org/r/833456 (https://phabricator.wikimedia.org/T317242) [21:19:33] (03PS2) 10Ahmon Dancy: Make "scap backport" skip sync for labs-only changes [tools/scap] - 10https://gerrit.wikimedia.org/r/833456 (https://phabricator.wikimedia.org/T317242) [21:19:35] (03PS14) 10Jeena Huneidi: Replace Lock with TimeoutLock [tools/scap] - 10https://gerrit.wikimedia.org/r/828075 (https://phabricator.wikimedia.org/T315531) [21:21:14] (03CR) 10Hashar: "CI fails with under Stretch / Python 3.5 with:" [tools/scap] - 10https://gerrit.wikimedia.org/r/831152 (owner: 10Hashar) [21:22:24] (03CR) 10CI reject: [V: 04-1] Replace Lock with TimeoutLock [tools/scap] - 10https://gerrit.wikimedia.org/r/828075 (https://phabricator.wikimedia.org/T315531) (owner: 10Jeena Huneidi) [21:28:21] (03CR) 10Ahmon Dancy: Replace Lock with TimeoutLock (032 comments) [tools/scap] - 10https://gerrit.wikimedia.org/r/828075 (https://phabricator.wikimedia.org/T315531) (owner: 10Jeena Huneidi) [22:00:06] !log deployment-db09: start replication # T318126 [22:00:08] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [22:00:09] T318126: Migrate deployment-prep db hosts to bullseye - https://phabricator.wikimedia.org/T318126 [22:15:14] (03CR) 10Thcipriani: [C: 03+1] "Concept looks good to me!" [tools/scap] - 10https://gerrit.wikimedia.org/r/833456 (https://phabricator.wikimedia.org/T317242) (owner: 10Ahmon Dancy) [22:38:48] (03PS15) 10Jeena Huneidi: Replace Lock with TimeoutLock [tools/scap] - 10https://gerrit.wikimedia.org/r/828075 (https://phabricator.wikimedia.org/T315531) [22:39:15] (03CR) 10Jeena Huneidi: Replace Lock with TimeoutLock (035 comments) [tools/scap] - 10https://gerrit.wikimedia.org/r/828075 (https://phabricator.wikimedia.org/T315531) (owner: 10Jeena Huneidi) [22:40:43] (03PS16) 10Jeena Huneidi: Replace Lock with TimeoutLock [tools/scap] - 10https://gerrit.wikimedia.org/r/828075 (https://phabricator.wikimedia.org/T315531) [22:40:55] (03CR) 10Jeena Huneidi: Replace Lock with TimeoutLock (031 comment) [tools/scap] - 10https://gerrit.wikimedia.org/r/828075 (https://phabricator.wikimedia.org/T315531) (owner: 10Jeena Huneidi) [22:42:42] (03PS17) 10Jeena Huneidi: Replace Lock with TimeoutLock [tools/scap] - 10https://gerrit.wikimedia.org/r/828075 (https://phabricator.wikimedia.org/T315531) [23:32:53] (03PS1) 10Ahmon Dancy: Remove legacy sync.flag check [tools/scap] - 10https://gerrit.wikimedia.org/r/833464 [23:38:23] (03CR) 10Ahmon Dancy: Replace Lock with TimeoutLock (032 comments) [tools/scap] - 10https://gerrit.wikimedia.org/r/828075 (https://phabricator.wikimedia.org/T315531) (owner: 10Jeena Huneidi) [23:39:10] (03CR) 10Ahmon Dancy: [C: 03+1] "The code looks good. I'll hand test tomorrow." [tools/scap] - 10https://gerrit.wikimedia.org/r/828075 (https://phabricator.wikimedia.org/T315531) (owner: 10Jeena Huneidi)