[00:09:54] 10Phabricator, 10Release-Engineering-Team, 10User-brennen: Phabricator loading outdated JavaScript - https://phabricator.wikimedia.org/T310850 (10brennen) Check on the mic: {F35266005} [00:15:14] danisztls I don't think you need to schedule it, once it merges to wmf config it'll be updated automatically on the beta cluster every 10 minutes (if I recall correctly) [00:15:46] DannyS712: thanks! [00:17:48] 10Phabricator, 10Release-Engineering-Team, 10User-brennen: Deploy Phabricator release/2022-06-22/1 - https://phabricator.wikimedia.org/T311175 (10brennen) 05Open→03Resolved a:03brennen [00:18:42] 10Phabricator (Upstream), 10Release-Engineering-Team, 10Upstream, 10User-brennen: Uploaded files via the drag-and-drop are defaulting to private-access - https://phabricator.wikimedia.org/T310833 (10brennen) [00:18:48] 10Phabricator, 10Release-Engineering-Team, 10User-brennen: Phabricator loading outdated JavaScript - https://phabricator.wikimedia.org/T310850 (10brennen) 05Open→03Resolved [00:19:03] 10Phabricator (Upstream), 10Release-Engineering-Team, 10Upstream, 10User-brennen: Uploaded files via the drag-and-drop are defaulting to private-access - https://phabricator.wikimedia.org/T310833 (10brennen) 05Open→03Resolved [00:20:25] 10Continuous-Integration-Infrastructure, 10Jenkins, 10Release-Engineering-Team (Doing): Use Jenkins upstream systemd unit instead of our own - https://phabricator.wikimedia.org/T308637 (10thcipriani) [00:20:35] 10Continuous-Integration-Infrastructure, 10Jenkins, 10Release-Engineering-Team (Next): Use Jenkins upstream systemd unit instead of our own - https://phabricator.wikimedia.org/T308637 (10thcipriani) [03:38:24] 10Release-Engineering-Team (Priority Backlog đŸ“Ĩ), 10Patch-For-Review, 10Release, 10Train Deployments: 1.39.0-wmf.17 deployment blockers - https://phabricator.wikimedia.org/T308070 (10brennen) Per discussion, let's go ahead and soft-block the train on {T311119}. It sounds like this should not hold the train... [03:40:11] 10Release-Engineering-Team (Priority Backlog đŸ“Ĩ), 10Patch-For-Review, 10Release, 10Train Deployments: 1.39.0-wmf.17 deployment blockers - https://phabricator.wikimedia.org/T308070 (10brennen) [08:56:04] 10GitLab (Project Migration), 10Release-Engineering-Team: Create new GitLab project group: - https://phabricator.wikimedia.org/T311206 (10M.shaffar19) [08:58:47] 10GitLab (Project Migration), 10Release-Engineering-Team: Create new GitLab project group: - https://phabricator.wikimedia.org/T311207 (10M.shaffar19) [10:42:58] 10Release-Engineering-Team (Radar), 10Scap, 10Patch-For-Review, 10User-jijiki: Update Scap to perform rolling restart for all MW deploy - https://phabricator.wikimedia.org/T266055 (10JMeybohm) [10:43:43] 10Release-Engineering-Team (Radar), 10Scap, 10Patch-For-Review, 10User-jijiki: Update Scap to perform rolling restart for all MW deploy - https://phabricator.wikimedia.org/T266055 (10JMeybohm) The ones left in prod are: `mwdebug[2001-2002].codfw.wmnet,mwdebug[1001-1002].eqiad.wmnet` [11:20:29] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Priority Backlog đŸ“Ĩ), 10Scap, 10Patch-For-Review: Add a ssh key pair for scap on beta cluster - https://phabricator.wikimedia.org/T310354 (10hashar) 05Open→03Resolved [11:20:32] 10Release-Engineering-Team (Deployment Autopilot 🛩ī¸), 10Scap, 10Infrastructure-Foundations, 10serviceops, 10Patch-For-Review: Use scap to deploy itself to scap targets - https://phabricator.wikimedia.org/T303559 (10hashar) [12:19:26] 10Continuous-Integration-Config: Please add mediawiki-i18n-check-docker to repository mediawiki/extensions/SemanticRESTAPI - https://phabricator.wikimedia.org/T311226 (10Nikerabbit) [12:31:27] 10Release-Engineering-Team (Deployment Autopilot 🛩ī¸), 10Scap: Create alerts for automated Tuesday Train timer - https://phabricator.wikimedia.org/T310396 (10hashar) I am pretty sure that is covered by our [[https://doc.wikimedia.org/puppet/puppet_defined_types/systemd_3A_3Atimer_3A_3Ajob.html | Puppet systemd:... [14:05:30] 10Continuous-Integration-Infrastructure, 10Jenkins, 10Release-Engineering-Team (Next): Use Jenkins upstream systemd unit instead of our own - https://phabricator.wikimedia.org/T308637 (10hashar) [14:09:45] 10Continuous-Integration-Infrastructure, 10Jenkins, 10Release-Engineering-Team: Upgrade Jenkins to latest LTS 2.346.1 - https://phabricator.wikimedia.org/T311174 (10hashar) I have upgraded releases1002 releases2002 and contint1001.wikimedia.org contint2001.wikimedia.org is the primary for CI. Will upgrade i... [14:10:16] 10Continuous-Integration-Infrastructure, 10Jenkins, 10Release-Engineering-Team: Upgrade Jenkins to latest LTS 2.346.1 - https://phabricator.wikimedia.org/T311174 (10hashar) a:03hashar [14:15:55] 10Release-Engineering-Team (Priority Backlog đŸ“Ĩ), 10Patch-For-Review, 10Release, 10Train Deployments: 1.39.0-wmf.17 deployment blockers - https://phabricator.wikimedia.org/T308070 (10Jdlrobson) [14:16:59] 10Continuous-Integration-Infrastructure, 10Jenkins, 10Release-Engineering-Team (Next): Use Jenkins upstream systemd unit instead of our own - https://phabricator.wikimedia.org/T308637 (10hashar) [15:05:07] 10GitLab (CI & Job Runners): DNS/networking not working on Trusted Runners - https://phabricator.wikimedia.org/T311241 (10Jelto) [15:19:26] 10Project-Admins: Create component for product infrastructure orphaned project tickets - https://phabricator.wikimedia.org/T311246 (10Jgiannelos) [15:20:19] 10Project-Admins: Create component for product infrastructure orphaned project tickets - https://phabricator.wikimedia.org/T311246 (10Jgiannelos) [15:21:51] 10Continuous-Integration-Infrastructure, 10Jenkins, 10Release-Engineering-Team: Upgrade Jenkins to latest LTS 2.346.1 - https://phabricator.wikimedia.org/T311174 (10hashar) 05Open→03Resolved I have upgraded the CI Jenkins. Success! [15:53:21] (03PS1) 10Majavah: zuul: remove stretch jobs for labs/toollabs [integration/config] - 10https://gerrit.wikimedia.org/r/808021 [15:53:23] (03PS1) 10Majavah: jjb: remove unused debian-glue-stretch* jobs [integration/config] - 10https://gerrit.wikimedia.org/r/808022 [15:56:59] (03CR) 10Majavah: [C: 03+2] zuul: remove stretch jobs for labs/toollabs [integration/config] - 10https://gerrit.wikimedia.org/r/808021 (owner: 10Majavah) [15:59:05] (03Merged) 10jenkins-bot: zuul: remove stretch jobs for labs/toollabs [integration/config] - 10https://gerrit.wikimedia.org/r/808021 (owner: 10Majavah) [15:59:44] !log reload zuul for https://gerrit.wikimedia.org/r/808021 [15:59:45] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:02:39] question about https://www.mediawiki.org/wiki/Continuous_integration/Jenkins_job_builder#Deploy_changes: how do I delete jobs? as far as I can tell, jjb update only works for jobs that still exist on the new version [16:09:27] 10Project-Admins: Please add new tag "Ganeti" - https://phabricator.wikimedia.org/T310683 (10Aklapper) Hi could you please provide a project description? (See the [best practices](https://www.mediawiki.org/wiki/Phabricator/Creating_and_renaming_projects#Good_practices_for_name_and_description) and include a sent... [16:09:28] (03PS1) 10Dduvall: gerrit: Wait up to 600 seconds for Gerrit to restart [tools/train-dev] - 10https://gerrit.wikimedia.org/r/808024 [16:11:41] (03CR) 10Ahmon Dancy: [C: 03+2] gerrit: Wait up to 600 seconds for Gerrit to restart (031 comment) [tools/train-dev] - 10https://gerrit.wikimedia.org/r/808024 (owner: 10Dduvall) [16:12:09] (03Merged) 10jenkins-bot: gerrit: Wait up to 600 seconds for Gerrit to restart [tools/train-dev] - 10https://gerrit.wikimedia.org/r/808024 (owner: 10Dduvall) [16:14:17] 10Project-Admins: Create component for product infrastructure orphaned project tickets - https://phabricator.wikimedia.org/T311246 (10Aklapper) 05Open→03Stalled @Jgiannelos: Hi, in my understanding such open tickets should simply get their #Product-Infrastructure-Team-Backlog and/or #Product_Infrastructure_R... [16:23:04] (03PS1) 10Ahmon Dancy: Extend requirements checks [tools/train-dev] - 10https://gerrit.wikimedia.org/r/808027 [16:25:34] (03CR) 10Dduvall: [C: 03+2] Extend requirements checks [tools/train-dev] - 10https://gerrit.wikimedia.org/r/808027 (owner: 10Ahmon Dancy) [16:26:08] (03Merged) 10jenkins-bot: Extend requirements checks [tools/train-dev] - 10https://gerrit.wikimedia.org/r/808027 (owner: 10Ahmon Dancy) [16:34:14] (03PS1) 10Hashar: cli.Application() fixtures require a clear env [tools/scap] - 10https://gerrit.wikimedia.org/r/808029 [16:34:23] (03CR) 10Hashar: Support a different user for Gerrit ssh interactions (033 comments) [tools/scap] - 10https://gerrit.wikimedia.org/r/805462 (https://phabricator.wikimedia.org/T306425) (owner: 10Hashar) [16:34:32] (03PS10) 10Hashar: Support a different user for Gerrit ssh interactions [tools/scap] - 10https://gerrit.wikimedia.org/r/805462 (https://phabricator.wikimedia.org/T306425) [16:34:46] I swear I had train-dev setup at some point [16:35:05] nod.. you had worked on the mirror-repos.sh script [16:37:20] and somehow my laptop which is from 3 or 4 years ago still has plenty of free and fressh SSD disk space: `Free PE / Size 43539 / 170,07 GiB` [16:37:34] Nice [16:37:41] somehow I got a 512G ssd on this laptop [16:37:52] for 20 euros less than the 256G version [16:38:01] That's a good deal. [16:38:11] yeah definitely [16:38:21] gotta resize my partitions, which I guess is an excellent task for a friday afternoon [16:39:45] anyway, I have iterated on the patch, thanks so much for catching the `ssh -n` and that when using a dedicated user we no more need SSH_AUTH_SOCK [16:40:09] Awesome! I'll run some tests today [17:05:43] (03CR) 10Ahmon Dancy: history: Mark latest history entry as synced following any sync command (032 comments) [tools/scap] - 10https://gerrit.wikimedia.org/r/805496 (https://phabricator.wikimedia.org/T287045) (owner: 10Dduvall) [17:06:46] (03CR) 10Ahmon Dancy: [C: 03+2] cli.Application() fixtures require a clear env [tools/scap] - 10https://gerrit.wikimedia.org/r/808029 (owner: 10Hashar) [17:10:34] (03CR) 10Ahmon Dancy: "Code looks good. I'll do testing today." [tools/scap] - 10https://gerrit.wikimedia.org/r/805462 (https://phabricator.wikimedia.org/T306425) (owner: 10Hashar) [17:11:14] (03Merged) 10jenkins-bot: cli.Application() fixtures require a clear env [tools/scap] - 10https://gerrit.wikimedia.org/r/808029 (owner: 10Hashar) [17:26:13] aaaah Jenkins got updated? [17:26:25] * TheresNoTime did not just do a double-take at the new icons [17:37:42] Yep. [17:46:54] 10Project-Admins: Please add new tag "Ganeti" - https://phabricator.wikimedia.org/T310683 (10MoritzMuehlenhoff) "Wikimedia's installations of Ganeti, a clustered virtual machine management stack" should do, along with a link to https://wikitech.wikimedia.org/wiki/Ganeti [17:59:23] (03CR) 10Hashar: [C: 03+2] "I crafted them specially for labs/toollabs and they are no more triggering since https://gerrit.wikimedia.org/r/c/integration/config/+/808" [integration/config] - 10https://gerrit.wikimedia.org/r/808022 (owner: 10Majavah) [18:00:01] hashar: repeating my question above: how do I delete jenkins jobs? as far as I can tell, jjb update only works for jobs that still exist on the new version [18:00:48] taavi: There is a `jjb-delete` script [18:00:59] (in integration/config) [18:01:06] ah! [18:01:09] * taavi documents that [18:01:21] (03Merged) 10jenkins-bot: jjb: remove unused debian-glue-stretch* jobs [integration/config] - 10https://gerrit.wikimedia.org/r/808022 (owner: 10Majavah) [18:02:58] https://www.mediawiki.org/w/index.php?title=Continuous_integration%2FJenkins_job_builder&type=revision&diff=5301914&oldid=4949257 [18:03:29] s/might not/will not/ ? [18:04:11] taavi: dancy: I have been manually deleting jobs through the web ui for a decade or so which is not ideal for sure [18:04:19] or let them pill up and would do a yearly cleanup [18:05:09] else yes `./jjb-delete` is a wrapper around tox which would invoke `jenkins-jobs delete [name ...]` [18:05:15] where name are the jobs you want to delete [18:22:30] i added a paragraph to the top of https://wikitech.wikimedia.org/wiki/Deployments/Holding_the_train clarifying non-emergency train blockers a bit. [18:22:50] i think that captures our actual practice, but feedback is of course welcome. [18:37:37] the new second paragraph sounds accurate [21:12:49] I like the new text. [21:26:42] 10Release-Engineering-Team, 10Scap, 10Patch-For-Review: scap-o-scap: Bootstrapping a new host fails - https://phabricator.wikimedia.org/T310740 (10dancy) 05Open→03Stalled p:05Triage→03Low Lowering the priority of this ticket and keeping open for a bit until we have verification that new nodes are boo... [21:26:48] 10Release-Engineering-Team (Deployment Autopilot 🛩ī¸), 10Scap, 10Infrastructure-Foundations, 10serviceops, 10Patch-For-Review: Use scap to deploy itself to scap targets - https://phabricator.wikimedia.org/T303559 (10dancy) [21:51:11] (03CR) 10Ahmon Dancy: [C: 03+2] "I need to get moving along so self-merging." [tools/train-dev] - 10https://gerrit.wikimedia.org/r/807615 (owner: 10Ahmon Dancy) [21:52:09] (03Merged) 10jenkins-bot: Set up keyholder-like ssh-agent for debian user [tools/train-dev] - 10https://gerrit.wikimedia.org/r/807615 (owner: 10Ahmon Dancy) [21:52:33] (03PS1) 10Ahmon Dancy: Add trainbranchbot Gerrit account [tools/train-dev] - 10https://gerrit.wikimedia.org/r/808085 [21:56:18] (03PS2) 10Ahmon Dancy: Add trainbranchbot Gerrit account [tools/train-dev] - 10https://gerrit.wikimedia.org/r/808085 [22:01:47] (03CR) 10Ahmon Dancy: "Worked mostly ok in train-dev with https://gerrit.wikimedia.org/r/c/mediawiki/tools/train-dev/+/808085 applied." [tools/scap] - 10https://gerrit.wikimedia.org/r/805462 (https://phabricator.wikimedia.org/T306425) (owner: 10Hashar) [22:04:39] (03CR) 10Ahmon Dancy: Support a different user for Gerrit ssh interactions (031 comment) [tools/scap] - 10https://gerrit.wikimedia.org/r/805462 (https://phabricator.wikimedia.org/T306425) (owner: 10Hashar) [22:11:20] (03PS1) 10Dduvall: rollback: Provide rollback subcommand for quick history replay and sync [tools/scap] - 10https://gerrit.wikimedia.org/r/808086 (https://phabricator.wikimedia.org/T287045) [22:11:54] (03PS1) 10Ahmon Dancy: get_keyholder_key: Return private key path instead of public key [tools/scap] - 10https://gerrit.wikimedia.org/r/808087 [22:12:27] (03PS2) 10Dduvall: rollback: Provide rollback subcommand for quick history replay and sync [tools/scap] - 10https://gerrit.wikimedia.org/r/808086 (https://phabricator.wikimedia.org/T287045) [22:12:55] (03CR) 10Ahmon Dancy: [C: 03+1] "I tested this in train-dev by running scap sync-wikiversions" [tools/scap] - 10https://gerrit.wikimedia.org/r/808087 (owner: 10Ahmon Dancy) [22:15:20] (03CR) 10CI reject: [V: 04-1] get_keyholder_key: Return private key path instead of public key [tools/scap] - 10https://gerrit.wikimedia.org/r/808087 (owner: 10Ahmon Dancy) [22:24:33] (03PS4) 10Dduvall: history: Mark latest history entry as synced following any sync command [tools/scap] - 10https://gerrit.wikimedia.org/r/805496 (https://phabricator.wikimedia.org/T287045) [22:24:35] (03PS3) 10Dduvall: rollback: Provide rollback subcommand for quick history replay and sync [tools/scap] - 10https://gerrit.wikimedia.org/r/808086 (https://phabricator.wikimedia.org/T287045) [22:25:58] (03CR) 10Jeena Huneidi: "This change is ready for review." [tools/scap] - 10https://gerrit.wikimedia.org/r/806352 (https://phabricator.wikimedia.org/T287046) (owner: 10Jeena Huneidi) [22:26:41] (03PS5) 10Dduvall: history: Mark latest history entry as synced following any sync command [tools/scap] - 10https://gerrit.wikimedia.org/r/805496 (https://phabricator.wikimedia.org/T287045) [22:26:43] (03PS4) 10Dduvall: rollback: Provide rollback subcommand for quick history replay and sync [tools/scap] - 10https://gerrit.wikimedia.org/r/808086 (https://phabricator.wikimedia.org/T287045) [22:27:40] (03PS2) 10Ahmon Dancy: get_keyholder_key: Return private key path instead of public key [tools/scap] - 10https://gerrit.wikimedia.org/r/808087 [22:29:57] (03CR) 10Dduvall: history: Mark latest history entry as synced following any sync command (032 comments) [tools/scap] - 10https://gerrit.wikimedia.org/r/805496 (https://phabricator.wikimedia.org/T287045) (owner: 10Dduvall) [22:31:40] (03CR) 10Ahmon Dancy: get_keyholder_key: Return private key path instead of public key (031 comment) [tools/scap] - 10https://gerrit.wikimedia.org/r/808087 (owner: 10Ahmon Dancy) [22:33:51] (03CR) 10CI reject: [V: 04-1] rollback: Provide rollback subcommand for quick history replay and sync [tools/scap] - 10https://gerrit.wikimedia.org/r/808086 (https://phabricator.wikimedia.org/T287045) (owner: 10Dduvall) [22:34:02] (03CR) 10CI reject: [V: 04-1] history: Mark latest history entry as synced following any sync command [tools/scap] - 10https://gerrit.wikimedia.org/r/805496 (https://phabricator.wikimedia.org/T287045) (owner: 10Dduvall) [22:40:25] 10Release-Engineering-Team (Priority Backlog đŸ“Ĩ), 10Patch-For-Review, 10Release, 10Train Deployments: 1.39.0-wmf.17 deployment blockers - https://phabricator.wikimedia.org/T308070 (10brennen) 05Open→03Resolved Currently stable on all wikis. [22:42:40] (03PS6) 10Dduvall: history: Mark latest history entry as synced following any sync command [tools/scap] - 10https://gerrit.wikimedia.org/r/805496 (https://phabricator.wikimedia.org/T287045) [22:42:42] (03PS5) 10Dduvall: rollback: Provide rollback subcommand for quick history replay and sync [tools/scap] - 10https://gerrit.wikimedia.org/r/808086 (https://phabricator.wikimedia.org/T287045) [22:50:19] (03CR) 10Thcipriani: [C: 03+1] "Long explanation inline for why this is the way it is. If it works now: it's probably fine." [tools/scap] - 10https://gerrit.wikimedia.org/r/808087 (owner: 10Ahmon Dancy) [22:58:51] (03PS23) 10Jeena Huneidi: scap backport: add revert option [tools/scap] - 10https://gerrit.wikimedia.org/r/806352 (https://phabricator.wikimedia.org/T287046) [23:30:27] (03CR) 10DannyS712: Add trainbranchbot Gerrit account (032 comments) [tools/train-dev] - 10https://gerrit.wikimedia.org/r/808085 (owner: 10Ahmon Dancy)