[00:55:31] 10Release-Engineering-Team (Doing), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.38.0-wmf.12 deployment blockers - https://phabricator.wikimedia.org/T293953 (10brennen) [02:05:56] 10Deployments, 10bacula, 10Sustainability (Incident Followup): Local private files on deployment host should be backed up somewhere - https://phabricator.wikimedia.org/T69818 (10Krinkle) a:05jcrespo→03Krinkle [02:28:17] 10Release-Engineering-Team (Done by Wed 24 Nov 🧟), 10Release, 10Train Deployments, 10User-Ladsgroup: 1.38.0-wmf.9 seems to have introduced a memory leak - https://phabricator.wikimedia.org/T296098 (10tstarling) I'm moving my work on the root cause to {T297667} [07:44:33] 10Release-Engineering-Team, 10MW-on-K8s, 10serviceops: Build MediaWiki images for kubernetes on the deployment servers - https://phabricator.wikimedia.org/T297673 (10Joe) [07:55:47] PROBLEM - SSH on contint1001.mgmt is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [09:56:02] RECOVERY - SSH on contint1001.mgmt is OK: SSH OK - OpenSSH_6.6 (protocol 2.0) https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [09:58:21] 10Continuous-Integration-Infrastructure, 10docker-pkg, 10Patch-For-Review: docker-pkg 3.0.1 fails with: 404 Client Error: Not Found ("pull access denied for wikimedia-bullseye, - https://phabricator.wikimedia.org/T297619 (10hashar) [12:39:13] hi releng what is the polkicy around private repos in gitlab? [12:57:42] 10Phabricator: Evaluate adding "In progress" status to Phabricator. - https://phabricator.wikimedia.org/T288956 (10jhsoby-WMNO) Thanks for adding this, @mmodell (and/or whoever else may have been involved). I have one minor issue; if you sort a workboard by status, the tasks that are set to "In progress" are sho... [13:41:41] jbond: I don't know whether there is any policy or any private/restricted repository yet! [13:42:17] jbond: but I guess brennen would know. Also we have #wikimedia-gitlab now :] [13:42:40] hashar: ack, it looks likethe privalige is disabled for my account, ah opk ill move over there thanks [13:42:47] joakino: #wikimedia-gitlab [13:47:58] jbond: and I have admin rights but I would rather not mess things up ! :D [13:48:25] hashar: yes sure thing, i dont want to start going against the grain :) [14:24:51] (03PS1) 10Hashar: stage-train: fix ssh-agent oneliner [tools/release] - 10https://gerrit.wikimedia.org/r/747121 [14:38:38] 10Release-Engineering-Team (Next), 10Patch-For-Review, 10Release, 10Train Deployments: 1.38.0-wmf.13 deployment blockers - https://phabricator.wikimedia.org/T293954 (10hashar) I haven't promoted testswikis to wmf.13 since there is an unsolved blocker from last week: T297517 . I have asked on that task whet... [14:48:19] so train is on hold cause there is some unsolved memory leak issue carried from wmf.12 [14:48:32] I have asked on the task whether we can push wmf.13 regardless [14:51:29] 10Release-Engineering-Team (Next), 10Patch-For-Review, 10Release, 10Train Deployments: 1.38.0-wmf.13 deployment blockers - https://phabricator.wikimedia.org/T293954 (10ssastry) >>! In T293954#7567725, @Arlolra wrote: > ##### Risky Patch! 🚂🔥 > > * **Change**: https://gerrit.wikimedia.org/r/c/mediawiki/ve... [15:21:57] 10Release-Engineering-Team (Next), 10Patch-For-Review, 10Release, 10Train Deployments: 1.38.0-wmf.13 deployment blockers - https://phabricator.wikimedia.org/T293954 (10hashar) I ran the `stage-train` but aborted before promoting the testwikis. I then ran `scap sync-world` to push all the wmf.13 code and wa... [15:55:11] (03CR) 10Ahmon Dancy: [C: 03+2] stage-train: fix ssh-agent oneliner [tools/release] - 10https://gerrit.wikimedia.org/r/747121 (owner: 10Hashar) [15:56:12] (03Merged) 10jenkins-bot: stage-train: fix ssh-agent oneliner [tools/release] - 10https://gerrit.wikimedia.org/r/747121 (owner: 10Hashar) [15:58:09] (03PS1) 10Jelto: helm-linter: remove helm2 from Docker image [integration/config] - 10https://gerrit.wikimedia.org/r/747147 (https://phabricator.wikimedia.org/T251305) [16:02:50] (03PS1) 10Jelto: jjb: update helm-linter job to releng/helm-linter:0.3.0 [integration/config] - 10https://gerrit.wikimedia.org/r/747148 (https://phabricator.wikimedia.org/T251305) [16:34:04] 10Deployments, 10Release-Engineering-Team: Deployment calendar: MediaWiki branch cut reported as between 3-4 AM UTC but actually it seems to be between 2-3 AM - https://phabricator.wikimedia.org/T297724 (10thcipriani) [16:43:22] 10Deployments, 10Release-Engineering-Team: Deployment calendar: MediaWiki branch cut reported as between 3-4 AM UTC but actually it seems to be between 2-3 AM - https://phabricator.wikimedia.org/T297724 (10thcipriani) p:05Triage→03Medium a:03thcipriani [17:43:45] 10Release-Engineering-Team (Doing), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.38.0-wmf.12 deployment blockers - https://phabricator.wikimedia.org/T293953 (10Legoktm) There's a possible regression at https://en.wikipedia.org/wiki/Wikipedia:Village_pump_(technical)#Edit_from_%22... [17:46:20] 10Deployments, 10Release-Engineering-Team: Deployment calendar: MediaWiki branch cut reported as between 3-4 AM UTC but actually it seems to be between 2-3 AM - https://phabricator.wikimedia.org/T297724 (10Zabe) I think https://gerrit.wikimedia.org/r/c/mediawiki/tools/release/+/742584 took care of this. [18:31:15] 10Phabricator, 10Design: Task symbol for "Open" suggests "alert" and is inconsistent in its placement when task types are also used - https://phabricator.wikimedia.org/T297249 (10mmodell) Proposal: How about the unckecked-box icon for open tasks: This: {icon square-o} instead of This: {icon exclamation-circle} [18:37:20] 10Release-Engineering-Team (Doing), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.38.0-wmf.12 deployment blockers - https://phabricator.wikimedia.org/T293953 (10matmarex) @Legoktm I bisected, caused by {ef458e894888d5d984f6b8dcf75cfae008d544d0} https://gerrit.wikimedia.org/r/c/med... [19:00:02] 10Release-Engineering-Team (Doing), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.38.0-wmf.12 deployment blockers - https://phabricator.wikimedia.org/T293953 (10matmarex) Filed as T297744. I'm not sure if this is necessarily a blocker. It's inconvenient but probably not a huge issue. [19:28:37] 10Release-Engineering-Team (Doing), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.38.0-wmf.12 deployment blockers - https://phabricator.wikimedia.org/T293953 (10Jdforrester-WMF) >>! In T293953#7570762, @matmarex wrote: > Filed as T297744. I'm not sure if this is necessarily a bloc... [19:31:30] 10Phabricator, 10Design: Task symbol for "Open" suggests "alert" and is inconsistent in its placement when task types are also used - https://phabricator.wikimedia.org/T297249 (10MBinder_WMF) Hmm. FWIW, I see that and think "I don't have some emoji coded properly" I am struggling to think of an alternative! I... [19:57:31] 10Release-Engineering-Team (Next), 10Patch-For-Review, 10Release, 10Train Deployments: 1.38.0-wmf.13 deployment blockers - https://phabricator.wikimedia.org/T293954 (10hashar) I am going to promote testwikis to wmf.13: * there are two blockers which seem fixed on beta cluster (master branch) and have thei... [20:01:42] PROBLEM - SSH on contint1001.mgmt is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [20:45:56] 10Phabricator, 10Design: Task symbol for "Open" suggests "alert" and is inconsistent in its placement when task types are also used - https://phabricator.wikimedia.org/T297249 (10mmodell) I can omit it from the workboard cards, however, now that it's been pointed out, the {icon exclamation-circle} just doesn't... [20:47:21] 10Phabricator: Editing a maniphest form has wrong preamble - https://phabricator.wikimedia.org/T295934 (10mmodell) That comes from the form for edit forms: https://phabricator.wikimedia.org/transactions/editengine/transactions.editengine.config/edit/101/ [20:49:32] 10Phabricator: Editing a maniphest form has wrong preamble - https://phabricator.wikimedia.org/T295934 (10mmodell) There shouldn't be more than one of these: https://phabricator.wikimedia.org/transactions/editengine/transactions.editengine.config/ [21:02:48] RECOVERY - SSH on contint1001.mgmt is OK: SSH OK - OpenSSH_6.6 (protocol 2.0) https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [21:04:14] 10Phabricator, 10Design: Task symbol for "Open" suggests "alert" and is inconsistent in its placement when task types are also used - https://phabricator.wikimedia.org/T297249 (10AntiCompositeNumber) On task lists and tasks themselves, the icon represents priority (with color) and status, not just status. I th... [21:06:46] 10Release-Engineering-Team (Next), 10Patch-For-Review, 10Release, 10Train Deployments: 1.38.0-wmf.13 deployment blockers - https://phabricator.wikimedia.org/T293954 (10hashar) I broke group 0 due to not bumping testwiki to wmf.13 when I ran `scap sync-world` earlier today. That means the l10n cache for wm... [21:23:25] 10Release-Engineering-Team (Next), 10Patch-For-Review, 10Release, 10Train Deployments: 1.38.0-wmf.13 deployment blockers - https://phabricator.wikimedia.org/T293954 (10Etonkovidova) [23:14:53] (03CR) 10jerkins-bot: [V: 04-1] scap backport: validate changes [tools/scap] - 10https://gerrit.wikimedia.org/r/743272 (https://phabricator.wikimedia.org/T294453) (owner: 10Jeena Huneidi) [23:20:52] (03PS4) 10Jeena Huneidi: scap backport: validate changes [tools/scap] - 10https://gerrit.wikimedia.org/r/743272 (https://phabricator.wikimedia.org/T294453) [23:21:25] (03CR) 10jerkins-bot: [V: 04-1] scap backport: validate changes [tools/scap] - 10https://gerrit.wikimedia.org/r/743272 (https://phabricator.wikimedia.org/T294453) (owner: 10Jeena Huneidi) [23:23:43] 10Phabricator, 10Design: Task symbol for "Open" suggests "alert" and is inconsistent in its placement when task types are also used - https://phabricator.wikimedia.org/T297249 (10Krinkle) >>! In T297249#7558358, @AntiCompositeNumber wrote: > I'm of the opinion that there doesn't need to be an icon to indicate... [23:37:59] (03PS5) 10Jeena Huneidi: scap backport: validate changes [tools/scap] - 10https://gerrit.wikimedia.org/r/743272 (https://phabricator.wikimedia.org/T294453) [23:38:32] (03CR) 10jerkins-bot: [V: 04-1] scap backport: validate changes [tools/scap] - 10https://gerrit.wikimedia.org/r/743272 (https://phabricator.wikimedia.org/T294453) (owner: 10Jeena Huneidi) [23:43:35] 10MediaWiki-Releasing, 10MW-1.38-release, 10MinervaNeue (Tracking), 10Patch-For-Review: Bundle MinervaNeue skin with MediaWiki - https://phabricator.wikimedia.org/T191743 (10Jdlrobson)