[07:43:27] (03CR) 10Hashar: [C: 03+2] operations-puppet: release new version of operations-puppet [integration/config] - 10https://gerrit.wikimedia.org/r/807180 (owner: 10Jbond) [07:45:29] (03Merged) 10jenkins-bot: operations-puppet: release new version of operations-puppet [integration/config] - 10https://gerrit.wikimedia.org/r/807180 (owner: 10Jbond) [07:46:07] !log Building operations-puppet docker image for https://gerrit.wikimedia.org/r/c/integration/config/+/807180 [07:46:08] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [07:47:20] (03PS1) 10Hashar: jjb: update operations-puppet image [integration/config] - 10https://gerrit.wikimedia.org/r/807478 [07:54:05] 10Release-Engineering-Team, 10Data-Engineering: Improve speed of Gitlab CI - https://phabricator.wikimedia.org/T311111 (10Antoine_Quhen) [08:00:49] 10Release-Engineering-Team, 10Data-Engineering: Improve speed of Gitlab CI - https://phabricator.wikimedia.org/T311111 (10Antoine_Quhen) [08:01:21] 10Release-Engineering-Team, 10Data-Engineering: Improve speed of Gitlab CI - https://phabricator.wikimedia.org/T311111 (10Antoine_Quhen) [08:01:56] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Doing), 10Zuul, 10Patch-For-Review: Delete .git/logs on zuul-merger git repo to prevent reflog from being created - https://phabricator.wikimedia.org/T307620 (10hashar) I tested the change by deleting the `test/gerrit-ping` repo on both z... [08:02:04] 10Release-Engineering-Team, 10Data-Engineering: Improve speed of Gitlab CI - https://phabricator.wikimedia.org/T311111 (10TheresNoTime) > tests suit is running in 4s locally, and in 1.5min in CI how is //that// possible? 😣 [08:03:37] (03CR) 10Hashar: [C: 03+2] "I have build the image:" [integration/config] - 10https://gerrit.wikimedia.org/r/807180 (owner: 10Jbond) [08:25:52] thanks hashar [08:30:58] 10GitLab, 10Release-Engineering-Team, 10Data-Engineering: Improve speed of Gitlab CI - https://phabricator.wikimedia.org/T311111 (10hashar) + #gitlab since there is surely caching optimization that would need to be added. It looks like building the image takes a while. Probably related, last week we had Gi... [08:32:00] jbond: you are welcome :) I have left a message about the puppet compiler in #wikimedia-sre-foundations which might be of "interest" to you (Hosts: P:foo:bar seems broken) [08:34:13] hashar: ack ill take a look cheers [08:39:30] 10GitLab (CI & Job Runners), 10Release-Engineering-Team, 10User-brennen: GitLab runners: allowed_images patterns need to be loosened to include subdirectories - https://phabricator.wikimedia.org/T310535 (10mmartorana) Hi @brennen, it seems that the issue is resolved now. Thank you so much. [08:52:26] 10GitLab, 10Release-Engineering-Team, 10Data-Engineering, 10Performance Issue: Improve speed of Gitlab CI - https://phabricator.wikimedia.org/T311111 (10Reedy) [08:59:47] 10GitLab (CI & Job Runners), 10Release-Engineering-Team, 10User-brennen: GitLab runners: allowed_images patterns need to be loosened to include subdirectories - https://phabricator.wikimedia.org/T310535 (10Jelto) I can confirm, pipeline works again for me. Re-configuring the Cloud Runners was block because o... [09:00:56] !log contint1001 , contint2002: setting `core.logallrefupdates=false` on all Zuul merger git repositories: `sudo -u zuul find /srv/zuul/git -type d -name .git -print -execdir git config core.logallrefupdates false \;` # T307620 [09:00:58] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [09:00:59] T307620: Delete .git/logs on zuul-merger git repo to prevent reflog from being created - https://phabricator.wikimedia.org/T307620 [09:08:11] !log contint1001 , contint2002: deleting `.git/logs` from all zuul-merger repositories. We do not need the reflog `sudo -u zuul find /srv/zuul/git -type d -name .git -print -execdir rm -fR .git/logs \;` # T307620 [09:08:13] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [09:08:13] T307620: Delete .git/logs on zuul-merger git repo to prevent reflog from being created - https://phabricator.wikimedia.org/T307620 [09:09:31] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Doing), 10Zuul: Delete .git/logs on zuul-merger git repo to prevent reflog from being created - https://phabricator.wikimedia.org/T307620 (10hashar) 05Openβ†’03Resolved Finally solved, the git reflog are no more created :] [09:24:43] 10Release-Engineering-Team (Priority Backlog πŸ“₯), 10Patch-For-Review, 10Release, 10Train Deployments: 1.39.0-wmf.17 deployment blockers - https://phabricator.wikimedia.org/T308070 (10hashar) Logs and metrics look good so far. It seems group 1 wikis are rather happy with 1.39.0-wmf.17 :) [09:26:53] (03CR) 10Hashar: [C: 03+2] "I have uploaded the job an hour or so ago at 8:04 UTC but forgot to +2 the change." [integration/config] - 10https://gerrit.wikimedia.org/r/807478 (owner: 10Hashar) [09:29:19] (03Merged) 10jenkins-bot: jjb: update operations-puppet image [integration/config] - 10https://gerrit.wikimedia.org/r/807478 (owner: 10Hashar) [09:35:33] !log Switched `gitlab-prod-1001.devtools.eqiad1.wikimedia.cloud` instance to use the project Puppet master `puppetmaster-1001.devtools.eqiad1.wikimedia.cloud` [09:35:34] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [09:36:59] 10GitLab, 10Data-Engineering: Experiencing pipeline failure due to disk-space issues - https://phabricator.wikimedia.org/T310593 (10Jelto) Docker cache is cleaned every 24h on GitLab Runner nodes now. So failing jobs due to full docker volume should happen less frequent. [09:42:04] 10GitLab, 10Data-Engineering: Experiencing pipeline failure due to disk-space issues - https://phabricator.wikimedia.org/T310593 (10hashar) 05Openβ†’03Resolved For the scope of this task, that solves the issue. Additional tasks can be filed to keep the cache longer, potentially share them across runners etc [10:05:12] 10GitLab, 10Data-Engineering: Experiencing pipeline failure due to disk-space issues - https://phabricator.wikimedia.org/T310593 (10Antoine_Quhen) Thanks! Also for space & speed, we may not using the ci cache properly: * https://docs.gitlab.com/ee/ci/caching/ * https://phabricator.wikimedia.org/T311111 [11:17:56] 10Release-Engineering-Team, 10Scap, 10Patch-For-Review: scap-o-scap: Bootstrapping a new host fails - https://phabricator.wikimedia.org/T310740 (10jbond) Sorry for the delay in commenting. > That patch currently applies the new bootstrapping config in the scap class, but it needs access to at least one of t... [12:07:52] 10GitLab (Administration, Settings & Policy), 10Release-Engineering-Team (GitLab-a-thon 🦊), 10Patch-For-Review, 10cloud-services-team (Kanban): gitlab: consider enabling docker container registry - https://phabricator.wikimedia.org/T304845 (10Ottomata) Aw, that is unfortunate! We were really hoping for th... [12:19:21] 10GitLab (Administration, Settings & Policy), 10Release-Engineering-Team (GitLab-a-thon 🦊), 10Patch-For-Review, 10cloud-services-team (Kanban): gitlab: consider enabling docker container registry - https://phabricator.wikimedia.org/T304845 (10akosiaris) >>! In T304845#8019564, @Ottomata wrote: > Aw, that i... [12:31:05] 10GitLab (Administration, Settings & Policy), 10Release-Engineering-Team (GitLab-a-thon 🦊), 10Patch-For-Review, 10cloud-services-team (Kanban): gitlab: consider enabling docker container registry - https://phabricator.wikimedia.org/T304845 (10Ottomata) Hm, yeah @hashar had told me not to do this, because t... [12:52:35] 10GitLab, 10Release-Engineering-Team, 10Data-Engineering, 10Performance Issue: Improve speed of Gitlab CI - https://phabricator.wikimedia.org/T311111 (10Ottomata) Also relevant: {T304845} discussion at https://phabricator.wikimedia.org/T304845#8017565 [14:10:19] (03CR) 10Jforrester: [C: 03+2] README.md: hint at composer test [tools/code-utils] - 10https://gerrit.wikimedia.org/r/799260 (https://phabricator.wikimedia.org/T309099) (owner: 10Hashar) [14:10:59] (03Merged) 10jenkins-bot: README.md: hint at composer test [tools/code-utils] - 10https://gerrit.wikimedia.org/r/799260 (https://phabricator.wikimedia.org/T309099) (owner: 10Hashar) [14:33:15] hey hashar - I think I might have messed up github sync again πŸ˜‡. Would you be so kind to take a look at some point (absolutely not urgent, ofc)? https://gerrit.wikimedia.org/g/operations/software/helm-state-metrics [15:29:47] 10Release-Engineering-Team, 10Scap, 10Patch-For-Review: scap-o-scap: Bootstrapping a new host fails - https://phabricator.wikimedia.org/T310740 (10dancy) >>! In T310740#8019500, @jbond wrote: > Sorry for the delay in commenting. > >> That patch currently applies the new bootstrapping config in the scap clas... [15:54:30] 10Release-Engineering-Team (Radar), 10Scap, 10Patch-For-Review, 10User-jijiki: Update Scap to perform rolling restart for all MW deploy - https://phabricator.wikimedia.org/T266055 (10dancy) >>! In T266055#8008163, @JMeybohm wrote: > Hey @dancy, do you think we're good to proceed with disabling revalidation... [16:06:08] (03PS1) 10Hashar: Move gerrit-jsonschemagenerator to Bazel [integration/config] - 10https://gerrit.wikimedia.org/r/807586 [16:06:59] (03CR) 10Hashar: [C: 03+2] "I have dropped Maven in https://gerrit.wikimedia.org/r/c/operations/software/gerrit/jsonschemagenerator/+/791642/6..7" [integration/config] - 10https://gerrit.wikimedia.org/r/807586 (owner: 10Hashar) [16:09:39] (03Merged) 10jenkins-bot: Move gerrit-jsonschemagenerator to Bazel [integration/config] - 10https://gerrit.wikimedia.org/r/807586 (owner: 10Hashar) [16:11:10] 10Release-Engineering-Team (Deployment Autopilot πŸ›©οΈ), 10Scap, 10Patch-For-Review: Automated Tuesday Train via a timer - https://phabricator.wikimedia.org/T310395 (10dancy) a:03hashar [16:13:20] 10Release-Engineering-Team (Deployment Autopilot πŸ›©οΈ), 10Scap, 10Patch-For-Review: Automated Tuesday Train via a timer - https://phabricator.wikimedia.org/T310395 (10hashar) [16:13:32] 10Release-Engineering-Team (Deployment Autopilot πŸ›©οΈ), 10Scap: Create alerts for automated Tuesday Train timer - https://phabricator.wikimedia.org/T310396 (10hashar) I will do the systemd timer (T310395) and I think the Puppet define has the logic to send alerts to AlertManager. [16:19:39] !log Reloaded Zuul for https://gerrit.wikimedia.org/r/807586 [16:19:40] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:23:44] 10Release-Engineering-Team (Deployment Autopilot πŸ›©οΈ), 10Scap, 10Patch-For-Review: Allow Scap to push to Gerrit without operator creds - https://phabricator.wikimedia.org/T306425 (10akosiaris) [16:35:14] 10Phabricator: "Restricted File" displayed in "Referenced Files" section on a task - https://phabricator.wikimedia.org/T311158 (10Stang) [16:36:36] 10Phabricator: "Restricted File" displayed in "Referenced Files" section on a task - https://phabricator.wikimedia.org/T311158 (10Stang) [16:36:49] 10Phabricator (Upstream), 10Release-Engineering-Team, 10Upstream, 10User-brennen: Uploaded files via the drag-and-drop are defaulting to private-access - https://phabricator.wikimedia.org/T310833 (10Stang) [16:37:00] 10Phabricator: "Restricted File" displayed in "Referenced Files" section on a task - https://phabricator.wikimedia.org/T311158 (10Stang) Sorry about that, failed to search it properly. [17:35:18] !log gerrit: create group extension-JsonData with robla in it, make it an owner of mediawiki/extensions/JsonData per request in T303147 [17:35:21] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:35:21] T303147: Request membership in MediaWiki-extensions-JsonData group for robla - https://phabricator.wikimedia.org/T303147 [17:36:48] !log gerrit: add tfellows to the extension-OpenBadges group per request in T308278 [17:36:51] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:36:51] T308278: Request membership in "mediawiki/extensions/OpenBadges" group for tfellows - https://phabricator.wikimedia.org/T308278 [18:03:09] (03CR) 10Ahmon Dancy: [C: 04-1] Support a different user for Gerrit ssh interactions (031 comment) [tools/scap] - 10https://gerrit.wikimedia.org/r/805462 (https://phabricator.wikimedia.org/T306425) (owner: 10Hashar) [18:07:12] (03PS1) 10Ahmon Dancy: rearrange SSH parameters for consistency with SSH_WITH_KEY [tools/scap] - 10https://gerrit.wikimedia.org/r/807599 [18:07:53] (03CR) 10Ahmon Dancy: [C: 03+2] rearrange SSH parameters for consistency with SSH_WITH_KEY [tools/scap] - 10https://gerrit.wikimedia.org/r/807599 (owner: 10Ahmon Dancy) [18:14:22] (03Merged) 10jenkins-bot: rearrange SSH parameters for consistency with SSH_WITH_KEY [tools/scap] - 10https://gerrit.wikimedia.org/r/807599 (owner: 10Ahmon Dancy) [18:24:21] 10Phabricator: Some deadline stamps change color, while others do not, despite having the same deadline - https://phabricator.wikimedia.org/T311168 (10MBinder_WMF) [18:24:54] 10Phabricator: Some deadline stamps change color, while others do not, despite having the same deadline - https://phabricator.wikimedia.org/T311168 (10MBinder_WMF) [18:32:29] hashar: where are we on doc1001? do we need to schedule something or we waiting for someone? [18:34:21] ok, stupid differential questions time: i've accepted https://phabricator.wikimedia.org/D1203 - how do i merge it? [18:41:40] 10Phabricator (Upstream), 10Release-Engineering-Team, 10Upstream, 10User-brennen: Uploaded files via the drag-and-drop are defaulting to private-access - https://phabricator.wikimedia.org/T310833 (10Jdlrobson) Just echoing the concerns above: this is causing **a lot** of confusion in my team's workflow. I... [18:45:01] brennen: just apply it locally and push, I believe [19:01:11] 10Phabricator (Upstream), 10Release-Engineering-Team, 10Upstream, 10User-brennen: Uploaded files via the drag-and-drop are defaulting to private-access - https://phabricator.wikimedia.org/T310833 (10RHo) +1 it would be super appreciated if this can be resolved soon. It is causing more work and lost time fo... [19:02:17] 10Phabricator (Upstream), 10Release-Engineering-Team, 10Upstream, 10User-brennen: Uploaded files via the drag-and-drop are defaulting to private-access - https://phabricator.wikimedia.org/T310833 (10TheresNoTime) {meme, src=scremcat, below="its a feature"} [19:03:01] 10Phabricator (Upstream), 10Release-Engineering-Team, 10Upstream, 10User-brennen: Uploaded files via the drag-and-drop are defaulting to private-access - https://phabricator.wikimedia.org/T310833 (10brennen) p:05Triageβ†’03High I am aware of the severity of the issue. We have a fix; I will get it deploy... [19:04:26] brennen: I thought the attachment bug was due to outdated JS (ref T310850), not due to an upstream phab bug? [19:04:27] T310850: Phabricator loading outdated JavaScript - https://phabricator.wikimedia.org/T310850 [19:05:01] i think it's a combination of outdated js breaking some UI and upstream's workflow being fairly shit. [19:12:15] 10Continuous-Integration-Infrastructure, 10Jenkins: Upgrade Jenkins to latest LTS 2.346.1 - https://phabricator.wikimedia.org/T311174 (10hashar) [19:13:13] 10Continuous-Integration-Infrastructure, 10Jenkins, 10Release-Engineering-Team: Upgrade Jenkins to latest LTS 2.346.1 - https://phabricator.wikimedia.org/T311174 (10hashar) [19:15:09] 10Phabricator (Upstream), 10Release-Engineering-Team, 10Upstream, 10User-brennen: Uploaded files via the drag-and-drop are defaulting to private-access - https://phabricator.wikimedia.org/T310833 (10MBinder_WMF) Thanks, @brennen ! I know you folks have a lot going on and really appreciate your attention on... [19:15:47] Krinkle: do you want to attend the upgrade? I will drop a mail to try to find a time slot that works [19:16:23] hashar: sure, I can make anytime [19:16:49] (03Abandoned) 10Hashar: Add SonarQube scanner [software/gerrit/jsonschemagenerator] - 10https://gerrit.wikimedia.org/r/791692 (owner: 10Hashar) [19:17:29] will try to sort out the mess my personal agenda is in June [19:36:26] 10Phabricator, 10Release-Engineering-Team, 10User-brennen: Deploy Phabricator release/2022-06-22/1 - https://phabricator.wikimedia.org/T311175 (10brennen) [19:40:47] 10Phabricator, 10Release-Engineering-Team, 10User-brennen: Deploy Phabricator release/2022-06-22/1 - https://phabricator.wikimedia.org/T311175 (10brennen) p:05Triageβ†’03High This is another relatively fast update of code and restart of PHP / phd / apache2. Includes no database migrations, etc. I don't r... [19:55:29] (03CR) 10Hashar: Support a different user for Gerrit ssh interactions (032 comments) [tools/scap] - 10https://gerrit.wikimedia.org/r/805462 (https://phabricator.wikimedia.org/T306425) (owner: 10Hashar) [20:01:49] 10Phabricator, 10Release-Engineering-Team, 10User-brennen: Deploy Phabricator release/2022-06-22/1 - https://phabricator.wikimedia.org/T311175 (10brennen) [20:19:41] `New in version 3.8.` [20:19:42] grbmblbl [20:28:41] (03PS9) 10Hashar: Support a different user for Gerrit ssh interactions [tools/scap] - 10https://gerrit.wikimedia.org/r/805462 (https://phabricator.wikimedia.org/T306425) [20:30:38] (03CR) 10Hashar: Support a different user for Gerrit ssh interactions (031 comment) [tools/scap] - 10https://gerrit.wikimedia.org/r/805462 (https://phabricator.wikimedia.org/T306425) (owner: 10Hashar) [21:03:08] (03PS1) 10Ahmon Dancy: Use $TRAIN_DEV_DIR instead of relative paths [tools/train-dev] - 10https://gerrit.wikimedia.org/r/807614 [21:03:11] (03PS1) 10Ahmon Dancy: Set up keyholder-like ssh-agent for debian user [tools/train-dev] - 10https://gerrit.wikimedia.org/r/807615 [21:04:43] (03CR) 10Ahmon Dancy: [C: 03+2] Use $TRAIN_DEV_DIR instead of relative paths [tools/train-dev] - 10https://gerrit.wikimedia.org/r/807614 (owner: 10Ahmon Dancy) [21:05:16] (03Merged) 10jenkins-bot: Use $TRAIN_DEV_DIR instead of relative paths [tools/train-dev] - 10https://gerrit.wikimedia.org/r/807614 (owner: 10Ahmon Dancy) [21:06:09] (03PS2) 10Ahmon Dancy: Set up keyholder-like ssh-agent for debian user [tools/train-dev] - 10https://gerrit.wikimedia.org/r/807615 [21:18:47] (03CR) 10Hashar: "I thought it could be split in smaller bits but I think it is good enough as is :)" [tools/scap] - 10https://gerrit.wikimedia.org/r/805462 (https://phabricator.wikimedia.org/T306425) (owner: 10Hashar) [21:23:40] (03CR) 10Ahmon Dancy: Support a different user for Gerrit ssh interactions (033 comments) [tools/scap] - 10https://gerrit.wikimedia.org/r/805462 (https://phabricator.wikimedia.org/T306425) (owner: 10Hashar) [21:30:41] 10Scap: Catalog of errors from scap (or subprocesses) related to php-fpm restart always being enabled - https://phabricator.wikimedia.org/T311181 (10dancy) [21:30:59] 10Release-Engineering-Team, 10Scap: Scap pool counter error while backporting - https://phabricator.wikimedia.org/T310835 (10dancy) [21:31:02] 10Scap: Catalog of errors from scap (or subprocesses) related to php-fpm restart always being enabled - https://phabricator.wikimedia.org/T311181 (10dancy) [21:34:38] 10Scap: scap error on 2022-06-21 deploy - https://phabricator.wikimedia.org/T311182 (10dancy) [21:34:45] 10Scap: scap error on 2022-06-21 deploy - https://phabricator.wikimedia.org/T311182 (10dancy) [21:34:47] 10Scap: Catalog of errors from scap (or subprocesses) related to php-fpm restart always being enabled - https://phabricator.wikimedia.org/T311181 (10dancy) [21:35:25] 10Release-Engineering-Team, 10Scap: Catalog of errors from scap (or subprocesses) related to php-fpm restart always being enabled - https://phabricator.wikimedia.org/T311181 (10dancy) [21:39:02] 10Scap: scap error on 2022-06-21 deploy - https://phabricator.wikimedia.org/T311182 (10dancy) @Lucas_Werkmeister_WMDE also wrote: > I think there’s two issues there – the failed connection to lvs2010 (understandable if that was being worked on at the moment), and the fact that https://gerrit.wikimedia.org/g/oper... [21:42:44] (03PS2) 10Dduvall: history: Mark latest history entry as synced following any sync command [tools/scap] - 10https://gerrit.wikimedia.org/r/805496 (https://phabricator.wikimedia.org/T287045) [21:43:02] (03PS3) 10Dduvall: history: Mark latest history entry as synced following any sync command [tools/scap] - 10https://gerrit.wikimedia.org/r/805496 (https://phabricator.wikimedia.org/T287045) [23:28:53] 10Release-Engineering-Team (Radar), 10Scap, 10Patch-For-Review, 10User-jijiki: Update Scap to perform rolling restart for all MW deploy - https://phabricator.wikimedia.org/T266055 (10Krinkle) [23:29:48] 10Release-Engineering-Team (Radar), 10Scap, 10Patch-For-Review, 10User-jijiki: Update Scap to perform rolling restart for all MW deploy - https://phabricator.wikimedia.org/T266055 (10Krinkle) Added to task description, and ticked off the first: > * [x] Set `opcache.validate_timestamps=0` on prod canaries.... [23:56:24] Hi. How do I schedule a deployment to Beta Cluster?