[00:01:40] (DatasourceError) firing: Queue (Jenkins jobs + Zuul functions) alert - https://grafana.wikimedia.org/alerting/grafana/iS0FSjJ4z/view - https://wikitech.wikimedia.org/wiki/Monitoring/DatasourceError - https://alerts.wikimedia.org/?q=alertname%3DDatasourceError [00:06:40] (DatasourceError) resolved: Queue (Jenkins jobs + Zuul functions) alert - https://grafana.wikimedia.org/alerting/grafana/iS0FSjJ4z/view - https://wikitech.wikimedia.org/wiki/Monitoring/DatasourceError - https://alerts.wikimedia.org/?q=alertname%3DDatasourceError [00:06:55] (DatasourceError) firing: Queue (Jenkins jobs + Zuul functions) alert - https://grafana.wikimedia.org/alerting/grafana/iS0FSjJ4z/view - https://wikitech.wikimedia.org/wiki/Monitoring/DatasourceError - https://alerts.wikimedia.org/?q=alertname%3DDatasourceError [00:11:55] (DatasourceError) resolved: Queue (Jenkins jobs + Zuul functions) alert - https://grafana.wikimedia.org/alerting/grafana/iS0FSjJ4z/view - https://wikitech.wikimedia.org/wiki/Monitoring/DatasourceError - https://alerts.wikimedia.org/?q=alertname%3DDatasourceError [01:46:23] (03CR) 10Krinkle: secnpm: initial commit (031 comment) [fresh] - 10https://gerrit.wikimedia.org/r/675346 (owner: 10Krinkle) [03:52:58] (03PS1) 10Subramanya Sastry: Some enwikivoyage specific noise-reduction hacks [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/1011027 [03:53:41] (03CR) 10Subramanya Sastry: "I am already running visual diff tests with this code in place on parsing-qa-02" [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/1011027 (owner: 10Subramanya Sastry) [03:53:43] (03CR) 10CI reject: [V:04-1] Some enwikivoyage specific noise-reduction hacks [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/1011027 (owner: 10Subramanya Sastry) [03:59:07] (03PS2) 10Subramanya Sastry: Some enwikivoyage specific noise-reduction hacks [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/1011027 [04:42:39] 10GitLab (Pipeline Services Migration🐤), 06collaboration-services: move security.wikimedia.org to kubernetes - https://phabricator.wikimedia.org/T350796#9629112 (10Arnoldokoth) [06:26:26] 10Gerrit, 06SRE, 10SRE-Access-Requests: 14Not able to access Gerrit - 14https://phabricator.wikimedia.org/T360006#9629155 (10cchen) 14@hashar Thank you so much! It works now! [09:25:09] (03PS1) 10Hslater: Add VisualEditorPlus as dependency to BlueSpiceVisualEditorConnector [integration/config] - 10https://gerrit.wikimedia.org/r/1011084 [09:45:50] Krinkle: nice finding. Linking to gitweb is hardcoded in Zuul and that happens for events not having a change number (aka a ref update such as a tag being created). That is made configurable eventually: https://review.opendev.org/c/zuul/zuul/+/515168 [09:45:54] but we don't have this change :) [10:34:01] (03CR) 10Hashar: [C:03+2] Add VisualEditorPlus as dependency to BlueSpiceVisualEditorConnector [integration/config] - 10https://gerrit.wikimedia.org/r/1011084 (owner: 10Hslater) [10:34:40] (03Merged) 10jenkins-bot: Add VisualEditorPlus as dependency to BlueSpiceVisualEditorConnector [integration/config] - 10https://gerrit.wikimedia.org/r/1011084 (owner: 10Hslater) [10:35:37] (03CR) 10Hashar: [C:03+2] "Deployed" [integration/config] - 10https://gerrit.wikimedia.org/r/1011084 (owner: 10Hslater) [13:08:16] 10Release-Engineering-Team (Priority Backlog 📥), 05Release, 05Train Deployments: 1.42.0-wmf.22 deployment blockers - https://phabricator.wikimedia.org/T354440#9630791 (10taavi) [13:08:48] (03CR) 10Subramanya Sastry: [C:04-1] "Needs more tweaking. too aggressive p-br-p stripping in some cases, and not agressive enough in others." [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/1011027 (owner: 10Subramanya Sastry) [13:20:39] (03PS1) 10Samtar: Allow list: Add PageTriage GSoC contributors [integration/config] - 10https://gerrit.wikimedia.org/r/1011116 [13:21:42] (03CR) 10Sohom Datta: [C:03+1] Allow list: Add PageTriage GSoC contributors [integration/config] - 10https://gerrit.wikimedia.org/r/1011116 (owner: 10Samtar) [13:22:16] (03CR) 10Novem Linguae: [C:03+1] "Regex looks good. These are PageTriage GSoC folks." [integration/config] - 10https://gerrit.wikimedia.org/r/1011116 (owner: 10Samtar) [13:23:35] (03CR) 10Hashar: [C:03+2] Allow list: Add PageTriage GSoC contributors [integration/config] - 10https://gerrit.wikimedia.org/r/1011116 (owner: 10Samtar) [13:23:43] (03Merged) 10jenkins-bot: Allow list: Add PageTriage GSoC contributors [integration/config] - 10https://gerrit.wikimedia.org/r/1011116 (owner: 10Samtar) [13:42:00] (03CR) 10Hashar: [C:03+2] "Deployed!" [integration/config] - 10https://gerrit.wikimedia.org/r/1011116 (owner: 10Samtar) [13:44:32] 10GitLab (Project Migration), 06collaboration-services: Migrate SRE repositories to GitLab - https://phabricator.wikimedia.org/T341468#9631120 (10BCornwall) [13:45:17] 10GitLab (Project Migration), 06Traffic, 13Patch-For-Review: Migrate Traffic repositories from Gerrit to Gitlab - https://phabricator.wikimedia.org/T347623#9631118 (10BCornwall) 05In progress→03Stalled Stalling since we're not entirely solid on what we want GitLab to be. [13:48:57] to whom and/or where should bring up https://wikitech.wikimedia.org/wiki/Template:Deploy for some feedback? It'd be nice to make adding deployments to https://wikitech.wikimedia.org/wiki/Deployments slightly easier and more standardised :) [13:49:54] TheresNoTime: the talk page of that template? [13:51:21] I'm not sure anyone will see that, unless I ping folx to it :/ I created the template a while back and am only now remembering it [13:57:43] 10Release-Engineering-Team (Priority Backlog 📥), 05Release, 05Train Deployments: 1.42.0-wmf.23 deployment blockers - https://phabricator.wikimedia.org/T354441#9631181 (10Jdforrester-WMF) ##### Risky Patch! 🚂🔥 * **Change**: https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/62 and https://gi... [14:01:32] 10Release-Engineering-Team (Now this 🫠), 05Release, 05Train Deployments: 141.42.0-wmf.21 deployment blockers - 14https://phabricator.wikimedia.org/T354439#9631214 (10Jgiannelos) [14:14:09] Hi Releng! Since recently, I'm getting `15:04:47 A dependency error was encountered while installing the extension "GrowthExperiments": Could not find the registration file for the extension "VisualEditor"` in CI for GrowthExperiments. To me, that looks like a CI error. Any idea why this is happening? [14:14:18] https://gerrit.wikimedia.org/r/c/mediawiki/extensions/GrowthExperiments/+/1011115 is the patch [14:23:06] Hello everyone. I’m going to work on adding a new wiki on beta cluster again. Just a heads up if something goes south again. [14:39:39] !log on deployment-deploy03 execution “mwscript extensions/WikimediaMaintenance/addWiki.php --wiki=aawiki --skipclusters=main,extstore,echo,growth,mediamoderation en wikipedia test2wiki test2.wikipedia.beta.wmcloud.org” failed with `Unknown database test2wiki’ T355281 [14:39:43] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:39:43] T355281: Set up some beta cluster wikis with different registrable domain - https://phabricator.wikimedia.org/T355281 [14:42:43] I'm running into a puppet tarpit with geoip support in deployment-prep. Can anyone comment on if I should try to fix or just rip it out entirely? [14:43:25] pmiazga: still not working? [14:46:48] 10Release-Engineering-Team (Now this 🫠), 13Patch-For-Review: gitlab-cloud-runner: Roll back pending helm releases before running terraform apply - https://phabricator.wikimedia.org/T354787#9631376 (10CodeReviewBot) dancy opened https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/360... [14:47:57] 10Release-Engineering-Team (Now this 🫠), 13Patch-For-Review: gitlab-cloud-runner: Roll back pending helm releases before running terraform apply - https://phabricator.wikimedia.org/T354787#9631386 (10CodeReviewBot) dancy merged https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/360... [14:50:10] 10Release-Engineering-Team (Priority Backlog 📥), 05Release, 05Train Deployments: 1.42.0-wmf.22 deployment blockers - https://phabricator.wikimedia.org/T354440#9631402 (10Lucas_Werkmeister_WMDE) [14:55:54] (03PS3) 10Subramanya Sastry: Some enwikivoyage specific noise-reduction hacks [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/1011027 [14:56:24] 10Release-Engineering-Team (Now this 🫠), 13Patch-For-Review: gitlab-cloud-runner: Roll back pending helm releases before running terraform apply - https://phabricator.wikimedia.org/T354787#9631409 (10CodeReviewBot) dancy opened https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/361... [14:56:31] (03CR) 10CI reject: [V:04-1] Some enwikivoyage specific noise-reduction hacks [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/1011027 (owner: 10Subramanya Sastry) [14:57:26] 10Release-Engineering-Team (Now this 🫠), 13Patch-For-Review: gitlab-cloud-runner: Roll back pending helm releases before running terraform apply - https://phabricator.wikimedia.org/T354787#9631410 (10CodeReviewBot) dancy merged https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/361... [14:58:28] !log on deployment-deploy03 execution “mwscript extensions/WikimediaMaintenance/addWiki.php --wiki=aawiki en wikipedia test2wiki test2.wikipedia.beta.wmcloud.org” failed with `Query::isWriteQuery called with incorrect flags parameter’ T355281 [14:58:31] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:58:31] T355281: Set up some beta cluster wikis with different registrable domain - https://phabricator.wikimedia.org/T355281 [14:58:37] TheresNoTime - nope, still not working [14:59:10] But I saw that with the db fixes db wasn’t present, so I executed the command without `skipClusters` to recreate schema from scratch. [15:00:24] this failed too, but now “Wikimedia\Rdbms\DBLanguageError from line 87 of /srv/mediawiki-staging/php-master/includes/libs/rdbms/database/Query.php: Wikimedia\Rdbms\Query::isWriteQuery called with incorrect flags parameter “ [15:00:36] I’m going trough addWiki and checking what that could be [15:01:06] (03PS4) 10Subramanya Sastry: Some enwikivoyage specific noise-reduction hacks [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/1011027 [15:03:07] 10Beta-Cluster-Infrastructure, 06MediaWiki-Platform-Team, 10MW-1.42-notes (1.42.0-wmf.20; 2024-02-27): Cannot create a new wiki on beta cluster - https://phabricator.wikimedia.org/T358236#9631435 (10pmiazga) Recreating a new wiki from scratch fails with: ` pmiazga@deployment-deploy03:~$ mwscript extensions/... [15:09:23] 10Release-Engineering-Team (Priority Backlog 📥), 05Release, 05Train Deployments: 141.42.0-wmf.22 deployment blockers - 14https://phabricator.wikimedia.org/T354440#9631479 (10hashar) 05Open→03Resolved 14Looks like 1.42.0-wmf.22 is well behaving \o/ [15:15:11] 10Beta-Cluster-Infrastructure, 06MediaWiki-Platform-Team, 10MW-1.42-notes (1.42.0-wmf.20; 2024-02-27): Cannot create a new wiki on beta cluster - https://phabricator.wikimedia.org/T358236#9631510 (10pmiazga) @aaron @Ladsgroup could you advise on what to do here? The `ExternalStoreDB`calls `$dbw->query()` wit... [15:15:22] Does anyone know of a config setting for gerrit that will make it show timestamps in UTC in the web gui instead of the browser's timezone? [15:15:44] (03CR) 10SBassett: secnpm: initial commit (031 comment) [fresh] - 10https://gerrit.wikimedia.org/r/675346 (owner: 10Krinkle) [15:16:50] TheresNoTime: there is probably a issue in ExternalStoreDB and latest Database lib changes, I already pinged Amir1 and AaronSchulz. [15:28:33] bd808: Couldn't find anything. :-( [15:28:59] (03CR) 10Subramanya Sastry: "I think I have finally tamed the noise." [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/1011027 (owner: 10Subramanya Sastry) [15:36:39] 10Release-Engineering-Team (Now this 🫠), 13Patch-For-Review: gitlab-cloud-runner: Roll back pending helm releases before running terraform apply - https://phabricator.wikimedia.org/T354787#9631552 (10CodeReviewBot) dancy opened https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/363... [15:37:37] 10Release-Engineering-Team (Now this 🫠), 13Patch-For-Review: gitlab-cloud-runner: Roll back pending helm releases before running terraform apply - https://phabricator.wikimedia.org/T354787#9631558 (10CodeReviewBot) dancy merged https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/363... [15:55:26] bd808: Gerrit uses `new Intl.DateTimeFormat(locale, options).formatToParts(date);` , and the options could take a `timeZone` but there is nothing to set in the UI code and user preferences [15:55:48] at a quick glance, it might not not be too hard to set :) [15:57:52] an alternative is to write a browser extension which change the timezone on a per host/domain basis [16:00:35] that per-domain tz variance actually sounds like a fun nerd snipe project... [16:03:43] :q [16:06:31] bd808: for the wikitech hook to block/unblock accounts in Gerrit ( https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/1010853 ) I am not entirely willing to setup a test environment to verify it and I feel like that can be checked direclty in prod [16:07:00] and it looks like the hooks are a bit copy pasted and could use to be a standalone function with some unit tests. I might tackle that as a friday project :) [16:07:45] hashar: agreed that testing it live is fine. See https://phabricator.wikimedia.org/T307558#9457469 for what I did the last time. [16:08:54] blocking/unblocking is what I had in mind while watching the log bucket on mwlogXXXX [16:09:16] hashar: and if you are nerd snipped into more work on those hooks, long ago tgr asked for me to do T222209. [16:09:18] T222209: Cleanup logging and curl use in wikitech post-block hooks - https://phabricator.wikimedia.org/T222209 [16:13:28] bd808: I am not sure what needs to be done [16:13:40] and it almost smells like maybe those can be turned into a MediaWiki extension [16:13:48] but maybe it is overkill [16:15:42] at least I have gained some familiarity with the hooks but I am not sure what the task is about [16:16:02] I guess remove the curl functions in favor of Guzzle / whatever high API MediaWiki exposes [16:16:37] and for logging replace `wfDebugLog()` by whatever is 2024 way of doing logging? [16:16:59] anyway, bd808 may you CR+2 https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/1010853 and I will babysit the deployment and testing ;) [16:22:45] hashar: +2 is your right in that repo. I'm not planning on being the deployer. ;) [16:22:57] I can deploy :) [16:23:15] then I guess I could self +2 :) [16:50:07] 10Gerrit, 06Release-Engineering-Team, 10wikitech.wikimedia.org, 07SecTeam-Processed, 07Security: wikitech hook to disable Gerrit user uses partial matches to identify account - https://phabricator.wikimedia.org/T307558#9631820 (10hashar) After deploying [[ https://gerrit.wikimedia.org/r/c/operations/medi... [17:08:25] 10Continuous-Integration-Infrastructure, 07Jenkins, 06Release-Engineering-Team, 13Patch-For-Review: Patch testing in releases Jenkins: prevent false positives - https://phabricator.wikimedia.org/T359809#9631919 (10CodeReviewBot) jnuche merged https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/... [17:10:09] (03PS1) 10Majavah: jjb, Zuul: Stop testing labs/toollabs on buster [integration/config] - 10https://gerrit.wikimedia.org/r/1011150 [17:11:50] taavi: poor Buster :/ [17:12:15] I think the labs/toollabs code resulted in Debian packages for each of the support Debian distros [17:12:31] and at some point we would build the packages for eg jessie/stretch/buster all out of the same code/branch [17:15:53] (03CR) 10Majavah: [C:03+2] "deleted Jenkins jobs" [integration/config] - 10https://gerrit.wikimedia.org/r/1011150 (owner: 10Majavah) [17:16:08] !log delete debian-glue-buster, debian-glue-buster-non-voting Jenkins jobs [17:16:10] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:17:36] (03Merged) 10jenkins-bot: jjb, Zuul: Stop testing labs/toollabs on buster [integration/config] - 10https://gerrit.wikimedia.org/r/1011150 (owner: 10Majavah) [17:17:48] taavi: you should drop the template while at it [17:17:59] I am pretty sure that was only used by toollabs [17:18:09] !log reloading zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/1011150 [17:18:11] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:18:14] and if there is no interest in testing against multiple distro, the logic can be cleaned up :) [17:18:20] i'll have a look [17:19:34] taavi: thanks :) [17:19:38] bd808: moaare fun https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/1011151/1/wmf-config/wikitech.php :) [17:19:46] Gerrit loves HTTP status codes [17:20:14] I am off for kids/dinner etc. I'd follow up on that patch if there is any review / hints [17:23:44] (03PS1) 10Majavah: jjb, Zuul: Drop per-distribution debian-glue jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1011154 [17:31:03] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Now this 🫠), 13Patch-For-Review: 14foreachwiki on beta does not include en_rtlwiki - 14https://phabricator.wikimedia.org/T357877#9631992 (10dancy) 05Open→03Resolved a:03dancy [17:34:30] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Now this 🫠), 13Patch-For-Review: 14foreachwiki on beta does not include en_rtlwiki - 14https://phabricator.wikimedia.org/T357877#9632024 (10dancy) 14` dancy@deployment-mwmaint02:~$ foreachwiki sql.php TIL about following bliss [17:37:58] * hashar dines [17:47:40] heh. sorry/you're welcome for the new idiom hashar [17:55:50] (03CR) 10Jforrester: "Maybe split a Zuul-only commit to drop usage, and a jjb-only one to drop the job-creation (and templates)? That way we can revert/etc. as " [integration/config] - 10https://gerrit.wikimedia.org/r/1011154 (owner: 10Majavah) [17:58:14] 10Beta-Cluster-Infrastructure, 06MediaWiki-Platform-Team, 10MW-1.42-notes (1.42.0-wmf.20; 2024-02-27), 13Patch-For-Review: Cannot create a new wiki on beta cluster - https://phabricator.wikimedia.org/T358236#9632117 (10Tgr) The logic of handling an SQL string in `Database::query()` looks solid to me, but..... [18:38:01] bd808: that brings me back in 2011/2014 when I was the sole non native english speaker [19:08:50] and of course `if ( !curl_exec( .. ) )` might not be doing what ones want [19:09:01] since the request can succeed but returns an empty body [19:09:11] which is then evaluated as a false value [19:09:32] that should be `if ( curl_exec( .. ) === false )` [19:09:37] * hashar whistles [19:18:01] I have send that as a patch as well [19:50:38] (03PS18) 10Krinkle: fresh-npm: initial commit [fresh] - 10https://gerrit.wikimedia.org/r/675346 [19:50:42] (03CR) 10Krinkle: fresh-npm: initial commit (031 comment) [fresh] - 10https://gerrit.wikimedia.org/r/675346 (owner: 10Krinkle) [20:44:36] Project beta-scap-sync-world build #146766: 04FAILURE in 9 min 22 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/146766/ [20:53:38] Project beta-scap-sync-world build #146767: 04STILL FAILING in 5 min 11 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/146767/ [20:57:11] Project beta-scap-sync-world build #146768: 04STILL FAILING in 1 min 18 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/146768/ [21:03:16] [21:03:16] 20:56:57 20:56:57 ['/usr/bin/scap', 'pull', '--no-php-restart', '--no-update-l10n', '--exclude-wikiversions.php', 'deployment-deploy03.deployment-prep.eqiad1.wikimedia.cloud'] (ran as mwdeploy@deployment-snapshot03.deployment-prep.eqiad1.wikimedia.cloud) returned [255]: Host key verification failed [21:06:35] Project beta-scap-sync-world build #146769: 04STILL FAILING in 1 min 14 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/146769/ [21:10:42] Looks like /etc/ssh/ssh_known_hosts changed at Mar 14 20:50 [21:10:54] (on deployment-deploy03) [21:13:19] dancy: first error was 20:44 [21:13:58] Looking at /var/log/puppet.log shows that the file was updated a few times recently. [21:14:18] Weird [21:14:43] I wouldn't expect it change at all unless a server was reimaged [21:16:25] Project beta-scap-sync-world build #146770: 04STILL FAILING in 1 min 13 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/146770/ [21:26:25] Project beta-scap-sync-world build #146771: 04STILL FAILING in 1 min 16 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/146771/ [21:28:37] thcipriani: I said earlier our Gerrit instance is rather small compared to other users. An example from today has one running on a 60 CPU / 180G RAM with a Java heap of 160G https://groups.google.com/g/repo-discuss/c/I2cWVO8GVk4 ;) [21:31:52] huh, that known_hosts thing triggered a (hopefully) very out-of-date memory: https://phabricator.wikimedia.org/T144647#2648622 [21:32:54] (but I guess that's still open, so maybe not an out-of-date memory) [21:33:32] thcipriani: see -cloud [21:33:41] thanks [21:35:28] thcipriani: moare https://sal.toolforge.org/releng?p=0&q=known_hosts&d= ;) [21:35:41] including a Tyler's oneliner: [21:35:41] ``` [21:35:41] deployment-deploy01$ while read host; do ssh-keyscan -H "$host"; done < <(grep -vP '^#' /etc/dsh/group/mediawiki-installation | grep -vP '^$' ) | sudo -u jenkins-deploy tee /mnt/home/jenkins-deploy/.ssh/known_hosts [21:35:41] ``` [21:36:16] and `add ssh host key for deployment-mediawiki07 to /mnt/home/jenkins-deploy/.ssh/known_hosts so that beta-scap-eqiad will work again` [21:36:27] Project beta-scap-sync-world build #146772: 04STILL FAILING in 1 min 14 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/146772/ [21:36:32] hashar: move to -cloud [21:36:39] Possibly related work ongoing [21:36:45] na na I am sticking to baremetal [21:36:58] joke aside [21:37:14] I don't know how the ssh known_hosts file is generated on WMCS machines [21:38:10] * hashar sleeps [21:38:31] hashar: answer in -cloud [21:46:29] Project beta-scap-sync-world build #146773: 04STILL FAILING in 1 min 16 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/146773/ [21:47:01] !log Disabled beta-scap-sync-world job until ssh_known_hosts issues are resolved (see #wikimedia-cloud) [21:47:04] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [22:14:18] !log Reenabled beta-scap-sync-world [22:14:20] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [22:26:26] Yippee, build fixed! [22:26:26] Project beta-scap-sync-world build #146774: 09FIXED in 11 min: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/146774/ [22:37:38] 10Release-Engineering-Team (Priority Backlog 📥), 05Release, 05Train Deployments: 1.42.0-wmf.24 deployment blockers - https://phabricator.wikimedia.org/T360156#9632736 (10thcipriani) p:05Triage→03Medium a:03jeena [22:38:17] 10Release-Engineering-Team (Priority Backlog 📥), 05Release, 05Train Deployments: 1.42.0-wmf.25 deployment blockers - https://phabricator.wikimedia.org/T360157#9632741 (10thcipriani) p:05Triage→03Medium a:03jnuche [23:50:39] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for USER[S] - https://phabricator.wikimedia.org/T360168 (10Yug) 03NEW