[03:35:56] !log Delete empty GitHub repositories that were created as mirror of a Gerrit repo but are empty and more than 5 years old [03:35:58] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [07:11:35] 10Phabricator, 10DBA, 10Patch-For-Review: Switchover m3 master db1159 -> db1164 - https://phabricator.wikimedia.org/T328404 (10Marostegui) [07:26:30] 10Phabricator, 10DBA, 10Patch-For-Review: Switchover m3 master db1159 -> db1164 - https://phabricator.wikimedia.org/T328404 (10Marostegui) [09:28:06] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Blocking 🧱), 10mwcli, 10Patch-For-Review, 10User-brennen: Add registry.gitlab.com/dependabot-gitlab/dependabot to list of allowed images for gitlab runners - https://phabricator.wikimedia.org/T326507 (10Jelto) 05In progress→03Resolved The image... [09:29:23] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Blocking 🧱), 10CirrusSearch, 10Discovery-Search, and 2 others: Consider adding "official" docker.elastic.co images to the list of allowed images for gitlab runners - https://phabricator.wikimedia.org/T327519 (10Jelto) 05In progress→03Resolved The... [09:47:32] 10Continuous-Integration-Infrastructure, 10Jenkins, 10Release-Engineering-Team: Cleanup some plugins from the releases Jenkins - https://phabricator.wikimedia.org/T328687 (10hashar) [09:58:17] 10Continuous-Integration-Infrastructure, 10Jenkins, 10Release-Engineering-Team: Cleanup some plugins from the releases Jenkins - https://phabricator.wikimedia.org/T328687 (10hashar) [10:32:31] 10GitLab (Infrastructure), 10serviceops-collab, 10Patch-For-Review: Make gitlab-replica-old accessible - https://phabricator.wikimedia.org/T328635 (10Jelto) p:05Triage→03Medium a:03Jelto [13:29:44] PROBLEM - Check systemd state on doc1002 is CRITICAL: CRITICAL - degraded: The following units failed: rsync-doc-doc2001.codfw.wmnet.service https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [14:16:04] jnuche: feel free to bring up the jenkins service on releases2002.codfw.wmnet if you need further testing. It is masked via puppet but can be resumed manually (`systemctl unmask jenkins ; systemctl start jenkins`) [14:16:31] I think the only potential issue is it might start triggering scheduled jobs, which afaik is only the train related ones [14:17:10] hashar: thanks, I already unmasked it yesterday, I'm currently fixing other issues, I cc'd you to a puppet config change that just got merged [14:17:17] then I don't even know whether the primary on releases2002 will be able to connect to releases1002 and attach it as an agent, so probably it will not even be able to run those jobs [14:17:27] and I guess puppet will mask it again and potentially stop the service [14:17:37] yeah the proxy thing, that is a nice one [14:22:57] meanwhile I have been digging in how the jenkins plugin manager resolve dependencies and well... [14:23:09] it is yet another dependency manager :] [14:26:09] interesting 👍 [14:27:24] RECOVERY - Check systemd state on doc1002 is OK: OK - running: The system is fully operational https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [14:32:45] it also do not take in account the plugins which are shipped by the jenkins.war [14:33:47] I wanted to make a list of plugins we want and a second file that locks down all the transitive dependencies, but could not find a good way to do so [14:33:51] i gotta dig a bit more :] [14:34:04] there are least some plugins we can remove [14:35:51] 10Continuous-Integration-Infrastructure, 10Jenkins, 10Accessibility, 10MinervaNeue (Tracking), 10User-zeljkofilipin: Save artifacts in Jenkins for selenium daily Minerva - https://phabricator.wikimedia.org/T328684 (10zeljkofilipin) [14:48:00] 10Continuous-Integration-Infrastructure, 10Jenkins, 10Accessibility, 10MinervaNeue (Tracking), 10User-zeljkofilipin: Save artifacts in Jenkins for selenium daily Minerva - https://phabricator.wikimedia.org/T328684 (10hashar) The CI job captures anything written to the log directory which, within the cont... [15:37:06] hashar: the plugins file for the releasing instance we are using right is exhaustive, it contains all the plugins in the prod instance, so that includes the transitive deps [15:37:18] *right now [15:54:17] 10GitLab, 10Release-Engineering-Team (GitLab IV: Mise En Place 🍱): Support passing environment variables to kokkuri's build-and-run-image - https://phabricator.wikimedia.org/T328054 (10dancy) 05Open→03Resolved [15:55:03] 10Release-Engineering-Team (GitLab IV: Mise En Place 🍱), 10Patch-For-Review, 10Release, 10Train Deployments: 1.40.0-wmf.21 deployment blockers - https://phabricator.wikimedia.org/T325584 (10dancy) 05Open→03Resolved [16:01:07] jnuche: yeah [16:02:04] I wanted to reach a state in which we would be able to tell we want plugin X Y Z , use the manager to resolve the dependencies and craft a plugin list that is exaustive [16:02:14] but is missing some bits here and there :] [16:04:29] I see, what I do is install the plugin on the local scap3 dev env and get an updated list of plugins that you can then pin the plugins file [16:04:56] *pin in [21:15:36] 10Continuous-Integration-Infrastructure, 10SRE, 10Traffic-Icebox, 10Patch-For-Review: Make CI run Varnish VCL tests - https://phabricator.wikimedia.org/T128188 (10BCornwall) 05Open→03Stalled Hi, @hashar! It's been quite a while but is there still any intention to add the CI integration? [22:53:22] (03CR) 10Arlolra: [C: 03+2] Use 'system: alphabetic' for custom counters to DRY out the counters [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/886130 (owner: 10Subramanya Sastry) [22:53:59] (03Merged) 10jenkins-bot: Use 'system: alphabetic' for custom counters to DRY out the counters [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/886130 (owner: 10Subramanya Sastry)