[00:18:12] (03CR) 10CI reject: [V: 04-1] build: Updating mediawiki/mediawiki-phan-config to 0.12.0 [integration/docroot] - 10https://gerrit.wikimedia.org/r/840677 (owner: 10Libraryupgrader) [05:20:26] PROBLEM - Check systemd state on doc1002 is CRITICAL: CRITICAL - degraded: The following units failed: rsync-doc-doc2001.codfw.wmnet.service https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [06:15:56] RECOVERY - Check systemd state on doc1002 is OK: OK - running: The system is fully operational https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [07:36:22] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Priority Backlog 📥): Support http_proxy, https_proxy and other proxy `build-arg:` options in blubber buildkit frontend - https://phabricator.wikimedia.org/T317997 (10hashar) Looks like https://gerrit.wikimedia.org/r/c/operations/puppet/+/833125/ broke Pu... [08:12:01] (03CR) 10Kosta Harlan: "We could perhaps also make use of `composer phpunit:coverage-edit`. Then in the future, any adjustments we'd make would be to includes/com" [integration/config] - 10https://gerrit.wikimedia.org/r/840337 (https://phabricator.wikimedia.org/T320332) (owner: 10Legoktm) [08:33:30] (03CR) 10Hashar: [V: 03+2 C: 03+2] Allow SRE to send annotated and signed tags [puppet] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/836711 (owner: 10Hashar) [08:34:13] (03CR) 10Hashar: [V: 03+2 C: 03+2] "Clément, that would let SRE push any kind of tags ;)" [puppet] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/836711 (owner: 10Hashar) [08:55:55] 10GitLab (Project Migration), 10Release-Engineering-Team, 10serviceops-collab: Create new GitLab project group: wm-juniors-il - https://phabricator.wikimedia.org/T313750 (10jijiki) [08:56:23] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Priority Backlog 📥), 10serviceops-collab, 10Patch-For-Review, 10User-brennen: Deploy buildkitd to trusted GitLab runners - https://phabricator.wikimedia.org/T308271 (10jijiki) [09:11:39] (03CR) 10Jaime Nuche: [C: 03+2] perform-release: Fix old version handling (031 comment) [tools/scap] - 10https://gerrit.wikimedia.org/r/838941 (owner: 10Ahmon Dancy) [09:15:31] (03Merged) 10jenkins-bot: perform-release: Fix old version handling [tools/scap] - 10https://gerrit.wikimedia.org/r/838941 (owner: 10Ahmon Dancy) [09:51:15] (03PS1) 10Clément Goubert: helm-linter: update helm3 to 3.9.4 [integration/config] - 10https://gerrit.wikimedia.org/r/841042 (https://phabricator.wikimedia.org/T317511) [09:52:42] (03CR) 10JMeybohm: [C: 03+1] helm-linter: update helm3 to 3.9.4 [integration/config] - 10https://gerrit.wikimedia.org/r/841042 (https://phabricator.wikimedia.org/T317511) (owner: 10Clément Goubert) [09:53:53] (03CR) 10Hashar: POST events asynchronously (032 comments) [software/gerrit/plugins/events-wikimedia] - 10https://gerrit.wikimedia.org/r/816115 (owner: 10Hashar) [09:54:13] (03PS3) 10Hashar: POST events asynchronously [software/gerrit/plugins/events-wikimedia] - 10https://gerrit.wikimedia.org/r/816115 [09:54:23] (03CR) 10CI reject: [V: 04-1] POST events asynchronously [software/gerrit/plugins/events-wikimedia] - 10https://gerrit.wikimedia.org/r/816115 (owner: 10Hashar) [09:54:34] fun times [09:54:43] (03PS1) 10Clément Goubert: jjb: update helm-linter job to releng/helm-linter:0.4.1 [integration/config] - 10https://gerrit.wikimedia.org/r/841043 (https://phabricator.wikimedia.org/T317511) [09:55:34] 10GitLab (Infrastructure), 10serviceops-collab: Create GitLab mirror - https://phabricator.wikimedia.org/T291322 (10jijiki) [09:56:38] claime: will do that after lunch ;) [09:56:49] hashar: np [10:05:52] (03CR) 10JMeybohm: [C: 03+1] jjb: update helm-linter job to releng/helm-linter:0.4.1 [integration/config] - 10https://gerrit.wikimedia.org/r/841043 (https://phabricator.wikimedia.org/T317511) (owner: 10Clément Goubert) [10:21:57] 10GitLab (Auth & Access), 10Release-Engineering-Team, 10User-brennen: Create bot to sync LDAP groups with related GitLab groups - https://phabricator.wikimedia.org/T319211 (10jbond) >>! In T319211#8280270, @brennen wrote: > Some background for current state on {T274461} and {J273}. tl;dr all the ways i have... [10:22:38] 10GitLab (CI & Job Runners), 10Release-Engineering-Team, 10Security-Team, 10Patch-For-Review, and 2 others: Allow registry.gitlab.com/security-products/**/* for gitlab shared runner docker images - https://phabricator.wikimedia.org/T312961 (10Jelto) To me it's not clear what images are in `registry.gitlab.... [10:25:29] 10GitLab (CI & Job Runners), 10Release-Engineering-Team, 10Security-Team, 10serviceops-collab, and 3 others: Allow registry.gitlab.com/security-products/**/* for gitlab shared runner docker images - https://phabricator.wikimedia.org/T312961 (10Jelto) [10:26:35] 10GitLab, 10Infrastructure-Foundations, 10SRE, 10CAS-SSO: migrate gitlab away from the CAS protocol - https://phabricator.wikimedia.org/T320390 (10jbond) [10:26:49] 10GitLab, 10Infrastructure-Foundations, 10SRE, 10CAS-SSO: migrate gitlab away from the CAS protocol - https://phabricator.wikimedia.org/T320390 (10jbond) p:05Triage→03Medium [10:46:39] 10Release-Engineering-Team, 10Wikibase Product Platform: Incorrect handling of ETags taking precedence over timestamps in conditional requests - https://phabricator.wikimedia.org/T320241 (10Silvan_WMDE) Don't exactly know where to put this ticket. Maybe releng knows something about Apache configs? [10:54:44] (03CR) 10Clément Goubert: Allow SRE to send annotated and signed tags (031 comment) [puppet] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/836711 (owner: 10Hashar) [11:06:13] claime: I am building the helm-linter image and will bump the jobs once it has completed [11:07:16] (03CR) 10Hashar: [C: 03+2] helm-linter: update helm3 to 3.9.4 [integration/config] - 10https://gerrit.wikimedia.org/r/841042 (https://phabricator.wikimedia.org/T317511) (owner: 10Clément Goubert) [11:08:16] hashar: ack, thank you [11:09:04] (03Merged) 10jenkins-bot: helm-linter: update helm3 to 3.9.4 [integration/config] - 10https://gerrit.wikimedia.org/r/841042 (https://phabricator.wikimedia.org/T317511) (owner: 10Clément Goubert) [11:17:17] (03CR) 10Hashar: [C: 03+2] "Successfully published image docker-registry.discovery.wmnet/releng/helm-linter:0.4.1" [integration/config] - 10https://gerrit.wikimedia.org/r/841042 (https://phabricator.wikimedia.org/T317511) (owner: 10Clément Goubert) [11:18:29] (03CR) 10Hashar: [C: 03+2] "INFO:jenkins_jobs.builder:Number of jobs generated: 1" [integration/config] - 10https://gerrit.wikimedia.org/r/841043 (https://phabricator.wikimedia.org/T317511) (owner: 10Clément Goubert) [11:18:36] claime: done ! :) [11:20:20] (03Merged) 10jenkins-bot: jjb: update helm-linter job to releng/helm-linter:0.4.1 [integration/config] - 10https://gerrit.wikimedia.org/r/841043 (https://phabricator.wikimedia.org/T317511) (owner: 10Clément Goubert) [11:21:00] hashar: thank you! [11:29:31] 10Beta-Cluster-Infrastructure, 10Patch-For-Review: deployment-ms-fe03 puppet failure - https://phabricator.wikimedia.org/T316845 (10TheresNoTime) >>! In T316845#8290817, @MatthewVernon wrote: > Devices are configured in the ring by name (e.g. /dev/sda3 appears in the ring as sda3), so I guess you have /dev/lv-... [11:35:29] (03CR) 10Jaime Nuche: [C: 04-1] "I created a local buildkit image with these changes and then used that to try to create another buildkit image for the arm64 arch. That fa" [blubber] - 10https://gerrit.wikimedia.org/r/836307 (https://phabricator.wikimedia.org/T318866) (owner: 10Dduvall) [11:42:17] 10Beta-Cluster-Infrastructure, 10Patch-For-Review: deployment-ms-fe03 puppet failure - https://phabricator.wikimedia.org/T316845 (10MatthewVernon) That sounds right, but it'd be worth eyeballing what's actually in the ring - if you do ` sudo swift-ring-builder /etc/swift/object.builder | less ` and check that... [11:44:44] (03CR) 10Jaime Nuche: [C: 04-1] "I think it'd be worth mentioning in the README that the build option is supported." [blubber] - 10https://gerrit.wikimedia.org/r/836307 (https://phabricator.wikimedia.org/T318866) (owner: 10Dduvall) [11:47:53] 10Beta-Cluster-Infrastructure, 10Patch-For-Review: deployment-ms-fe03 puppet failure - https://phabricator.wikimedia.org/T316845 (10TheresNoTime) Looks good: ` Devices: id region zone ip address:port replication ip:port name weight partitions balance flags meta 2 1 3 172.16.7.114:6000... [12:04:36] !log cherry 836953 picking for T316845 to deployment-prep/Swift [12:04:38] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [12:04:38] T316845: deployment-ms-fe03 puppet failure - https://phabricator.wikimedia.org/T316845 [12:06:11] 10Beta-Cluster-Infrastructure, 10Patch-For-Review: deployment-ms-fe03 puppet failure - https://phabricator.wikimedia.org/T316845 (10TheresNoTime) Does this seem promising? ` samtar@deployment-ms-fe03:~$ sudo swift_ring_manager -v Checking dispersion for policy standard Checking dispersion for policy lowlatency... [12:14:10] 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-Phonos, 10Community-Tech (CommTech-Sprint-34), 10MW-1.40-notes (1.40.0-wmf.2; 2022-09-19), 10Patch-For-Review: Phonos links to an unauthorized URL - https://phabricator.wikimedia.org/T317417 (10TheresNoTime) [12:14:55] 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-Phonos, 10SRE-swift-storage, 10Community-Tech (CommTech-Sprint-34), and 2 others: Phonos links to an unauthorized URL - https://phabricator.wikimedia.org/T317417 (10TheresNoTime) [12:15:50] 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-Phonos, 10SRE-swift-storage, 10Community-Tech (CommTech-Sprint-34), and 2 others: Phonos links to an unauthorized URL - https://phabricator.wikimedia.org/T317417 (10TheresNoTime) [12:26:24] 10Phabricator, 10Release-Engineering-Team (Bonus Level 🕹️), 10DBA, 10serviceops, and 2 others: sort out mysql privileges for phab1004/phab2002 - https://phabricator.wikimedia.org/T315713 (10LSobanski) p:05Triage→03Medium [12:53:19] 10GitLab (CI & Job Runners), 10serviceops-collab: Evaluate if Docker can be executed as non-root on Trusted Runners - https://phabricator.wikimedia.org/T320411 (10Jelto) [12:53:56] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Radar), 10Security-Team, 10serviceops, and 3 others: Setup GitLab Runner in trusted environment - https://phabricator.wikimedia.org/T295481 (10Jelto) [12:57:49] 10Beta-Cluster-Infrastructure, 10Patch-For-Review: deployment-ms-fe03 puppet failure - https://phabricator.wikimedia.org/T316845 (10MatthewVernon) Hi, that looks like the devices are correct, but the weights are different - I think you should set all the weights in the scheme to 1000 (which I infer is what the... [12:58:59] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Radar), 10Security-Team, 10serviceops, and 3 others: Setup GitLab Runner in trusted environment - https://phabricator.wikimedia.org/T295481 (10Jelto) Some more explanation to the above edit: Further security hardening of Docker daemon got a dedicate... [14:14:27] (03CR) 10Clément Goubert: buildkit: Support builds for specific target platforms (031 comment) [blubber] - 10https://gerrit.wikimedia.org/r/836307 (https://phabricator.wikimedia.org/T318866) (owner: 10Dduvall) [16:09:29] 10Scap: scap backport includes ProxyWarning errors - https://phabricator.wikimedia.org/T320436 (10Urbanecm) [16:09:57] (03PS1) 10Jaime Nuche: buildkit: Support Docker's `.dockerfile` [blubber] - 10https://gerrit.wikimedia.org/r/841175 (https://phabricator.wikimedia.org/T319337) [16:35:50] 10GitLab (Project Migration), 10Striker, 10Tools, 10Developer-Advocacy (Jul-Sep 2022), 10User-bd808: Fix misleading "Diffusion" creation in Striker (actually creates on GitLab) - https://phabricator.wikimedia.org/T320438 (10valerio.bozzolan) [22:09:20] 10GitLab, 10Release-Engineering-Team (GitLab II: Wrath of Kahn 👾): Fork buildkitd and disable the auth token cache that is currently shared between client connections - https://phabricator.wikimedia.org/T319694 (10dduvall) [22:10:06] 10GitLab, 10Release-Engineering-Team (GitLab II: Wrath of Kahn 👾): Fork buildkitd and disable the auth token cache that is currently shared between client connections - https://phabricator.wikimedia.org/T319694 (10dduvall)