[05:03:55] 10Release-Engineering-Team (Priority Backlog 📥), 05Release, 05Train Deployments: 1.42.0-wmf.22 deployment blockers - https://phabricator.wikimedia.org/T354440#9609896 (10abi_) ##### Risky Patch! 🚂🔥 * **Change**: https://gerrit.wikimedia.org/r/c/mediawiki/core/+/977085 * **Summary**: ** Nothing particularl... [10:38:26] Yippee, build fixed! [10:38:26] Project beta-update-databases-eqiad build #74340: 09FIXED in 10 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/74340/ [10:39:36] 10Continuous-Integration-Infrastructure, 07Jenkins, 06Release-Engineering-Team, 07Security: Jenkins plugins security advisory 2024-03-06 - https://phabricator.wikimedia.org/T359415#9610483 (10jnuche) 05Open→03Resolved a:03jnuche Both instances have now been updated. [10:46:27] 10Release-Engineering-Team (Now this 🫠), 13Patch-For-Review, 05Release, 05Train Deployments: 1.42.0-wmf.21 deployment blockers - https://phabricator.wikimedia.org/T354439#9610504 (10jnuche) [11:12:26] (03PS2) 10Ilias Sarantopoulos: ml-services: renamed directory for readability [integration/config] - 10https://gerrit.wikimedia.org/r/1009218 (https://phabricator.wikimedia.org/T353461) [11:24:42] 06Release-Engineering-Team, 06Tech-Docs-Team, 07Documentation: Review supporting deployment pipeline documentation - https://phabricator.wikimedia.org/T352262#9610640 (10CodeReviewBot) kbach updated https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/74 Draft: Improve autogenerated documentat... [12:08:24] 10Phabricator, 10GerritBot: GerritBot comments for 7-digit Gerrit changes conflict with Diffusion commit hashes - https://phabricator.wikimedia.org/T358940#9610824 (10matmarex) More examples: T89947#9538996, T352308#9609776 (I swear I'm not going looking for them, these are just from tasks in my inbox, so this... [12:57:20] (03PS1) 10Jaime Nuche: Makefile.deploy: split installation in two steps [integration/zuul/deploy] - 10https://gerrit.wikimedia.org/r/1009524 (https://phabricator.wikimedia.org/T358237) [12:57:55] (03CR) 10CI reject: [V: 04-1] Makefile.deploy: split installation in two steps [integration/zuul/deploy] - 10https://gerrit.wikimedia.org/r/1009524 (https://phabricator.wikimedia.org/T358237) (owner: 10Jaime Nuche) [13:02:28] (03PS2) 10Jaime Nuche: Makefile.deploy: split installation in two steps [integration/zuul/deploy] - 10https://gerrit.wikimedia.org/r/1009524 (https://phabricator.wikimedia.org/T358237) [13:10:56] (03CR) 10Jaime Nuche: [C: 03+2] Makefile.deploy: split installation in two steps [integration/zuul/deploy] - 10https://gerrit.wikimedia.org/r/1009524 (https://phabricator.wikimedia.org/T358237) (owner: 10Jaime Nuche) [13:11:28] (03Merged) 10jenkins-bot: Makefile.deploy: split installation in two steps [integration/zuul/deploy] - 10https://gerrit.wikimedia.org/r/1009524 (https://phabricator.wikimedia.org/T358237) (owner: 10Jaime Nuche) [13:24:02] 10Release-Engineering-Team (Now this 🫠), 10Scap, 10MW-on-K8s, 06SRE, 06serviceops: Scap should check errors coming from mw-on-k8s canaries during deployments - https://phabricator.wikimedia.org/T357402#9611194 (10Clement_Goubert) [13:24:18] 10Release-Engineering-Team (Seen), 10MW-on-K8s, 06SRE, 06Traffic, 06serviceops: Serve production traffic via Kubernetes - https://phabricator.wikimedia.org/T290536#9611195 (10Clement_Goubert) [13:24:46] 10Release-Engineering-Team (Seen), 10MW-on-K8s, 06SRE, 06Traffic, 06serviceops: Serve production traffic via Kubernetes - https://phabricator.wikimedia.org/T290536#9611196 (10Clement_Goubert) [13:26:25] 06Release-Engineering-Team, 10MW-on-K8s, 06SRE, 06Traffic, and 2 others: Move 60% of mediawiki external requests to mw on k8s - https://phabricator.wikimedia.org/T357508#9611192 (10Clement_Goubert) 05In progress→03Resolved [14:06:18] 10Diffusion, 10Phabricator, 06collaboration-services: Drop our mirroring of code to Diffusion and empty the repos - https://phabricator.wikimedia.org/T359549 (10Jdforrester-WMF) 03NEW [14:57:38] 10Phabricator: Requesting Phabricator admin rights for Urbanecm - https://phabricator.wikimedia.org/T359557 (10Urbanecm) 03NEW [15:19:04] 06Release-Engineering-Team, 06Tech-Docs-Team, 07Documentation: Review supporting deployment pipeline documentation - https://phabricator.wikimedia.org/T352262#9611835 (10bd808) @kbach, out of curiosity more than anything, why choose Vitepress as the static site generator over the many, many other options? [15:31:59] 10Release-Engineering-Team (Seen), 06Content-Transform-Team-WIP, 10MW-on-K8s, 06SRE, and 3 others: A lot of `[info] Wikitext for this page has duplicate ids:` in logstash for mw-parsoid. Possibly related to PageBundle - https://phabricator.wikimedia.org/T358588#9611908 (10MSantos) [16:00:02] 10Release-Engineering-Team (Now this 🫠), 10Scap, 10MW-on-K8s, 06SRE, and 2 others: Adapt scap's testing strategy to mw-on-k8s - https://phabricator.wikimedia.org/T358117#9611979 (10CodeReviewBot) dancy merged https://gitlab.wikimedia.org/repos/releng/train-dev/-/merge_requests/48 exp/files/php/scap.cfg: S... [16:07:17] 10Release-Engineering-Team (Radar), 06collaboration-services: Ensure that gitlab.wikimedia.org adheres to Google's sender guidelines - https://phabricator.wikimedia.org/T355776#9611991 (10Dzahn) It's "userReportedSpamRate" and you can also play around with the "dr=" parameter which I think means days but that... [16:08:03] 10Continuous-Integration-Infrastructure, 06SRE, 06collaboration-services, 10vm-requests, 13Patch-For-Review: Ganeti VM for contint migration - https://phabricator.wikimedia.org/T358237#9611992 (10Dzahn) zuul has now succesfully been deployed to this machine by @jnuche :) [16:15:51] 06Release-Engineering-Team, 10Release Pipeline, 06Security-Team, 07SecTeam-Processed: Solve OSV Double-Pipeline Problem without Requiring Many Default Rules - https://phabricator.wikimedia.org/T358728#9612020 (10cmassaro) Yep, it looks like this is working much better now. I've adopted @dduvall 's simpler... [16:26:11] 10Phabricator, 10GerritBot: GerritBot comments for 7-digit Gerrit changes conflict with Diffusion commit hashes - https://phabricator.wikimedia.org/T358940#9612068 (10Tacsipacsi) Both increasing the minimum number of digits and emptying the repos would break existing links. Couldn’t the above patch work with a... [16:34:36] 10Release-Engineering-Team (Radar), 06collaboration-services: Ensure that gitlab.wikimedia.org adheres to Google's sender guidelines - https://phabricator.wikimedia.org/T355776#9612092 (10Dzahn) 05Open→03Resolved [16:49:32] 06Release-Engineering-Team, 06Tech-Docs-Team, 07Documentation: Review supporting deployment pipeline documentation - https://phabricator.wikimedia.org/T352262#9612192 (10KBach) >>! In T352262#9611835, @bd808 wrote: > @kbach, out of curiosity more than anything, why choose Vitepress as the static site generat... [16:59:20] 10Release-Engineering-Team (Seen), 10Scap: Run Swagger checks in Scap before exposing to prod MW traffic - https://phabricator.wikimedia.org/T239376#9612263 (10dancy) 05Open→03Resolved a:03dancy @Krinkle I think the work on T358117 satisfies the goals stated here so I'm resolving this ticket. [17:02:35] 06Release-Engineering-Team, 10Release Pipeline, 06Security-Team, 07SecTeam-Processed: Solve OSV Double-Pipeline Problem without Requiring Many Default Rules - https://phabricator.wikimedia.org/T358728#9612294 (10sbassett) >>! In T358728#9612020, @cmassaro wrote: > Yep, it looks like this is working much be... [17:23:05] 10Release-Engineering-Team (Seen), 10Scap: Run Swagger checks in Scap before exposing to prod MW traffic - https://phabricator.wikimedia.org/T239376#9612439 (10Krinkle) Thanks @dancy. Does the below match your understanding? * Swagger checks have been removed in favour of `httpbb` checks. * The `httpbb` check... [17:24:23] 10GitLab (Pipeline Services Migration🐤), 06collaboration-services: move security.wikimedia.org to kubernetes - https://phabricator.wikimedia.org/T350796#9612461 (10Arnoldokoth) a:03Arnoldokoth [17:26:00] Project beta-scap-sync-world build #145742: 04FAILURE in 45 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/145742/ [17:28:41] 10GitLab (Pipeline Services Migration🐤), 06collaboration-services: move security.wikimedia.org to kubernetes - https://phabricator.wikimedia.org/T350796#9612470 (10Arnoldokoth) [17:30:17] 10Release-Engineering-Team (Seen), 10Scap: Run Swagger checks in Scap before exposing to prod MW traffic - https://phabricator.wikimedia.org/T239376#9612482 (10dancy) >>! In T239376#9612439, @Krinkle wrote: > Thanks @dancy. Does the below match your understanding? > > * Swagger checks have been removed in fav... [17:30:42] 10Release-Engineering-Team (Now this 🫠), 10Scap, 10MW-on-K8s, 06SRE, 06serviceops: Scap should check errors coming from mw-on-k8s canaries during deployments - https://phabricator.wikimedia.org/T357402#9612486 (10dancy) [17:31:22] 10Release-Engineering-Team (Now this 🫠), 10Scap, 10MW-on-K8s, 06SRE, 06serviceops: Adapt scap's testing strategy to mw-on-k8s - https://phabricator.wikimedia.org/T358117#9612484 (10dancy) 05In progress→03Resolved Changes deployed and tested. Resolving this task. [17:32:28] 10Release-Engineering-Team (Now this 🫠), 10Scap, 10MW-on-K8s, 06SRE, 06serviceops: Adapt scap's testing strategy to mw-on-k8s - https://phabricator.wikimedia.org/T358117#9612505 (10Clement_Goubert) @dancy Thanks a bunch! \o/ [17:32:56] !log Upgrading scap to latest code revision in beta cluster [17:32:58] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:36:26] Yippee, build fixed! [17:36:26] Project beta-scap-sync-world build #145743: 09FIXED in 1 min 13 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/145743/ [17:53:47] 10Phabricator: Requesting Phabricator admin rights for Urbanecm - https://phabricator.wikimedia.org/T359557#9612682 (10DannyS712) [19:33:21] 10Release-Engineering-Team (Now this 🫠), 05Release, 05Train Deployments: 1.42.0-wmf.21 deployment blockers - https://phabricator.wikimedia.org/T354439#9613187 (10Etonkovidova) [19:52:51] 06Release-Engineering-Team, 06Tech-Docs-Team, 07Documentation: Review supporting deployment pipeline documentation - https://phabricator.wikimedia.org/T352262#9613251 (10bd808) Thank you for this description of the process and criteria @KBach. I might personally quibble that 59MiB of node modules is not exac... [20:59:01] 10Release-Engineering-Team (Now this 🫠), 10Cassandra, 13Patch-For-Review: Find alternative to scap for deployment of logstash-logback-encoder - https://phabricator.wikimedia.org/T357739#9613422 (10dancy) 05Open→03Resolved [21:05:40] 10Release-Engineering-Team (Now this 🫠), 13Patch-For-Review: gitlab-cloud-runner: Roll back pending helm releases before running terraform apply - https://phabricator.wikimedia.org/T354787#9613447 (10CodeReviewBot) sandeeps merged https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/3... [21:06:57] 10Release-Engineering-Team (Now this 🫠), 10Cassandra, 13Patch-For-Review: Find alternative to scap for deployment of logstash-logback-encoder - https://phabricator.wikimedia.org/T357739#9613448 (10Eevans) 05Resolved→03Open p:05Medium→03Low I think that ultimately we may still want to do this (even if... [21:08:26] 10Release-Engineering-Team (Now this 🫠), 10Cassandra, 13Patch-For-Review: Package logstash-logback-encoder for Debian - https://phabricator.wikimedia.org/T357739#9613455 (10Eevans) [21:08:43] 10Release-Engineering-Team (Now this 🫠), 10Cassandra, 13Patch-For-Review: Package logstash-logback-encoder for Debian - https://phabricator.wikimedia.org/T357739#9613457 (10dancy) a:05dancy→03None [21:09:22] 10Release-Engineering-Team (Now this 🫠), 10Cassandra, 13Patch-For-Review: Package logstash-logback-encoder for Debian - https://phabricator.wikimedia.org/T357739#9613459 (10Eevans) a:03dancy [21:09:34] 10Release-Engineering-Team (Now this 🫠), 10Cassandra, 13Patch-For-Review: Package logstash-logback-encoder for Debian - https://phabricator.wikimedia.org/T357739#9613465 (10Eevans) a:05dancy→03None [21:14:10] 10GitLab (CI & Job Runners), 06Release-Engineering-Team: Upgrade production gitlab-cloud-runners to kubernetes 1.29 - https://phabricator.wikimedia.org/T359594 (10dancy) 03NEW [21:14:51] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Now this 🫠): Upgrade production gitlab-cloud-runners to kubernetes 1.29 - https://phabricator.wikimedia.org/T359594#9613493 (10dancy) p:05Triage→03High [21:34:55] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Now this 🫠), 13Patch-For-Review: Upgrade production gitlab-cloud-runners to kubernetes 1.29 - https://phabricator.wikimedia.org/T359594#9613550 (10CodeReviewBot) sandeeps opened https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requ... [21:52:57] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Now this 🫠), 13Patch-For-Review: Upgrade production gitlab-cloud-runners to kubernetes 1.29 - https://phabricator.wikimedia.org/T359594#9613584 (10dancy) Make sure https://gitlab.wikimedia.org/repos/releng/scap/-/pipelines/44180/builds completes succes... [22:19:09] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Now this 🫠), 13Patch-For-Review: Upgrade production gitlab-cloud-runners to kubernetes 1.29 - https://phabricator.wikimedia.org/T359594#9613605 (10dancy) >>! In T359594#9613584, @dancy wrote: > Make sure https://gitlab.wikimedia.org/repos/releng/scap/-...