[00:13:21] 10Continuous-Integration-Infrastructure, 10Zuul, 10Upstream: Change force merged cause a deadlock in Zuul gate-and-submit pipeline - https://phabricator.wikimedia.org/T93812 (10Krinkle) [00:13:23] 10Continuous-Integration-Infrastructure: Remove pywiki* and integration* from 'mediawiki' gate queue. - https://phabricator.wikimedia.org/T93304 (10Krinkle) [00:13:36] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Seen), 10Maintenance-Worktype: Re-evaluate use of "Dependent Pipeline" in Zuul for gate-and-submit - https://phabricator.wikimedia.org/T94322 (10Krinkle) 05Stalled→03Resolved a:03Antoine Mostly solved. Generally speaking, as long as y... [00:13:57] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Seen), 10Maintenance-Worktype: Re-evaluate use of "Dependent Pipeline" in Zuul for gate-and-submit - https://phabricator.wikimedia.org/T94322 (10Krinkle) a:05Antoine→03hashar [05:31:21] 10Phabricator, 10Release-Engineering-Team (Radar), 10DBA, 10Data-Persistence-Backup, and 2 others: phabricator->phorge migration - database handling - https://phabricator.wikimedia.org/T335080 (10Marostegui) >>! In T335080#9082181, @brennen wrote: >> Considering Manuel's and Jaime's availability could this... [05:31:36] 10Phabricator, 10Release-Engineering-Team (Radar), 10DBA, 10Data-Persistence-Backup, and 2 others: phabricator->phorge migration - database handling - https://phabricator.wikimedia.org/T335080 (10Marostegui) [05:57:14] 10Phabricator, 10Release-Engineering-Team (Radar), 10DBA, 10Data-Persistence-Backup, and 2 others: phabricator->phorge migration - database handling - https://phabricator.wikimedia.org/T335080 (10brennen) > If something explodes, we can simply promote that host to master on our proxies, as it will contain... [07:00:15] 10Phabricator, 10Release-Engineering-Team (Radar), 10DBA, 10Data-Persistence-Backup, and 2 others: phabricator->phorge migration - database handling - https://phabricator.wikimedia.org/T335080 (10jcrespo) Wednesday would be ideal for me, too, not only because I will be around but because we will also have... [08:44:55] 10Beta-Cluster-Infrastructure: Emails sent from @wikimedia.beta.wmflabs.org do not reach intended recipients - https://phabricator.wikimedia.org/T343925 (10Peachey88) [09:02:59] 10Continuous-Integration-Infrastructure, 10Jenkins, 10Release-Engineering-Team (Escape Goats🐐), 10Patch-For-Review: Add Scap3 config option to disable systemd services on secondary hosts - https://phabricator.wikimedia.org/T343447 (10CodeReviewBot) jnuche opened https://gitlab.wikimedia.org/repos/releng/sc... [09:29:26] 10GitLab (Pipeline Services Migration🐤), 10Research, 10collaboration-services: Move research webpages to gitlab - https://phabricator.wikimedia.org/T334511 (10Jelto) [09:51:14] 10Phabricator, 10Release-Engineering-Team (Radar), 10DBA, 10Data-Persistence-Backup, and 2 others: phabricator->phorge migration - database handling - https://phabricator.wikimedia.org/T335080 (10Marostegui) >>! In T335080#9082547, @brennen wrote: > >> @brennen apart from the date of the migration, are yo... [11:27:52] 10Release-Engineering-Team, 10Scap: scap backport sometimes crashes when running with two stacked patches - https://phabricator.wikimedia.org/T343967 (10taavi) [11:33:16] 10Continuous-Integration-Infrastructure, 10Jenkins, 10Release-Engineering-Team (Escape Goats🐐), 10Patch-For-Review: Add Scap3 config option to disable systemd services on secondary hosts - https://phabricator.wikimedia.org/T343447 (10jnuche) [12:04:36] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Escape Goats🐐), 10Patch-For-Review: Replace deprecated `CI_JOB_JWT` CI variable in Kokkuri - https://phabricator.wikimedia.org/T337474 (10CodeReviewBot) jnuche merged https://gitlab.wikimedia.org/repos/releng/jwt-authorizer/-/merge_requests/11 JWT iss... [12:12:07] 10GitLab (Auth & Access), 10Release-Engineering-Team (Radar), 10CAS-SSO, 10Infrastructure-Foundations, and 4 others: migrate gitlab away from the CAS protocol - https://phabricator.wikimedia.org/T320390 (10jbond) >>! In T320390#9068521, @Jelto wrote: > @jbond @SLyngshede-WMF do you have a idea how to chang... [12:39:16] 10GitLab (CI & Job Runners), 10Release-Engineering-Team: Consider adding sonarsource images to list of allowed images for gitlab runners - https://phabricator.wikimedia.org/T343975 (10pwangai) [13:15:27] 10GitLab (CI & Job Runners), 10Release-Engineering-Team: Consider adding sonarsource images to list of allowed images for gitlab runners - https://phabricator.wikimedia.org/T343975 (10Jelto) Thanks for opening the request! Adding @thcipriani and some RelEng folks for approval @brennen @dancy [13:16:27] 10GitLab (CI & Job Runners), 10Release-Engineering-Team, 10collaboration-services, 10Patch-For-Review: Consider adding sonarsource images to list of allowed images for gitlab runners - https://phabricator.wikimedia.org/T343975 (10Jelto) [13:24:43] 10Beta-Cluster-Infrastructure, 10IP Masking, 10Wikidata, 10wdwb-tech, and 2 others: [IPM] Enable temporary accounts (IP Masking) on Beta Wikidata - https://phabricator.wikimedia.org/T343980 (10Lucas_Werkmeister_WMDE) [13:26:58] 10GitLab (Auth & Access), 10Release-Engineering-Team (Radar), 10CAS-SSO, 10Infrastructure-Foundations, and 4 others: migrate gitlab away from the CAS protocol - https://phabricator.wikimedia.org/T320390 (10brennen) > But users may not want to have their full name (cn?) in GitLab displayed (for example Rand... [14:09:45] 10Beta-Cluster-Infrastructure, 10IP Masking, 10Wikidata, 10wdwb-tech, and 3 others: [IPM] Enable temporary accounts (IP Masking) on Beta Wikidata - https://phabricator.wikimedia.org/T343980 (10Lucas_Werkmeister_WMDE) 05Open→03Resolved a:03Lucas_Werkmeister_WMDE Seems to have worked fine: https://wiki... [14:22:55] 10GitLab (Pipeline Services Migration🐤), 10Research, 10collaboration-services, 10Patch-For-Review: Move research webpages to gitlab - https://phabricator.wikimedia.org/T334511 (10CodeReviewBot) jelto opened https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/40 add research mi... [14:25:21] 10GitLab (Pipeline Services Migration🐤), 10Research, 10collaboration-services, 10Patch-For-Review: Move research webpages to gitlab - https://phabricator.wikimedia.org/T334511 (10CodeReviewBot) jelto merged https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/40 add research mi... [14:40:10] 10Phabricator: Development environment with deployable Phab/Phorge - https://phabricator.wikimedia.org/T343989 (10jnuche) [15:04:25] (03PS1) 10Hnowlan: jjb: add entry for geo-analytics service [integration/config] - 10https://gerrit.wikimedia.org/r/947860 (https://phabricator.wikimedia.org/T336400) [15:05:34] (03PS1) 10Hnowlan: zuul: add geo-analytics configuration [integration/config] - 10https://gerrit.wikimedia.org/r/947861 (https://phabricator.wikimedia.org/T336400) [15:08:43] 10Phabricator, 10Release-Engineering-Team (Radar), 10DBA, 10Data-Persistence-Backup, and 2 others: phabricator->phorge migration - database handling - https://phabricator.wikimedia.org/T335080 (10brennen) > Wednesday would be ideal for me, too, not only because I will be around but because we will also hav... [15:12:10] (03CR) 10Jforrester: jjb: Use composer phpunit:entrypoint (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/803525 (https://phabricator.wikimedia.org/T90875) (owner: 10Kosta Harlan) [15:14:39] (03CR) 10Lucas Werkmeister (WMDE): jjb: Use composer phpunit:entrypoint (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/803525 (https://phabricator.wikimedia.org/T90875) (owner: 10Kosta Harlan) [15:46:50] ty brennen. I can see that image now [15:55:29] cool [16:22:04] 10Gitlab-Application-Security-Pipeline, 10Security Team AppSec, 10Security-Team, 10Patch-For-Review, and 3 others: Deprecate unused / problematic Gitlab AppSec CI includes - https://phabricator.wikimedia.org/T343262 (10CodeReviewBot) mmartorana merged https://gitlab.wikimedia.org/repos/security/gitlab-ci-s... [16:38:48] 10GitLab (Project Migration), 10collaboration-services: Migrate SRE repositories to GitLab - operations/software - https://phabricator.wikimedia.org/T341504 (10LSobanski) [16:41:42] (03CR) 10Ahmon Dancy: [C: 03+2] fab: Don't try to manipulate /tmp/docker-pkg-build.log [integration/config] - 10https://gerrit.wikimedia.org/r/946587 (owner: 10Ahmon Dancy) [16:43:14] (03Merged) 10jenkins-bot: fab: Don't try to manipulate /tmp/docker-pkg-build.log [integration/config] - 10https://gerrit.wikimedia.org/r/946587 (owner: 10Ahmon Dancy) [16:50:32] 10Release-Engineering-Team, 10Scap: scap backport sometimes crashes when running with two stacked patches - https://phabricator.wikimedia.org/T343967 (10dancy) Notes: https://gitlab.wikimedia.org/repos/releng/scap/-/blob/master/scap/plugins/backport.py#L646 The exception "Gerrit could not merge the change '%... [17:20:51] 10GitLab (CI & Job Runners), 10Release-Engineering-Team, 10collaboration-services, 10Patch-For-Review: Consider adding sonarsource images to list of allowed images for gitlab runners - https://phabricator.wikimedia.org/T343975 (10dancy) OK with me. [18:01:31] (03PS1) 10Sohom Datta: VideoCutTool: Use node18-docker to run tests [integration/config] - 10https://gerrit.wikimedia.org/r/947902 [18:20:44] 10Gitlab-Application-Security-Pipeline, 10Security Team AppSec, 10Security-Team, 10SecTeam-Processed, and 2 others: Deprecate unused / problematic Gitlab AppSec CI includes - https://phabricator.wikimedia.org/T343262 (10sbassett) 05Open→03Resolved p:05Triage→03Low [18:20:47] 10Gitlab-Application-Security-Pipeline, 10Security-Team, 10SecTeam-Processed, 10Security, 10user-sbassett: [EPIC] Application Security Pipeline Components for Gitlab - Phase 2 Work - https://phabricator.wikimedia.org/T342177 (10sbassett) [18:56:34] 10Release-Engineering-Team (Priority Backlog 📥), 10Scap, 10Data-Platform-SRE, 10Wikidata, 10Wikidata-Query-Service: wdqs: replace git-fat with git-lfs - https://phabricator.wikimedia.org/T316876 (10Gehel) [19:45:50] 10GitLab (CI & Job Runners), 10Release-Engineering-Team, 10collaboration-services, 10Patch-For-Review: Consider adding sonarsource images to list of allowed images for gitlab runners - https://phabricator.wikimedia.org/T343975 (10brennen) LGTM as well. [20:06:37] (Queue (Jenkins jobs + Zuul functions) alert) firing: - https://alerts.wikimedia.org/?q=alertname%3DQueue+%28Jenkins+jobs+%2B+Zuul+functions%29+alert [20:16:08] PROBLEM - Work requests waiting in Zuul Gearman server on contint2002 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [400.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/d/000000322/zuul-gearman?orgId=1&viewPanel=10 [20:22:46] Good lawd that's a lot of queued CI jobs :/ [20:26:37] (Queue (Jenkins jobs + Zuul functions) alert) resolved: - https://alerts.wikimedia.org/?q=alertname%3DQueue+%28Jenkins+jobs+%2B+Zuul+functions%29+alert [20:40:04] TheresNoTime: turn it off and on again [23:47:04] RECOVERY - Work requests waiting in Zuul Gearman server on contint2002 is OK: OK: Less than 100.00% above the threshold [200.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/d/000000322/zuul-gearman?orgId=1&viewPanel=10