[00:05:10] (03CR) 10Thcipriani: [C:03+2] Decomissioning Article Summaries [integration/config] - 10https://gerrit.wikimedia.org/r/1217260 (https://phabricator.wikimedia.org/T411558) (owner: 10Kimberly Sarabia) [00:06:38] (03Merged) 10jenkins-bot: Decomissioning Article Summaries [integration/config] - 10https://gerrit.wikimedia.org/r/1217260 (https://phabricator.wikimedia.org/T411558) (owner: 10Kimberly Sarabia) [00:08:16] !log Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1217260 (Decomissioning Article Summaries) [00:08:17] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [05:44:35] 10Phabricator: MediaWiki-general has two default backlog columns - https://phabricator.wikimedia.org/T413157 (10Pppery) 03NEW [06:07:07] (03open) 10jelto: add wikipedia25 to Trusted Runners [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/138 (https://phabricator.wikimedia.org/T408592) [06:08:33] (03approved) 10jelto: add wikipedia25 to Trusted Runners [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/138 (https://phabricator.wikimedia.org/T408592) [06:08:39] (03merge) 10jelto: add wikipedia25 to Trusted Runners [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/138 (https://phabricator.wikimedia.org/T408592) [08:24:43] I'm seeing a new and strange failure in CI: https://integration.wikimedia.org/ci/job/quibble-vendor-mysql-php83/24745/console [08:25:30] it doesn't look like anything has changed recently in TestKitchen or MetricsPlatform, and I couldn't quickly see a relevant change in core. Has anyone seen something similar? [08:26:20] ah. But this could be it: https://gerrit.wikimedia.org/r/1218396 [08:31:18] (03PS1) 10Arthur taylor: Revert "Add references to Test Kitchen" [integration/config] - 10https://gerrit.wikimedia.org/r/1219728 [08:31:52] (03PS2) 10Arthur taylor: Revert "Add references to Test Kitchen" [integration/config] - 10https://gerrit.wikimedia.org/r/1219728 (https://phabricator.wikimedia.org/T413174) [08:32:01] 06Release-Engineering-Team (Radar), 06Infrastructure-Foundations, 06serviceops, 06SRE, and 2 others: Cross-datacenter Docker Registry replication broken since 2025-04-27 - https://phabricator.wikimedia.org/T413008#11475596 (10MatthewVernon) This morning, there are 1519692 objects in `docker_registry_codfw`... [08:33:48] 06Release-Engineering-Team (Doing 😎), 10Scap: Add scap wheels for Trixie - https://phabricator.wikimedia.org/T413102#11475601 (10elukey) 05In progressβ†’03Resolved @dancy works nicely, thanks! [08:39:54] anybody excited to +1 this revert so that we can redeploy Zuul and test again? https://gerrit.wikimedia.org/r/c/integration/config/+/1219728 [08:40:11] (or +2 I guess) [09:13:47] codders: lgtm, but I'm not sure who is around to deploy that (do you have rights for it?) [09:14:23] (03CR) 10Harroyo-wmf: [C:03+1] Revert "Add references to Test Kitchen" [integration/config] - 10https://gerrit.wikimedia.org/r/1219728 (https://phabricator.wikimedia.org/T413174) (owner: 10Arthur taylor) [09:14:28] I do technically have the rights - I've redeployed Zuul before. Would appreciate some more eyes on it though [09:19:52] codders: are you OK with the +1s you got or are you waiting for additional confirmation? [09:20:52] I mean, ideally I would hear from hashar or someone zuul-responsible. Do any of you have +2 on the repo? [09:20:54] hasharAway: FYI ^ [09:21:03] codders: to me the change looks fine [09:26:11] (03CR) 10Arthur taylor: [C:03+2] Revert "Add references to Test Kitchen" [integration/config] - 10https://gerrit.wikimedia.org/r/1219728 (https://phabricator.wikimedia.org/T413174) (owner: 10Arthur taylor) [09:27:42] (03Merged) 10jenkins-bot: Revert "Add references to Test Kitchen" [integration/config] - 10https://gerrit.wikimedia.org/r/1219728 (https://phabricator.wikimedia.org/T413174) (owner: 10Arthur taylor) [09:29:22] !log Reloading Zuul to deploy https://phabricator.wikimedia.org/T413174 [09:29:24] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [09:30:11] should be deployed now [09:34:46] codders: sobanski thank you for the fix :] [09:35:10] you're welcome! [09:41:52] 10Continuous-Integration-Config, 10Wikidata, 07ci-test-error (WMF-deployed Build Failure): Wikibase CI failing - "attempted to load MetricsPlatform twice" - https://phabricator.wikimedia.org/T413174#11475958 (10A_smart_kitten) [09:47:47] 10Continuous-Integration-Config, 06Test Kitchen, 10Wikidata, 07ci-test-error (WMF-deployed Build Failure): Wikibase CI failing - "attempted to load MetricsPlatform twice" - https://phabricator.wikimedia.org/T413174#11475999 (10hashar) It looks like `TestKitchen` is a fork of `MetricsPlatform` which has... [09:47:51] oh [09:47:55] I will file another task [09:55:40] filed at https://phabricator.wikimedia.org/T413194 :] [09:56:28] codders: thank you for the quick revert! [09:56:33] I am off again [10:41:28] 06Release-Engineering-Team (Radar), 06Infrastructure-Foundations, 06serviceops, 06SRE, and 2 others: Cross-datacenter Docker Registry replication broken since 2025-04-27 - https://phabricator.wikimedia.org/T413008#11476200 (10MatthewVernon) ` Dec 19 10:25:35 ms-be2081 container-sync: Since Fri Dec 19 09:25... [11:20:02] 10GitLab (Administration, Settings & Policy), 06Release-Engineering-Team (Priority Backlog πŸ“₯), 06cloud-services-team, 06collaboration-services: gitlab: consider enabling docker container registry - https://phabricator.wikimedia.org/T304845#11476305 (10Jelto) [12:20:03] maintenance-disconnect-full-disks build 764615 integration-agent-docker-1044 (/: 36%, /srv: 100%, /var/lib/docker: 37%): OFFLINE due to disk space [12:25:03] maintenance-disconnect-full-disks build 764616 integration-agent-docker-1044 (/: 36%, /srv: 13%, /var/lib/docker: 35%): RECOVERY disk space OK [12:50:34] 06Project-Admins: Create project tag for SourceWiki tool - https://phabricator.wikimedia.org/T412256#11476653 (10Aklapper) 05Openβ†’03Stalled [12:55:15] 06Project-Admins: Create project tag for SourceWiki tool - https://phabricator.wikimedia.org/T412256#11476660 (10Eugene233) I am tagging those concerned who asked for help with this request @Udehb @Dnshitobu [12:57:40] 10Gerrit, 10MediaWiki-Releasing, 10MediaWiki-General, 06MW-Interfaces-Team: Consider deleting the `list` Git tag from the mediawiki/core repository - https://phabricator.wikimedia.org/T413213 (10A_smart_kitten) 03NEW [13:07:54] 10Beta-Cluster-Infrastructure, 06Reader Experience Team: Add wordmark logo on beta cluster for Minerva (mobile) - https://phabricator.wikimedia.org/T413217 (10aude) 03NEW [13:30:10] 10Continuous-Integration-Infrastructure, 07Jenkins, 07Upstream: "Copy" button on Jenkins console output page copies "null" in clipboard - https://phabricator.wikimedia.org/T410633#11476786 (10zeljkofilipin) [14:32:28] FIRING: PuppetAgentFailure: Puppet agent failure detected on instance deployment-mx03 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [14:55:29] 06Project-Admins: Create project tag for SourceWiki tool - https://phabricator.wikimedia.org/T412256#11476981 (10Dnshitobu) Thank for the headsup but we will definitely need a tag here on Phabricator [15:38:24] 06Release-Engineering-Team (Priority Backlog πŸ“₯), 07Essential-Work, 05Release, 05Train Deployments: 1.46.0-wmf.7 deployment blockers - https://phabricator.wikimedia.org/T408277#11477096 (10dancy) 05Openβ†’03Resolved [16:00:41] (03update) 10dancy: train: Act as if testwikis are not part of group0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1056 (https://phabricator.wikimedia.org/T412868) [16:24:50] 06Project-Admins, 06Release-Engineering-Team: Remove ArticleSummaries from doc.wikimedia.org/cover-extensions - https://phabricator.wikimedia.org/T413232 (10mfossati) 03NEW [16:25:20] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team: Remove ArticleSummaries from doc.wikimedia.org/cover-extensions - https://phabricator.wikimedia.org/T413232#11477320 (10A_smart_kitten) [16:25:50] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team: Remove ArticleSummaries from doc.wikimedia.org/cover-extensions - https://phabricator.wikimedia.org/T413232#11477327 (10A_smart_kitten) [16:26:58] Is there more to setting up the pipelines of a new gitlab project than a `cp -a` of `.gitlab-ci.yml` and `.pipeline/*` from a comparable project? [16:27:56] That's pretty much it. [16:28:22] ok, what are the steps for when that doesn't work? [16:28:47] Feel free to message here or in Slack for support. I'm around today. [16:30:04] * Lucas_WMDE tries two random guesses at gitlab ci errors [16:30:17] make sure any needed CI variables are defined (could be on the repo or on a parent group) [16:30:49] and, if you’re experimenting with CI on a non-`main` branch, you might need to protect that branch (somewhere in the settings – branch rules) so that protected variables are available in its CI [16:31:01] (that last one bit me a few times) [16:32:53] 10Beta-Cluster-Infrastructure, 06collaboration-services: Puppet failure: "Unable to locate package spamd" on deployment-mx03.deployment-prep - https://phabricator.wikimedia.org/T412975#11477380 (10LSobanski) @dancy do you have any clues as to the need for this MX server? Also cc'ing @taavi as it was suggested... [16:45:03] maintenance-disconnect-full-disks build 764668 integration-agent-docker-1052 (/: 36%, /srv: 96%, /var/lib/docker: 38%): OFFLINE due to disk space [16:50:03] maintenance-disconnect-full-disks build 764669 integration-agent-docker-1052 (/: 36%, /srv: 48%, /var/lib/docker: 37%): RECOVERY disk space OK [16:52:11] 10Beta-Cluster-Infrastructure, 06collaboration-services: Puppet failure: "Unable to locate package spamd" on deployment-mx03.deployment-prep - https://phabricator.wikimedia.org/T412975#11477479 (10dancy) >>! In T412975#11477379, @LSobanski wrote: > @dancy do you have any clues as to the need for this MX server... [17:11:22] 10Beta-Cluster-Infrastructure, 06collaboration-services: Puppet failure: "Unable to locate package spamd" on deployment-mx03.deployment-prep - https://phabricator.wikimedia.org/T412975#11477568 (10thcipriani) There's been an MX in beta for over a decade and it's there because beta strives to be "production-lik... [17:14:41] 06Release-Engineering-Team (Priority Backlog πŸ“₯), 07Essential-Work, 05Release, 05Train Deployments: 1.46.0-wmf.7 deployment blockers - https://phabricator.wikimedia.org/T408277#11477581 (10cscott) [17:14:46] 06Project-Admins: Create project tag for SourceWiki tool - https://phabricator.wikimedia.org/T412256#11477582 (10Aklapper) Are you going to disable GitHub issues once there is a tag in Phabricator? [17:21:39] 10Beta-Cluster-Infrastructure, 06collaboration-services: Puppet failure: "Unable to locate package spamd" on deployment-mx03.deployment-prep - https://phabricator.wikimedia.org/T412975#11477609 (10MoritzMuehlenhoff) >>! In T412975#11477568, @thcipriani wrote: > @MoritzMuehlenhoff if @Dzahn 's workaround is les... [17:36:39] 06Release-Engineering-Team (Radar), 06Infrastructure-Foundations, 06serviceops, 06SRE, and 2 others: Cross-datacenter Docker Registry replication broken since 2025-04-27 - https://phabricator.wikimedia.org/T413008#11477686 (10MatthewVernon) This has continued to proceed today: ` Dec 19 17:26:37 ms-be2081 c... [18:06:56] 06Project-Admins: Create project tag for SourceWiki tool - https://phabricator.wikimedia.org/T412256#11477841 (10Dnshitobu) Yes [19:04:46] 10Beta-Cluster-Infrastructure, 06collaboration-services, 13Patch-For-Review: Puppet failure: "Unable to locate package spamd" on deployment-mx03.deployment-prep - https://phabricator.wikimedia.org/T412975#11478073 (10thcipriani) >>! In T412975#11477609, @MoritzMuehlenhoff wrote: >>>! In T412975#11477568, @th... [19:28:51] (03update) 10thcipriani: train: Act as if testwikis are not part of group0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1056 (https://phabricator.wikimedia.org/T412868) (owner: 10dancy) [19:32:19] (03merge) 10thcipriani: train: Act as if testwikis are not part of group0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1056 (https://phabricator.wikimedia.org/T412868) (owner: 10dancy) [20:11:43] (03open) 10dancy: Drop `scap test-train` [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1062 [20:11:47] (03update) 10dancy: Drop `scap test-train` [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1062 [20:14:44] (03update) 10dancy: Drop `scap test-train` [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1062 [20:19:24] (03merge) 10thcipriani: Drop `scap test-train` [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1062 (owner: 10dancy) [21:07:49] 06Project-Admins: Create project tag for SourceWiki tool - https://phabricator.wikimedia.org/T412256#11478516 (10Peachey88) 05Stalledβ†’03Resolved a:03Peachey88 Project created, please feel free to encourage those interested in the project to watch it to receive automated alerts from phabricator on new o... [21:11:48] 06Project-Admins, 10SourceWiki: Create project tag for SourceWiki tool - https://phabricator.wikimedia.org/T412256#11478579 (10Peachey88) [23:34:17] 06Release-Engineering-Team (Doing 😎), 10Catalyst (Luka Moli Telo), 07OKR-Work: Wikifarms: CampaignEvents wikifarm features should be enabled when we create a wikifarm - https://phabricator.wikimedia.org/T408832#11479158 (10thcipriani) [23:39:34] 10GitLab (Upstream pit of despair πŸ•³οΈ), 10Phabricator (2025-12-16), 06Release-Engineering-Team (Doing 😎), 10Wikimedia-Phabricator-Extensions, 07Essential-Work: 'Related GitLab patches' box includes patches for unrelated tasks (that begin with the current... - https://phabricator.wikimedia.org/T395728#11479177 [23:42:24] 10GitLab, 06Release-Engineering-Team, 06collaboration-services, 07Essential-Work: gitlab: Make @daniel an owner of repos/mediawiki/services/smoke-alarm - https://phabricator.wikimedia.org/T411502#11479178 (10thcipriani)