[00:14:43] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Priority Backlog 📥), 10serviceops-collab, 10Datacenter-Switchover: Create a runbook for switching CI master - https://phabricator.wikimedia.org/T256396 (10Dzahn) [00:15:20] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Priority Backlog 📥), 10serviceops-collab, 10Datacenter-Switchover: Create a runbook for switching CI master - https://phabricator.wikimedia.org/T256396 (10Dzahn) @jijiki We can take this based on: hieradata/role/common/ci/master.yaml:pro... [00:16:44] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Priority Backlog 📥), 10serviceops-collab, 10Datacenter-Switchover: Create a runbook for switching CI master - https://phabricator.wikimedia.org/T256396 (10Dzahn) https://www.mediawiki.org/wiki/Continuous_integration/Data_center_switch exi... [00:17:48] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Priority Backlog 📥), 10serviceops-collab, 10Datacenter-Switchover: Create a runbook for switching CI master - https://phabricator.wikimedia.org/T256396 (10Dzahn) https://gerrit.wikimedia.org/r/q/topic:contint2002 is a topic branch that l... [00:27:16] 10Phabricator: enable DarkConsole in phabricator - https://phabricator.wikimedia.org/T11 (10Dzahn) it's funny and a reason to sigh that the comments above were added here, on T11, just because there is a timestamp like "2022-12-08-T11:00Z" in SAL I was first wondering why I get updates for this ticket from 2014. [10:13:24] 10Gerrit: Support OAuth for login onto gerrit.wikimedia.org - https://phabricator.wikimedia.org/T147864 (10akosiaris) [10:36:54] 10GitLab, 10Release-Engineering-Team, 10serviceops-collab: Align the GitLab runner tags - https://phabricator.wikimedia.org/T325069 (10Jelto) Shared and Trusted Runners have a additional tag `wmcs` and `trusted` now. The next step is to create MR to move from `protected` tag to `trusted` tag and adjust the [... [10:37:50] 10GitLab, 10Release-Engineering-Team, 10serviceops-collab: Align the GitLab runner tags - https://phabricator.wikimedia.org/T325069 (10Jelto) [11:33:12] 10GitLab, 10serviceops-collab: Optimize Gitlab Backups - https://phabricator.wikimedia.org/T324506 (10jcrespo) Alternatively, as discussed in the past, one thing that we could do is incremental backups in a raw way- if we do a filesystem snapshot (e.g. with LVM) and perform hourly incrementals, we could go bac... [11:51:35] (03CR) 10Hashar: "On top of the unconfiguration we also need to add the "archived" job template to the repository." [integration/config] - 10https://gerrit.wikimedia.org/r/868203 (https://phabricator.wikimedia.org/T323225) (owner: 10Jeena Huneidi) [11:52:36] 10Continuous-Integration-Config, 10Release-Engineering-Team (Priority Backlog 📥), 10MediaWiki-Core-Tests, 10Code-Health, and 8 others: Reduce runtime of MW shared gate Jenkins jobs to 5 min - https://phabricator.wikimedia.org/T225730 (10zeljkofilipin) [11:55:28] (03Abandoned) 10Hashar: build: manage dependencies with rules_jvm_external [software/gerrit/plugins/events-wikimedia] - 10https://gerrit.wikimedia.org/r/816172 (owner: 10Hashar) [11:55:30] (03Abandoned) 10Hashar: Json schema from Gerrit Java event classes [software/gerrit/plugins/events-wikimedia] - 10https://gerrit.wikimedia.org/r/830654 (https://phabricator.wikimedia.org/T304947) (owner: 10Hashar) [11:55:33] (03Abandoned) 10Hashar: Implement REST API and Ssh commands [software/gerrit/plugins/events-wikimedia] - 10https://gerrit.wikimedia.org/r/814725 (https://phabricator.wikimedia.org/T304947) (owner: 10Hashar) [11:55:36] (03Abandoned) 10Hashar: Send events to Wikimedia EventGate [software/gerrit/plugins/events-wikimedia] - 10https://gerrit.wikimedia.org/r/814807 (owner: 10Hashar) [13:02:38] (03PS1) 10Umherirrender: Zuul: [mediawiki/extensions/Variables] Use extension-broken [integration/config] - 10https://gerrit.wikimedia.org/r/868399 (https://phabricator.wikimedia.org/T250963) [13:05:34] (03CR) 10Clément Goubert: "This change is ready for review." [tools/release] - 10https://gerrit.wikimedia.org/r/868391 (https://phabricator.wikimedia.org/T288375) (owner: 10Clément Goubert) [13:06:00] (03CR) 10Clément Goubert: "This change is ready for review." [tools/release] - 10https://gerrit.wikimedia.org/r/868398 (owner: 10Clément Goubert) [13:59:16] 10Continuous-Integration-Config, 10Quality-and-Test-Engineering-Team (QTE), 10phpunit-patch-coverage: Most (all?) phpunit-patch-coverage jobs failing with "String could not be parsed as XML" instead of actually running - https://phabricator.wikimedia.org/T323139 (10Dreamy_Jazz) I've seen this error pretty co... [14:05:55] (03PS2) 10Dreamy Jazz: Make CentralAuth a clone dependency for MassMessage [integration/config] - 10https://gerrit.wikimedia.org/r/868193 (https://phabricator.wikimedia.org/T145430) [14:23:15] 10GitLab, 10Release-Engineering-Team, 10serviceops-collab: Align the GitLab runner tags - https://phabricator.wikimedia.org/T325069 (10Jelto) >>! In T325069#8468648, @bd808 wrote: > I know that I am a leading contributor to this problem, but "cloud" is pretty closely tied to [[https://wikitech.wikimedia.org/... [14:54:04] (03PS1) 10Hashar: wm-checks-api: add support for Puppet Catalogue Compiler [software/gerrit] (deploy/wmf/stable-3.5) - 10https://gerrit.wikimedia.org/r/868424 [14:58:10] (03CR) 10Hashar: "Hi John, this patch is for the Gerrit Checks API UI enhancement which I have deployed this week. The code should be able to recognize mess" [software/gerrit] (deploy/wmf/stable-3.5) - 10https://gerrit.wikimedia.org/r/868424 (owner: 10Hashar) [15:28:02] 10Release-Engineering-Team (Priority Backlog 📥), 10Patch-For-Review, 10Release, 10Train Deployments: 1.40.0-wmf.14 deployment blockers - https://phabricator.wikimedia.org/T320519 (10hashar) 05Open→03Resolved I am claiming the last MediaWiki train of the year 2022 to be a `SUCCESS`. Congratulations eve... [15:40:31] hashar: congrats on the completed train schedule of 2022. \o/ [15:42:06] \o/ [15:46:24] 10Beta-Cluster-Infrastructure: Request administrator role for user "Kindrobot" on en.wikipedia.beta.wmflabs.org - https://phabricator.wikimedia.org/T325301 (10SDunlap) [15:49:16] Hey, that's me! :) [15:51:08] 10Continuous-Integration-Config, 10PHP 8.2 support: CI composer-package-php82-docker needs newer composer/symfony components - https://phabricator.wikimedia.org/T325302 (10Umherirrender) [15:53:27] 10Continuous-Integration-Config, 10PHP 8.2 support: CI composer-package-php82-docker needs newer composer/symfony components - https://phabricator.wikimedia.org/T325302 (10Reedy) >Not sure what composer the package is using, but needs some updates 2.3.3 since {f03c29f2bf8e5477b3888b657834ae526c0f6a21} I woul... [16:02:52] greg-g: thanks, that was a nice team work ;) [16:02:58] + org + others [16:03:47] hashar: the best work :) [16:05:06] (03PS1) 10Ilias Sarantopoulos: inference-services: remove old revscoring pipelines [integration/config] - 10https://gerrit.wikimedia.org/r/868437 (https://phabricator.wikimedia.org/T323586) [16:13:07] (03CR) 10Elukey: [C: 03+1] inference-services: remove old revscoring pipelines [integration/config] - 10https://gerrit.wikimedia.org/r/868437 (https://phabricator.wikimedia.org/T323586) (owner: 10Ilias Sarantopoulos) [16:29:00] (03CR) 10Ahmon Dancy: build-image-incr.py: No new layer on empty rsync (031 comment) [tools/release] - 10https://gerrit.wikimedia.org/r/868398 (owner: 10Clément Goubert) [16:41:47] (03CR) 10Hashar: [C: 03+2] "I am not deleting the jobs since they still have results. They will eventually be manually garbage collected at some point in the future ;" [integration/config] - 10https://gerrit.wikimedia.org/r/868437 (https://phabricator.wikimedia.org/T323586) (owner: 10Ilias Sarantopoulos) [16:43:38] (03Merged) 10jenkins-bot: inference-services: remove old revscoring pipelines [integration/config] - 10https://gerrit.wikimedia.org/r/868437 (https://phabricator.wikimedia.org/T323586) (owner: 10Ilias Sarantopoulos) [16:51:41] !log Reloading Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/868437/ "inference-services: remove old revscoring pipelines" | T323586 [16:51:43] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:51:44] T323586: Reduce number of published docker images for revscoring models - https://phabricator.wikimedia.org/T323586 [16:51:56] (03CR) 10Hashar: [C: 03+2] "Deployed!" [integration/config] - 10https://gerrit.wikimedia.org/r/868437 (https://phabricator.wikimedia.org/T323586) (owner: 10Ilias Sarantopoulos) [17:16:26] (03CR) 10Hashar: [C: 03+2] "Removing Variables from the Arrays dependencies might break it as well. I am not sure what those extensions are but maybe they are candida" [integration/config] - 10https://gerrit.wikimedia.org/r/868399 (https://phabricator.wikimedia.org/T250963) (owner: 10Umherirrender) [17:18:15] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/Variables] Use extension-broken [integration/config] - 10https://gerrit.wikimedia.org/r/868399 (https://phabricator.wikimedia.org/T250963) (owner: 10Umherirrender) [17:20:11] !log Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/868399 [17:20:12] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:20:17] (03CR) 10Hashar: [C: 03+2] "deployed" [integration/config] - 10https://gerrit.wikimedia.org/r/868399 (https://phabricator.wikimedia.org/T250963) (owner: 10Umherirrender) [17:20:45] (03CR) 10Ahmon Dancy: "I'm going to do some experiments to see if this can be moved into make-container-image/multiversion-base/ where it seems more appropriate." [tools/release] - 10https://gerrit.wikimedia.org/r/868391 (https://phabricator.wikimedia.org/T288375) (owner: 10Clément Goubert) [18:11:06] (03PS1) 10Subramanya Sastry: Followup to 6fb18f41: DT adds mulitple comments now [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/868455 [18:28:33] (03CR) 10Ahmon Dancy: gitlab: train-dev has migrated to gitlab (031 comment) [tools/train-dev] - 10https://gerrit.wikimedia.org/r/868202 (https://phabricator.wikimedia.org/T323225) (owner: 10Jeena Huneidi) [18:38:50] (03CR) 10Ahmon Dancy: gitlab: train-dev has migrated to gitlab (031 comment) [tools/train-dev] - 10https://gerrit.wikimedia.org/r/868202 (https://phabricator.wikimedia.org/T323225) (owner: 10Jeena Huneidi) [18:53:32] 10GitLab, 10Release-Engineering-Team, 10serviceops-collab: Align the GitLab runner tags - https://phabricator.wikimedia.org/T325069 (10dduvall) >>! In T325069#8470932, @Jelto wrote: >>>! In T325069#8468648, @bd808 wrote: >> I know that I am a leading contributor to this problem, but "cloud" is pretty closely... [18:56:02] 10GitLab, 10Release-Engineering-Team, 10serviceops-collab: Align the GitLab runner tags - https://phabricator.wikimedia.org/T325069 (10dduvall) One more thought: Since the DigitalOcean runners will be instance wide starting in the new year and will be configured to run untagged jobs, I'm not sure their tags... [19:06:51] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Priority Backlog 📥), 10serviceops-collab: Move cloud runner CI jobs to trusted runners - https://phabricator.wikimedia.org/T322344 (10dduvall) We're now seeing errors during our [[ https://gitlab.wikimedia.org/repos/releng/gitlab-terraform-images/-/job... [19:08:14] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Priority Backlog 📥), 10serviceops-collab: Move cloud runner CI jobs to trusted runners - https://phabricator.wikimedia.org/T322344 (10dduvall) I'm thinking the best course for us at this point might be to rely on binaries verified by checksum for now a... [19:08:38] 10GitLab, 10Release-Engineering-Team, 10serviceops-collab: Align the GitLab runner tags - https://phabricator.wikimedia.org/T325069 (10brennen) > maybe let's be explicit using domain names and stop saying just "cloud" I think this or something like it is probably the best solution for @bd808's original conc... [19:16:18] (03PS1) 10Ahmon Dancy: multiversion-base: Include /usr/share/{GeoIP,GeoIPInfo} from host if available [tools/release] - 10https://gerrit.wikimedia.org/r/868122 (https://phabricator.wikimedia.org/T288375) [19:17:51] (03PS2) 10Ahmon Dancy: multiversion-base: Include /usr/share/{GeoIP,GeoIPInfo} from host if available [tools/release] - 10https://gerrit.wikimedia.org/r/868122 (https://phabricator.wikimedia.org/T288375) [19:19:40] (03CR) 10Ahmon Dancy: "This is an alternate for https://gerrit.wikimedia.org/r/c/mediawiki/tools/release/+/868391" [tools/release] - 10https://gerrit.wikimedia.org/r/868122 (https://phabricator.wikimedia.org/T288375) (owner: 10Ahmon Dancy) [19:20:03] (03CR) 10Ahmon Dancy: "Tested in train-dev." [tools/release] - 10https://gerrit.wikimedia.org/r/868122 (https://phabricator.wikimedia.org/T288375) (owner: 10Ahmon Dancy) [19:51:03] (03PS1) 10Hashar: deploy_artifacts: add dry run mode [software/gerrit] (deploy/wmf/stable-3.5) - 10https://gerrit.wikimedia.org/r/868461 [19:51:05] (03PS1) 10Hashar: deploy_artifacts: --version is a required option [software/gerrit] (deploy/wmf/stable-3.5) - 10https://gerrit.wikimedia.org/r/868462 [21:40:06] just a heads up that I'm seeing what at least looks like a significant slow down during this deploy — e.g. 2m+ to `sync-testservers`. Keeping a paste at https://phabricator.wikimedia.org/P42715 [21:48:20] 10Phabricator: Mass remove RhinosF1 from some tasks - https://phabricator.wikimedia.org/T325328 (10RhinosF1) [22:19:57] 10Phabricator: Mass remove RhinosF1 from some tasks - https://phabricator.wikimedia.org/T325328 (10TheresNoTime) For someone with phab shell access, the job has been [[ https://phabricator.wikimedia.org/daemon/bulk/view/3461/ | prepared here ]] (382 tasks not including this one), and can be run silently via `la... [22:23:28] TheresNoTime: Can you also include a copy of scap-image-build-and-push-log (from your home directory on deploy1002) in https://phabricator.wikimedia.org/P42715 ? [22:23:38] dancy: sure :) [22:24:35] oh, nevermind, I see the issue: 474 languages rebuilt out of 474 [22:25:31] would that cause a 2min testserver sync though> [22:25:37] s/>/? [22:26:05] Yeah. It results in ~2.6GB of data being regenerated and redeployed [22:26:14] oh! [22:26:17] wow [22:26:23] (mainly not "re"generated, since it's new data based on the new l10n files) [22:26:47] well TIL, thank you [22:27:49] dancy: why not only rebuild changed? [22:28:42] https://gerrit.wikimedia.org/r/c/mediawiki/extensions/GrowthExperiments/+/868364 changed en.json, which is the fallback language for all languages... so touching that effectively causes all langs to be rebuilt. [22:29:10] Ah [22:29:16] Of course [22:29:48] New message means all [22:30:00] Nod. Sadly [23:25:45] (03PS3) 10Jeena Huneidi: gitlab: train-dev has migrated to gitlab [tools/train-dev] - 10https://gerrit.wikimedia.org/r/868202 (https://phabricator.wikimedia.org/T323225) [23:26:04] (03CR) 10CI reject: [V: 04-1] gitlab: train-dev has migrated to gitlab [tools/train-dev] - 10https://gerrit.wikimedia.org/r/868202 (https://phabricator.wikimedia.org/T323225) (owner: 10Jeena Huneidi) [23:34:51] 10Phabricator: Mass remove RhinosF1 from some tasks - https://phabricator.wikimedia.org/T325328 (10Aklapper) Feel free to proceed: ` aklapper@phab1004:~$ sudo /srv/phab/phabricator/bin/bulk make-silent --id 3461 Configured job "3461" to run silently. ` [23:35:59] 10Phabricator: Mass remove RhinosF1 from some tasks - https://phabricator.wikimedia.org/T325328 (10TheresNoTime) 05Open→03In progress a:03TheresNoTime [23:37:36] 10Phabricator: Mass remove RhinosF1 from some tasks - https://phabricator.wikimedia.org/T325328 (10TheresNoTime) 05In progress→03Resolved https://phabricator.wikimedia.org/maniphest/query/VCbgmpshWGqE/#R now shows you subscribed to this task only @RhinosF1 :-) [23:54:09] 10Scap: Scap get stuck when trying to deploy a patch chain (from latest patch) - https://phabricator.wikimedia.org/T325074 (10jeena) So far I haven't reproduced this issue in our test environment. If it happens again please update with a copy of the output up until it breaks if you can.