[00:01:38] (DatasourceError) firing: Queue (Jenkins jobs + Zuul functions) alert - https://grafana.wikimedia.org/alerting/grafana/iS0FSjJ4z/view - https://wikitech.wikimedia.org/wiki/Monitoring/DatasourceError - https://alerts.wikimedia.org/?q=alertname%3DDatasourceError [00:06:38] (DatasourceError) resolved: Queue (Jenkins jobs + Zuul functions) alert - https://grafana.wikimedia.org/alerting/grafana/iS0FSjJ4z/view - https://wikitech.wikimedia.org/wiki/Monitoring/DatasourceError - https://alerts.wikimedia.org/?q=alertname%3DDatasourceError [06:09:53] 10Phabricator: Requesting for phabricator username change - https://phabricator.wikimedia.org/T356592 (10Tryvix1509) Hmm, I will choose “HideonRosie” for my Phab username. [07:19:31] (DatasourceError) firing: Queue (Jenkins jobs + Zuul functions) alert - https://grafana.wikimedia.org/alerting/grafana/iS0FSjJ4z/view - https://wikitech.wikimedia.org/wiki/Monitoring/DatasourceError - https://alerts.wikimedia.org/?q=alertname%3DDatasourceError [07:29:31] (DatasourceError) resolved: Queue (Jenkins jobs + Zuul functions) alert - https://grafana.wikimedia.org/alerting/grafana/iS0FSjJ4z/view - https://wikitech.wikimedia.org/wiki/Monitoring/DatasourceError - https://alerts.wikimedia.org/?q=alertname%3DDatasourceError [10:36:12] (03CR) 10Hashar: "https://gerrit.wikimedia.org/r/c/integration/config/+/792203 added ``, ``, `
` and `
`. Not sure why `
` was not includ" [integration/config] - 10https://gerrit.wikimedia.org/r/995373 (https://phabricator.wikimedia.org/T356548) (owner: 10Amire80) [10:37:16] (03CR) 10Hashar: [C: 03+2] Add dependency of MagicLinkAuthentication on PluggableAuth [integration/config] - 10https://gerrit.wikimedia.org/r/995367 (owner: 10Cicalese) [10:39:39] (03Merged) 10jenkins-bot: Add dependency of MagicLinkAuthentication on PluggableAuth [integration/config] - 10https://gerrit.wikimedia.org/r/995367 (owner: 10Cicalese) [12:14:35] 10Release-Engineering-Team (Priority Backlog 📥), 10Scap, 10MediaWiki-Platform-Team (Radar): "scap backport" unable to deploy change that has abandoned sibling in another branch - https://phabricator.wikimedia.org/T345304 (10CodeReviewBot) jnuche merged https://gitlab.wikimedia.org/repos/releng/scap/-/merge_r... [12:24:44] 10Release-Engineering-Team (Priority Backlog 📥), 10Scap, 10MediaWiki-Platform-Team (Radar): "scap backport" unable to deploy change that has abandoned sibling in another branch - https://phabricator.wikimedia.org/T345304 (10jnuche) 05Open→03Resolved Version 4.65.3 wit fix is deployed now [12:38:45] 10Phabricator, 10Release-Engineering-Team (Priority Backlog 📥), 10collaboration-services, 10Patch-For-Review, and 2 others: Many (all?) of the phabricator/tools scripts are in Python 2 - https://phabricator.wikimedia.org/T355574 (10LSobanski) p:05Triage→03Medium [13:26:51] 10GitLab (Pipeline Services Migration🐤), 10LibUp, 10Patch-For-Review: Migrate LibUp repositories to Wikimedia GitLab - https://phabricator.wikimedia.org/T341417 (10Jdforrester-WMF) Since this was filed, there's now a `mediawiki` ACL/repo group in GitLab – https://gitlab.wikimedia.org/groups/repos/mediawiki/-... [14:15:37] (03CR) 10Hashar: [C: 03+2] build: Updating npm dependencies [integration/docroot] - 10https://gerrit.wikimedia.org/r/996282 (owner: 10Libraryupgrader) [14:30:46] 10Gerrit, 10Release-Engineering-Team: Phase out All-Projects-In-Phabricator.git Gerrit project - https://phabricator.wikimedia.org/T355070 (10hashar) I have reparented the repositories: ` | |-- phabricator | |-- phabricator/antivandalism | |-- phabricator/arcanist | |-- phabricator/deployme... [14:35:45] (03CR) 10Hashar: [C: 03+2] Add rename-project plugin [software/gerrit] (wmf/stable-3.7) - 10https://gerrit.wikimedia.org/r/995035 (https://phabricator.wikimedia.org/T201953) (owner: 10Hashar) [14:37:50] (03PS1) 10Hashar: Add rename-project plugin [software/gerrit] (deploy/wmf/stable-3.7) - 10https://gerrit.wikimedia.org/r/997437 (https://phabricator.wikimedia.org/T201953) [14:39:20] 10Gerrit, 10SyntaxHighlight: Rename git repo to "SyntaxHighlight" - https://phabricator.wikimedia.org/T103614 (10hashar) [14:39:30] (03CR) 10Hashar: [C: 03+2] Add rename-project plugin [software/gerrit] (deploy/wmf/stable-3.7) - 10https://gerrit.wikimedia.org/r/997437 (https://phabricator.wikimedia.org/T201953) (owner: 10Hashar) [14:42:23] (03Merged) 10jenkins-bot: Add rename-project plugin [software/gerrit] (wmf/stable-3.7) - 10https://gerrit.wikimedia.org/r/995035 (https://phabricator.wikimedia.org/T201953) (owner: 10Hashar) [14:42:29] (03Merged) 10jenkins-bot: Add rename-project plugin [software/gerrit] (deploy/wmf/stable-3.7) - 10https://gerrit.wikimedia.org/r/997437 (https://phabricator.wikimedia.org/T201953) (owner: 10Hashar) [14:47:28] 10Gerrit, 10Release-Engineering-Team (Seen): Support renaming repositories in Gerrit - https://phabricator.wikimedia.org/T239693 (10hashar) [14:48:32] 10Gerrit, 10Release-Engineering-Team (Seen), 10Patch-For-Review: Install rename-project plugin - https://phabricator.wikimedia.org/T201953 (10hashar) 05Open→03Resolved a:03hashar I have build / deployed the plugin solving this task. The process to rename the repositories is to be defined in the parent... [14:49:32] 10Release-Engineering-Team (Seen), 10Gerrit (Gerrit 3.6): replication.ReplicationTasksStorage : Error while renaming task - https://phabricator.wikimedia.org/T354897 (10hashar) [15:02:07] 10Release-Engineering-Team (Seen), 10Gerrit (Gerrit 3.6): replication.ReplicationTasksStorage : Error while renaming task - https://phabricator.wikimedia.org/T354897 (10hashar) This also happened when deploying a plugin which causes Gerrit to reload the replication plugin and leads to the issue. On disk I can... [15:09:36] 10Gerrit: Gerrit checks display lists "merge failed" as success - https://phabricator.wikimedia.org/T356647 (10taavi) [15:21:22] (03PS2) 10Hashar: Zuul: [mediawiki/extensions/WikiLambda] Run selenium tests yet again [integration/config] - 10https://gerrit.wikimedia.org/r/994973 (owner: 10Jforrester) [15:21:33] (03CR) 10Hashar: [C: 03+1] Zuul: [mediawiki/extensions/WikiLambda] Run selenium tests yet again [integration/config] - 10https://gerrit.wikimedia.org/r/994973 (owner: 10Jforrester) [15:31:54] 10Continuous-Integration-Infrastructure: Add Python 3.10, 3.11 and 3.12 to Wikimedia CI - https://phabricator.wikimedia.org/T342019 (10Xqt) [15:33:57] 10Gerrit: Gerrit checks display lists "merge failed" as success - https://phabricator.wikimedia.org/T356647 (10hashar) Self note, the raw message is: ` lang=json { "id": "6f5ef26bd9e7d0490c37e499195c44d49a9284d9", "tag": "autogenerated:ci-test", "author": { "_account_id": 75, "name": "jenkins-bot",... [15:34:08] 10Gerrit: wmf-checks-api: Gerrit checks display lists "merge failed" as success - https://phabricator.wikimedia.org/T356647 (10hashar) [16:58:35] (Queue (Jenkins jobs + Zuul functions) alert) firing: - https://alerts.wikimedia.org/?q=alertname%3DQueue+%28Jenkins+jobs+%2B+Zuul+functions%29+alert [17:05:35] PROBLEM - Work requests waiting in Zuul Gearman server on contint2002 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [400.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/d/000000322/zuul-gearman?orgId=1&viewPanel=10 [17:08:35] (Queue (Jenkins jobs + Zuul functions) alert) resolved: - https://alerts.wikimedia.org/?q=alertname%3DQueue+%28Jenkins+jobs+%2B+Zuul+functions%29+alert [17:14:26] (03CR) 10Tchanders: [C: 03+1] Add BetaFeatures as a dependency for IPInfo [integration/config] - 10https://gerrit.wikimedia.org/r/995372 (https://phabricator.wikimedia.org/T356533) (owner: 10Dreamy Jazz) [17:23:33] 10GitLab (Account Approval), 10Release-Engineering-Team: Requesting GitLab account activation for Ahecht - https://phabricator.wikimedia.org/T356586 (10Legoktm) 05Open→03Resolved a:03Legoktm [17:24:14] any clues why pretty much all patches to labs/libraryupgrader are failing with an "unable to be automatically merged" error? (/cc James_F legoktm) [17:24:34] taavi: Probably a zuul-cloner bug. [17:25:39] 10Beta-Cluster-Infrastructure, 10MediaWiki-Platform-Team: Set up some beta cluster wikis with different registrable domain - https://phabricator.wikimedia.org/T355281 (10ArielGlenn) As I try to help plan this out, I have accumulated some questions. - Can we add the new domain right to the unified cert for ac... [17:27:48] ok, how do I fix that? [17:30:55] Migrate to GitLab to leave it behind? It's been a source of issues for years. [17:33:27] 10GitLab (Account Approval), 10Release-Engineering-Team: Requesting GitLab account activation for qwerfjkl - https://phabricator.wikimedia.org/T356663 (10Qwerfjkl) [17:33:56] I would, but CI config there is not self-service and lives in a repo I have no write access to (https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/62) [17:34:31] RECOVERY - Work requests waiting in Zuul Gearman server on contint2002 is OK: OK: Less than 100.00% above the threshold [200.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/d/000000322/zuul-gearman?orgId=1&viewPanel=10 [17:34:54] Trusted runners are only needed for doing scary things like pushing to prod docker-registry. [17:35:24] But anyway, I imagine someone from RelEng will approve that some time today/this week. [17:36:52] Taavi: I'll have a look at that MR [17:36:58] <3 [17:38:21] taavi: Given no-one can merge in the gerrit repo anyway… [17:39:11] 10GitLab (Pipeline Services Migration🐤), 10LibUp, 10Patch-For-Review: Migrate LibUp repositories to Wikimedia GitLab - https://phabricator.wikimedia.org/T341417 (10CodeReviewBot) dancy merged https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/62 Enable trusted runners for LibUp [17:40:52] thanks! [17:41:04] Taavi: https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/62 merged and deployed. [17:41:16] taavi: No new commits in https://gerrit.wikimedia.org/g/labs/libraryupgrader since the copy to https://gitlab.wikimedia.org/repos/ci-tools/libup so should we just close the gerrit repo? [17:41:38] James_F: yeah I think it can be archived. I'll send some patches to setup CI and image builds in GitLab [17:41:45] Awesome. [17:41:58] There's a script to auto-convert pipeline files to GitLab CI. [17:42:08] So theoretically it Might Just Work™. [17:43:04] https://gitlab.wikimedia.org/repos/releng/pipeline-to-gitlab [17:43:14] neat. let me try. [17:43:47] It's very basic [17:49:30] 10GitLab (Account Approval), 10Release-Engineering-Team: Requesting GitLab account activation for qwerfjkl - https://phabricator.wikimedia.org/T356663 (10bd808) 05Open→03Resolved a:03glaab The bot did the needful: https://wikitech.wikimedia.org/w/index.php?title=Tool:Gitlab-account-approval/Log&diff=prev... [18:19:11] 10GitLab (Pipeline Services Migration🐤), 10LibUp, 10Patch-For-Review: Migrate LibUp repositories to Wikimedia GitLab - https://phabricator.wikimedia.org/T341417 (10CodeReviewBot) taavi opened https://gitlab.wikimedia.org/repos/ci-tools/libup/-/merge_requests/1 Configure GitLab CI [18:22:23] hrmh I don't seem to be able to merge https://gitlab.wikimedia.org/repos/ci-tools/libup/-/merge_requests/1 [18:22:27] "Ready to merge by members who can write to the target branch." [18:22:46] and I am only a developer in /repos/ci-tools [18:26:22] taavi: I'll fix. [18:27:50] taavi: Try now. [18:28:26] much better [18:28:32] 10GitLab (Pipeline Services Migration🐤), 10LibUp, 10Patch-For-Review: Migrate LibUp repositories to Wikimedia GitLab - https://phabricator.wikimedia.org/T341417 (10CodeReviewBot) taavi merged https://gitlab.wikimedia.org/repos/ci-tools/libup/-/merge_requests/1 Configure GitLab CI [18:29:27] hm, it did not trigger a new image to be published [18:29:46] Hmm, indeed. [18:32:47] https://gitlab.wikimedia.org/repos/ci-tools/libup/-/merge_requests/2 may fix that [18:41:38] taavi: Also you can delete your gerrit Change-Id git hook for the LibUp repo. [18:41:55] good point [18:42:15] 10Release-Engineering-Team, 10Release Pipeline (Blubber): python: error: externally-managed-environment - https://phabricator.wikimedia.org/T356684 (10taavi) [18:43:24] taavi: #19 pushing manifest for registry.cloud.releng.team/repos/ci-tools/libup:master@sha256:f78e1885819e791a5cdb1cc9c2fec58aad1afd90807ed15480469848c67e9211 0.2s done [18:43:28] Yay. [18:43:44] hm, did that not have the /web label in the name? [18:44:14] 10GitLab (Misc), 10Release-Engineering-Team (Priority Backlog 📥), 10Documentation, 10User-brennen: Document GitLab Development Kit setup on-wiki - https://phabricator.wikimedia.org/T268269 (10dduvall) Just a note that I had the best luck with https://gitlab.com/gitlab-org/gitlab-compose-kit on a Debian sys... [18:44:44] Project beta-scap-sync-world build #141339: 04FAILURE in 9 min 31 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/141339/ [18:45:26] taavi: Oh, you'll need to set PUBLISH_IMAGE_NAME like https://gitlab.wikimedia.org/repos/abstract-wiki/wikifunctions/function-evaluator/-/blob/main/.gitlab-ci.yml?ref_type=heads#L82 if you want the image to have a different name from the repo. [18:46:18] ok, let me see [18:46:51] 10Release-Engineering-Team, 10Tech-Docs-Team, 10Documentation, 10User-brennen: Review critical-path deployment pipeline documentation - https://phabricator.wikimedia.org/T352259 (10dduvall) a:03dduvall Sorry for the delay on my side. Assigning to myself as I think I'm the only one that still needs to per... [18:47:45] attempt 3: https://gitlab.wikimedia.org/repos/ci-tools/libup/-/merge_requests/3 [18:50:31] Project beta-scap-sync-world build #141340: 04STILL FAILING in 1 min 56 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/141340/ [18:56:56] Project beta-scap-sync-world build #141341: 04STILL FAILING in 1 min 46 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/141341/ [18:57:42] 18:49:57 rsync: write failed on "/srv/mediawiki/php-master/.gitignore": No space left on device (28) [18:58:55] (On deployment-mediawiki12) [19:02:06] There's a lingering `/srv/mediawiki/php-master/cache/l10n/upstream/.~tmp~` directory from a failed rsync attempt. [19:02:18] 2.7GiB [19:02:57] dancy: Thanks. Deleted. [19:03:05] Overall I'd say the 20GB volume is too meager. [19:03:15] Now 86% used down from 100%. [19:03:16] Yeah. [19:04:49] !log jforrester@deployment-mediawiki12:/srv/mediawiki/php-master/cache/l10n/upstream$ sudo rm -rf .~tmp~/ # Clean up disk usage spotted by Dancy [19:04:51] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:06:15] 10GitLab (Misc), 10Release-Engineering-Team (Priority Backlog 📥), 10Documentation, 10User-brennen: Document GitLab Development Kit setup on-wiki - https://phabricator.wikimedia.org/T268269 (10brennen) 05Open→03Declined Yeah, I haven't touched GDK in ages, and I can't say this exactly feels pressing at... [19:08:27] Project beta-scap-sync-world build #141342: 04STILL FAILING in 1 min 49 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/141342/ [19:11:42] James_F: I'll add a volume for /srv [19:11:48] Ty. [19:16:59] Project beta-scap-sync-world build #141343: 04STILL FAILING in 1 min 46 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/141343/ [19:25:17] Ran out again; probably the run is what's filling the disk. [19:26:51] nod. [19:27:02] Workin' on the volume thing now. [19:27:08] Awesome. [19:28:14] what is registry.cloud.releng.team? [19:28:41] Project beta-scap-sync-world build #141344: 04STILL FAILING in 1 min 57 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/141344/ [19:34:44] !log Disable https://integration.wikimedia.org/ci/job/beta-scap-sync-world/ while dealing with storage space issues on deployment-mediawiki12.deployment-prep.eqiad1.wikimedia.cloud [19:34:46] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:39:11] taavi: That is a registry for storing temporary container images, used by the kubernetes gitlab-cloud-runners. [19:39:36] (mainly for caching) [19:40:00] ok, why is the libup job pushing things there and not to docker-registry.wikimedia.org? [19:40:19] ah I missed the trusted tag I think [19:40:27] :100 [19:51:36] 10Beta-Cluster-Infrastructure, 10MediaWiki-Platform-Team: Set up some beta cluster wikis with different registrable domain - https://phabricator.wikimedia.org/T355281 (10Tgr) >>! In T355281#9514060, @ArielGlenn wrote: > - Can we use the current cache-text08 to handle requests to wikis with the new domain, or... [19:56:16] Yippee, build fixed! [19:56:16] Project beta-scap-sync-world build #141345: 09FIXED in 4 min 1 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/141345/ [20:04:39] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team: Add a volume for /srv on deployment-mediawiki12.deployment-prep - https://phabricator.wikimedia.org/T356692 (10dancy) 05Open→03Resolved p:05Triage→03Medium a:03dancy [20:58:57] 10Release-Engineering-Team, 10Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10Fundraising Sprint: brb(): CiviCRM tests timing out in CI - https://phabricator.wikimedia.org/T356361 (10AKanji-WMF) 05Open→03Resolved a:03AKanji-WMF @Eileenmcnaughton resolved [21:24:17] (03PS1) 10QChris: Allow “Gerrit Managers” to import history [skins/MediaWikiWordpressThemes] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/997581 [21:24:19] (03CR) 10QChris: [V: 03+2 C: 03+2] Allow “Gerrit Managers” to import history [skins/MediaWikiWordpressThemes] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/997581 (owner: 10QChris) [21:29:22] (03PS1) 10QChris: Import done. Revoke import grants [skins/MediaWikiWordpressThemes] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/997583 [21:29:24] (03CR) 10QChris: [V: 03+2 C: 03+2] Import done. Revoke import grants [skins/MediaWikiWordpressThemes] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/997583 (owner: 10QChris) [23:30:22] Project beta-update-databases-eqiad build #73701: 04FAILURE in 10 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/73701/ [23:56:08] 10Release-Engineering-Team, 10Release Pipeline (Blubber): python: error: externally-managed-environment - https://phabricator.wikimedia.org/T356684 (10bd808) Updating Blubber's python support to do something like what is described at https://pythonspeed.com/articles/activate-virtualenv-dockerfile/#the-elegant-...