[04:28:01] 10GitLab (CI & Job Runners): Allow variants of rustlang/rust:nightly images on GitLab CI - https://phabricator.wikimedia.org/T326515 (10Legoktm) [04:56:52] 10GitLab (CI & Job Runners), 10mwbot-rs: Unable to run Rust coverage reports in GitLab CI - https://phabricator.wikimedia.org/T325947 (10Legoktm) >>! In T325947#8490087, @Legoktm wrote: > Note to self to determine whether using alternative tools as described in https://www.collabora.com/news-and-blog/blog/2021... [08:16:38] PROBLEM - Work requests waiting in Zuul Gearman server on contint2001 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [400.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/d/000000322/zuul-gearman?orgId=1&viewPanel=10 [08:34:42] (Queue (Jenkins jobs + Zuul functions) alert) firing: Queue (Jenkins jobs + Zuul functions) alert - https://alerts.wikimedia.org/?q=alertname%3DQueue+%28Jenkins+jobs+%2B+Zuul+functions%29+alert [09:09:06] 10Continuous-Integration-Infrastructure, 10Jenkins: Upgrade Jenkins to latest LTS 2.375.1 - https://phabricator.wikimedia.org/T326531 (10hashar) [09:11:36] 10Gerrit, 10Release-Engineering-Team (GitLab III: GitLab in LA πŸͺƒ), 10SRE-OnFire, 10serviceops-collab, and 2 others: gerrit1001 running out of space on / - https://phabricator.wikimedia.org/T323262 (10hashar) Thanks @Jelto I will look at preparing patches for option B (//relocate Gerrit installation from... [09:20:53] (03CR) 10Hashar: layout.yaml: Update gate-and-submit-1_35 (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/876355 (https://phabricator.wikimedia.org/T274966) (owner: 10Reedy) [09:33:02] 10Project-Admins, 10Jornadas WMES: Create milestone for Wikimedia EspaΓ±a for WMES meeting 2023 - https://phabricator.wikimedia.org/T326532 (10Olea) [09:33:15] 10GitLab (CI & Job Runners), 10serviceops-collab: runner-1026.gitlab-runners.eqiad1.wikimedia.cloud ran out of disk space - https://phabricator.wikimedia.org/T326378 (10Jelto) p:05Triageβ†’03Medium Adjusting the watermarks and volume filter labels (thanks @Dzahn and @dancy ) fixed the cleanup of volumes and... [09:51:20] (03CR) 10Hashar: [C: 03+1] "Looks great! There are a few nitpicks here and there but overall the script looks extremely legit ;)" [tools/release] - 10https://gerrit.wikimedia.org/r/870944 (owner: 10Thcipriani) [10:02:30] 10Phabricator: Change the source repo field for #MediaWiki-extensions-ArrayFunctions - https://phabricator.wikimedia.org/T326534 (10Aklapper) [10:20:45] 10Project-Admins: Change the source repo field for #MediaWiki-extensions-ArrayFunctions - https://phabricator.wikimedia.org/T326534 (10Aklapper) 05Openβ†’03Resolved a:03Aklapper Thanks for the attention to details! Done. [10:34:33] !log Restarted zuul-merger on contint1002 it was listening but not processing any requests [10:34:34] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [10:45:23] 10Gerrit, 10Release-Engineering-Team, 10Sustainability (Incident Followup): Investigate dropping diff.h2.db and quota.repo_size.h2.db disk caches - https://phabricator.wikimedia.org/T324771 (10hashar) 05Openβ†’03Resolved a:03hashar On gerrit1001 and gerrit2002 I have removed from `/var/lib/gerrit2/review... [10:46:19] 10Gerrit, 10Release-Engineering-Team (GitLab III: GitLab in LA πŸͺƒ), 10SRE-OnFire, 10serviceops-collab, and 2 others: gerrit1001 running out of space on / - https://phabricator.wikimedia.org/T323262 (10hashar) [10:47:28] RECOVERY - Work requests waiting in Zuul Gearman server on contint2001 is OK: OK: Less than 100.00% above the threshold [200.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/d/000000322/zuul-gearman?orgId=1&viewPanel=10 [10:49:42] (Queue (Jenkins jobs + Zuul functions) alert) firing: (2) Queue (Jenkins jobs + Zuul functions) alert - https://alerts.wikimedia.org/?q=alertname%3DQueue+%28Jenkins+jobs+%2B+Zuul+functions%29+alert [10:53:11] 10GitLab (CI & Job Runners), 10serviceops-collab: runner-1026.gitlab-runners.eqiad1.wikimedia.cloud ran out of disk space - https://phabricator.wikimedia.org/T326378 (10LSobanski) a:03Jelto [10:55:58] 10Phabricator, 10Trust-and-Safety: Reset Phabricator 2FA for OTichonova - https://phabricator.wikimedia.org/T326390 (10Aklapper) 05Openβ†’03Resolved a:03Aklapper Sure! Stripped 2FA after verifying the involved accounts in this request. [11:09:42] (Queue (Jenkins jobs + Zuul functions) alert) resolved: Queue (Jenkins jobs + Zuul functions) alert - https://alerts.wikimedia.org/?q=alertname%3DQueue+%28Jenkins+jobs+%2B+Zuul+functions%29+alert [12:27:11] 10Beta-Cluster-Infrastructure, 10ContentTranslation: Special:ContentTranslation raise internal error on beta cluster - https://phabricator.wikimedia.org/T323417 (10YedidyaPopper) While using the translation tool for translating from English to Hebrew, I've tried to load saved unpublished translation, and met s... [12:40:16] 10Release-Engineering-Team (Priority Backlog πŸ“₯), 10MW-on-K8s, 10SRE, 10serviceops, 10Patch-For-Review: Automated validation of mediawiki-multiversion images - https://phabricator.wikimedia.org/T288629 (10LSobanski) [13:52:43] 10Phabricator (Upstream), 10Upstream: Unclear error message when uploading a larger attachment: "Exception: No configured storage engine can store this file." - https://phabricator.wikimedia.org/T155130 (10xcollazo) Folks, it seems like this is a longstanding (since 2017) and recurrent issue, as shown by the m... [14:04:25] 10Release-Engineering-Team (Radar), 10MediaWiki-Release-Tools, 10MW-1.35-release: mediawiki/extensions and mediawiki/skins missing the REL1_XX branches - https://phabricator.wikimedia.org/T264365 (10Jdforrester-WMF) Given the magic-updating sub-repo tech is gerrit-specific and we won't have it for GitLab, is... [14:11:31] 10Release-Engineering-Team: Runner runner-1030.gitlab-runners.eqiad1.wikimedia.cloud out of space - https://phabricator.wikimedia.org/T326558 (10xcollazo) [14:45:38] 10Gerrit, 10Release-Engineering-Team (Seen), 10SRE, 10serviceops-collab: Expand Gerrit Manager permissions - https://phabricator.wikimedia.org/T234474 (10LSobanski) [15:11:26] 10Release-Engineering-Team: Runner runner-1030.gitlab-runners.eqiad1.wikimedia.cloud out of space - https://phabricator.wikimedia.org/T326558 (10xcollazo) This happened again on `Runner: #318 (XpVY4EkJ) runner-1025.gitlab-runners.eqiad1.wikimedia.cloud`, so I now suspect it is a configuration issue in which fold... [15:22:28] (03CR) 10Hashar: [C: 03+1] "Sorry I hadn't notice this change. I have a few word of cautions. Since bats will be shipped by the Docker image, the version in use will" [integration/config] - 10https://gerrit.wikimedia.org/r/871178 (owner: 10David Caro) [15:25:44] (03CR) 10Hashar: [C: 04-1] zuul: Move Puppet tests from test-prop to test [integration/config] - 10https://gerrit.wikimedia.org/r/874926 (owner: 10Majavah) [15:49:28] 10GitLab (CI & Job Runners), 10serviceops-collab: runner-1026.gitlab-runners.eqiad1.wikimedia.cloud ran out of disk space - https://phabricator.wikimedia.org/T326378 (10dancy) @Jelto There's still an outstanding matter of tuning the size of buildkitd's cache (which exists inside of its container). For examp... [15:55:19] (03CR) 10Ahmon Dancy: [C: 03+2] webserver-base-image: Pass http(s)_proxy to docker build [tools/release] - 10https://gerrit.wikimedia.org/r/875932 (https://phabricator.wikimedia.org/T326351) (owner: 10Ahmon Dancy) [15:56:02] (03Merged) 10jenkins-bot: webserver-base-image: Pass http(s)_proxy to docker build [tools/release] - 10https://gerrit.wikimedia.org/r/875932 (https://phabricator.wikimedia.org/T326351) (owner: 10Ahmon Dancy) [15:57:46] 10Release-Engineering-Team: Gitlab runner image for building and extracting an artifact via a Dockerfile. - https://phabricator.wikimedia.org/T326569 (10xcollazo) [16:10:41] 10Phabricator, 10Release-Engineering-Team, 10Diffusion-Repository-Administrators, 10serviceops-collab: Audit Diffusion-Repository-Administrators group membership and rights - https://phabricator.wikimedia.org/T324171 (10LSobanski) a:03Dzahn [16:26:26] 10GitLab (Infrastructure), 10serviceops-collab, 10Patch-For-Review: Align and refactor GitLab backup scripts - https://phabricator.wikimedia.org/T326315 (10LSobanski) p:05Triageβ†’03Medium [16:31:24] 10Release-Engineering-Team, 10Wikimedia-Phabricator-Extensions, 10serviceops-collab: Disable "Browse Gerrit Projects" on https://phabricator.wikimedia.org/r/ - https://phabricator.wikimedia.org/T228507 (10LSobanski) p:05Triageβ†’03Low a:03Dzahn [16:34:34] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Priority Backlog πŸ“₯), 10serviceops-collab, 10Datacenter-Switchover: Create a runbook for switching CI master - https://phabricator.wikimedia.org/T256396 (10LSobanski) @hashar What are the next steps to consider this done? [16:50:38] 10Release-Engineering-Team, 10Wikimedia-GitHub: stop syncing and delete labs/private repo from github - https://phabricator.wikimedia.org/T315925 (10LSobanski) [16:53:19] 10Release-Engineering-Team (Seen), 10Epic, 10Technical-Debt: Figure out which production extensions have conflicting tests and fix them - https://phabricator.wikimedia.org/T238492 (10akosiaris) 05Openβ†’03Invalid No updates since 2020, I think we can close this as invalid. Feel free to reopen though. [16:53:26] 10Continuous-Integration-Config, 10Release-Engineering-Team (Seen), 10MW-on-K8s, 10Epic, 10Patch-For-Review: Have all Wikimedia production extensions and skins in the CI gate - https://phabricator.wikimedia.org/T249674 (10akosiaris) [16:53:29] 10Release-Engineering-Team (Seen), 10MW-on-K8s, 10Release Pipeline, 10serviceops-radar, 10Epic: Deploy MediaWiki to Wikimedia production in containers - https://phabricator.wikimedia.org/T238770 (10akosiaris) [17:00:21] 10Phabricator, 10Release-Engineering-Team, 10Patch-For-Review, 10Upstream: Unclear error message when uploading a larger attachment: "Exception: No configured storage engine can store this file." - https://phabricator.wikimedia.org/T155130 (10Dzahn) [17:01:07] 10Release-Engineering-Team (Seen), 10Epic, 10Technical-Debt: Figure out which production extensions have conflicting tests and fix them - https://phabricator.wikimedia.org/T238492 (10Jdforrester-WMF) 05Invalidβ†’03Open >>! In T238492#8509993, @akosiaris wrote: > No updates since 2020, I think we can close... [17:01:10] 10Continuous-Integration-Config, 10Release-Engineering-Team (Seen), 10MW-on-K8s, 10Epic, 10Patch-For-Review: Have all Wikimedia production extensions and skins in the CI gate - https://phabricator.wikimedia.org/T249674 (10Jdforrester-WMF) [17:42:53] 10Release-Engineering-Team (Priority Backlog πŸ“₯), 10Release, 10Train Deployments: 1.40.0-wmf.18 deployment blockers - https://phabricator.wikimedia.org/T325581 (10Zabe) [18:02:43] (Queue (Jenkins jobs + Zuul functions) alert) firing: Queue (Jenkins jobs + Zuul functions) alert - https://alerts.wikimedia.org/?q=alertname%3DQueue+%28Jenkins+jobs+%2B+Zuul+functions%29+alert [18:04:26] 10Release-Engineering-Team (Priority Backlog πŸ“₯), 10Release, 10Train Deployments: 1.40.0-wmf.18 deployment blockers - https://phabricator.wikimedia.org/T325581 (10kostajh) [18:07:43] (Queue (Jenkins jobs + Zuul functions) alert) firing: (2) Queue (Jenkins jobs + Zuul functions) alert - https://alerts.wikimedia.org/?q=alertname%3DQueue+%28Jenkins+jobs+%2B+Zuul+functions%29+alert [18:08:10] Hi folks! It's payments-wiki-upgrade season again. I'm finding the 1.35->1.39 upgrade a whole lot smoother than the last time around [18:08:21] We'll want to update the tests to run on php 7.4 images [18:09:02] and maybe we can even just run all of the tests on that image rather than maintaining two separate tracks like we did during the last upgrade [18:09:31] I've got a ticket for the work here: https://phabricator.wikimedia.org/T326372 with one wip zuulconfig patch attached [18:10:39] Would any releng-er have time this week to help? [18:27:43] (Queue (Jenkins jobs + Zuul functions) alert) resolved: Queue (Jenkins jobs + Zuul functions) alert - https://alerts.wikimedia.org/?q=alertname%3DQueue+%28Jenkins+jobs+%2B+Zuul+functions%29+alert [18:30:56] ejegg: As an aside, I'd love to get through some other tech debt stuff in CentralNotice and DonationInterface after the upgrade [18:31:36] James also left a comment on your gerrit patch a few hours ago offering to deploy ;) [18:48:01] (03PS1) 10Cwhite: Support OpenSearch-Dashboards 2.4.1 [releng/phatality] - 10https://gerrit.wikimedia.org/r/876371 (https://phabricator.wikimedia.org/T323287) [18:57:32] 10Project-Admins: Requests for addition to the #acl*Project-Admins group (in comments) - https://phabricator.wikimedia.org/T706 (10Aklapper) @Zache: Sorry that this didn't get picked up. I've added you! [19:12:34] thanks, Reedy ! I'll take a look at that note. We're hoping to ditch a good chunk of DonationInterface code soon, as we have moved a lot of business logic into a support library. [19:13:20] CentralNotice has been in a near-orphaned state for a couple of years - we're making the bare minimum of updates to keep it running, but we keep hoping to hand it off to another team [19:31:15] (03CR) 10Ejegg: [WIP] jjb: Add test jobs for fundraising/REL1_39 (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/876010 (https://phabricator.wikimedia.org/T326372) (owner: 10Ejegg) [19:51:49] ebernhardson: Found it. My goof. [19:54:10] (03PS1) 10Jforrester: Zuul: Fix extension-quibble-only-standalone to not run vendor jobs on branches [integration/config] - 10https://gerrit.wikimedia.org/r/877251 [19:55:32] (03CR) 10Jforrester: [C: 03+2] Zuul: Fix extension-quibble-only-standalone to not run vendor jobs on branches [integration/config] - 10https://gerrit.wikimedia.org/r/877251 (owner: 10Jforrester) [19:56:15] James_F: thanks! [19:56:38] (03Merged) 10jenkins-bot: Zuul: Fix extension-quibble-only-standalone to not run vendor jobs on branches [integration/config] - 10https://gerrit.wikimedia.org/r/877251 (owner: 10Jforrester) [19:56:57] !log Zuul: Fix extension-quibble-only-standalone to not run vendor jobs on branches [19:56:57] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:57:04] ebernhardson: Deployed. Sorry again! [19:58:43] (03PS4) 10Jforrester: jjb: Add test jobs for fundraising/REL1_39 [integration/config] - 10https://gerrit.wikimedia.org/r/876010 (https://phabricator.wikimedia.org/T326372) (owner: 10Ejegg) [20:00:24] (03PS1) 10Jforrester: Zuul: Add experimental jobs for fundraising/REL1_39 [integration/config] - 10https://gerrit.wikimedia.org/r/877253 (https://phabricator.wikimedia.org/T326372) [20:00:34] (03CR) 10Jforrester: [C: 03+2] "Deployed." [integration/config] - 10https://gerrit.wikimedia.org/r/876010 (https://phabricator.wikimedia.org/T326372) (owner: 10Ejegg) [20:01:41] (03Merged) 10jenkins-bot: jjb: Add test jobs for fundraising/REL1_39 [integration/config] - 10https://gerrit.wikimedia.org/r/876010 (https://phabricator.wikimedia.org/T326372) (owner: 10Ejegg) [20:17:46] thanks James_F [20:18:10] ejegg: Not deployed the Zuul change yet, in meetings sorry. [20:18:38] (03CR) 10Ahmon Dancy: [C: 03+2] Support OpenSearch-Dashboards 2.4.1 [releng/phatality] - 10https://gerrit.wikimedia.org/r/876371 (https://phabricator.wikimedia.org/T323287) (owner: 10Cwhite) [20:18:41] k, no rush [20:18:59] (03CR) 10Ahmon Dancy: [C: 03+2] "build-zip-using-docker worked on my machine." [releng/phatality] - 10https://gerrit.wikimedia.org/r/876371 (https://phabricator.wikimedia.org/T323287) (owner: 10Cwhite) [20:19:02] (03Merged) 10jenkins-bot: Support OpenSearch-Dashboards 2.4.1 [releng/phatality] - 10https://gerrit.wikimedia.org/r/876371 (https://phabricator.wikimedia.org/T323287) (owner: 10Cwhite) [20:57:09] 10Release-Engineering-Team (Priority Backlog πŸ“₯), 10Release, 10Train Deployments: 1.40.0-wmf.18 deployment blockers - https://phabricator.wikimedia.org/T325581 (10matmarex) ##### Risky Patch! πŸš‚πŸ”₯ * **Change**: https://gerrit.wikimedia.org/r/c/mediawiki/extensions/VisualEditor/+/870692 * **Summary**: This ch... [21:01:29] (03PS1) 10Ahmon Dancy: Update deployment notes and scripts [releng/phatality] - 10https://gerrit.wikimedia.org/r/876372 [21:02:33] (03PS2) 10Ahmon Dancy: Update deployment notes and scripts [releng/phatality] - 10https://gerrit.wikimedia.org/r/876372 [21:04:24] (03CR) 10Ahmon Dancy: [C: 03+2] "@Cwhite I built deploy/phatality-2.4.1.zip and added it to the repository. I'll leave it to you to deploy when you see fit." [releng/phatality] - 10https://gerrit.wikimedia.org/r/876371 (https://phabricator.wikimedia.org/T323287) (owner: 10Cwhite) [21:14:36] (03CR) 10Cwhite: Support OpenSearch-Dashboards 2.4.1 (031 comment) [releng/phatality] - 10https://gerrit.wikimedia.org/r/876371 (https://phabricator.wikimedia.org/T323287) (owner: 10Cwhite)