[00:00:38] (DatasourceError) firing: Queue (Jenkins jobs + Zuul functions) alert - https://grafana.wikimedia.org/alerting/grafana/iS0FSjJ4z/view - https://wikitech.wikimedia.org/wiki/Monitoring/DatasourceError - https://alerts.wikimedia.org/?q=alertname%3DDatasourceError [00:05:38] (DatasourceError) resolved: Queue (Jenkins jobs + Zuul functions) alert - https://grafana.wikimedia.org/alerting/grafana/iS0FSjJ4z/view - https://wikitech.wikimedia.org/wiki/Monitoring/DatasourceError - https://alerts.wikimedia.org/?q=alertname%3DDatasourceError [12:38:14] 10Continuous-Integration-Config, 10Release-Engineering-Team, 10Add-Link, 10Growth-Team, and 2 others: research/mwaddlink has failing CI on the main branch - https://phabricator.wikimedia.org/T357217 (10kevinbazira) Hi @Urbanecm_WMF, both you and I share the same concerns regarding model training matching m... [13:50:06] (03PS1) 10Amire80: Sort XML tags and attributes alphabetically [integration/config] - 10https://gerrit.wikimedia.org/r/1003759 [14:01:22] (03PS2) 10Jforrester: jjb: [mediawiki-i18n-check-docker] Sort alphabetically for readability [integration/config] - 10https://gerrit.wikimedia.org/r/1003759 (owner: 10Amire80) [14:02:12] (03CR) 10Jforrester: [C: 03+2] "Deployed." [integration/config] - 10https://gerrit.wikimedia.org/r/1003759 (owner: 10Amire80) [14:04:13] (03Merged) 10jenkins-bot: jjb: [mediawiki-i18n-check-docker] Sort alphabetically for readability [integration/config] - 10https://gerrit.wikimedia.org/r/1003759 (owner: 10Amire80) [14:04:54] 10Beta-Cluster-Infrastructure, 10Patch-For-Review: Emails sent from @wikimedia.beta.wmflabs.org do not reach intended recipients - https://phabricator.wikimedia.org/T343925 (10Ladsgroup) 05Open→03Resolved a:03Ladsgroup [14:06:20] 10GitLab (CI & Job Runners), 10collaboration-services: Create a special-purpose Trusted Runner with Dockerfile frontend - https://phabricator.wikimedia.org/T357612 (10Jelto) [14:06:45] 10Beta-Cluster-Infrastructure: Use a third party email service for beta cluster emails - https://phabricator.wikimedia.org/T356108 (10Ladsgroup) FWIW, I don't think this would be a good idea. Beside using propriety systems, it's not even really justified, fixing that bug was one day of work at most which is much... [14:18:12] does anyone know what’s going on with hudson.AbortException: castor-save-workspace-cache aborted. ? [14:18:24] I’ve had two gate-and-submit builds fail due to this error in the post build script now [14:18:46] https://integration.wikimedia.org/ci/job/mediawiki-quibble-selenium-vendor-mysql-php74-docker/31602/console and https://integration.wikimedia.org/ci/job/wmf-quibble-core-vendor-mysql-php74-docker/37978/console (both for https://gerrit.wikimedia.org/r/c/mediawiki/core/+/1003386) [14:19:48] 10Continuous-Integration-Config, 10I18n, 10Language-Technical Support (Language-Technical Support (Current) ), 10affects-translatewiki.net: Automatically allow tag in message translations - https://phabricator.wikimedia.org/T357670 (10Amire80) [14:20:18] seems to be tracked as T352319 [14:20:21] T352319: castor-save-workspace-cache aborted during postbuild - https://phabricator.wikimedia.org/T352319 [14:20:23] 10Continuous-Integration-Infrastructure, 10Jenkins, 10castor: castor-save-workspace-cache aborted during postbuild - https://phabricator.wikimedia.org/T352319 (10Lucas_Werkmeister_WMDE) Happened in two gate-and-submit builds ([1](https://integration.wikimedia.org/ci/job/mediawiki-quibble-selenium-vendor-mysq... [14:21:33] 10Continuous-Integration-Config, 10I18n, 10Language-Team (Language-2024-January-March), 10Language-Technical Support (Language-Technical Support (Current) ), 10affects-translatewiki.net: Automatically allow tag in message translations - https://phabricator.wikimedia.org/T357670 (10Amire80) [14:23:42] 10Continuous-Integration-Config, 10I18n, 10Language-Team (Language-2024-January-March), 10Language-Technical Support (Language-Technical Support (Current) ), 10affects-translatewiki.net: Automatically allow tag in message translations - https://phabricator.wikimedia.org/T357670 (10Amire80) a:03A... [14:24:35] (03PS1) 10Amire80: Change to in message XML tag checks [integration/config] - 10https://gerrit.wikimedia.org/r/1003764 (https://phabricator.wikimedia.org/T357670) [14:28:45] (03PS2) 10Jaime Nuche: jjb: overhaul Zuul deploy jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1003437 (https://phabricator.wikimedia.org/T342346) [14:42:57] 10Beta-Cluster-Infrastructure: Use a third party email service for beta cluster emails - https://phabricator.wikimedia.org/T356108 (10kostajh) 05Open→03Declined >>! In T356108#9546481, @Ladsgroup wrote: > FWIW, I don't think this would be a good idea. Beside using propriety systems, it's not even really just... [14:59:44] gate-and-submit seems to be currently blocked on https://integration.wikimedia.org/ci/job/quibble-vendor-mysql-php74-selenium-docker/93796/console which has been stuck since 14:17:20 [15:00:30] yikes [15:00:46] probably just abort that build then? [15:00:58] Sure. Done. [15:01:03] I think [15:01:05] it would’ve timed out in 10 minutes anyway, I think [15:01:09] yeah, I see the abort in the console now [15:01:36] I didn't realise I had permissions to abort jobs until I tried just now :) [15:05:06] heh [15:09:59] 10Release-Engineering-Team (Seen), 10Content-Transform-Team, 10MW-on-K8s, 10SRE, and 2 others: Create parsoid mediawiki deployment - https://phabricator.wikimedia.org/T357392 (10cscott) Seems fine, most parsoid traffic is probably happening on the main cluster these days anyway. We just want to make sure t... [15:31:36] 10GitLab (CI & Job Runners), 10Gitlab-Application-Security-Pipeline, 10Security Team AppSec, 10Security-Team, and 2 others: Investigate container scanning options within the context of the Gitlab appsec pipeline - https://phabricator.wikimedia.org/T307523 (10MoritzMuehlenhoff) >>! In T307523#9537237, @acoo... [16:42:18] (03PS3) 10Jaime Nuche: jjb: overhaul Zuul deploy jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1003437 (https://phabricator.wikimedia.org/T342346) [16:51:52] 10Release-Engineering-Team (Seen), 10Content-Transform-Team, 10MW-on-K8s, 10SRE, and 2 others: Create parsoid mediawiki deployment - https://phabricator.wikimedia.org/T357392 (10cscott) Also worth keeping in mind: https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/965608 -- in order to avoid in... [16:54:16] (03PS5) 10Jaime Nuche: Track wheels individually [integration/zuul/deploy] - 10https://gerrit.wikimedia.org/r/998933 (https://phabricator.wikimedia.org/T259611) (owner: 10Hashar) [16:54:18] (03PS6) 10Jaime Nuche: generate Python2 wheels for bullseye targets [integration/zuul/deploy] - 10https://gerrit.wikimedia.org/r/1002466 (https://phabricator.wikimedia.org/T342346) [16:58:33] 10Release-Engineering-Team (Radar), 10collaboration-services: Ensure that gitlab.wikimedia.org adheres to Google's sender guidelines - https://phabricator.wikimedia.org/T355776 (10LSobanski) [16:59:10] 10GitLab (Infrastructure), 10Release-Engineering-Team (Radar), 10collaboration-services: GitLab email confirmation mail ends up in spam folder - https://phabricator.wikimedia.org/T346607 (10LSobanski) 05Open→03Resolved Looking at the most recent comments, this should be OK to resolve, please reopen if yo... [17:20:11] 10Continuous-Integration-Config, 10Abstract Wikipedia team, 10Patch-For-Review: Explore impact of adding WikiLambda to the general MediaWiki CI gate - https://phabricator.wikimedia.org/T289741 (10Jdforrester-WMF) p:05Triage→03Low a:03Jdforrester-WMF [19:18:08] 10Release-Engineering-Team, 10Scap, 10Patch-For-Review: scap wikiversions-inuse and wikiversions-compile: Always use the staging directory - https://phabricator.wikimedia.org/T357581 (10dancy) p:05Triage→03Low [19:28:34] 10Release-Engineering-Team (Seen), 10Scap: `scap clean` failure - https://phabricator.wikimedia.org/T218783 (10hashar) The branch cleaning got moved behind a feature flag `scap clean --delete-gerrit-branch` which mean the wmf branches are never deleted and keep accumulating. I eventually wondered why they were... [19:30:08] 10Release-Engineering-Team (Onboarding 🚀): Delete wmf branches from Gerrit repositories - https://phabricator.wikimedia.org/T303828 (10hashar) The branch are no more cleaned since T218783 which moved the logic behind a feature flag (`scap clean --delete-gerrit-branch`) due to some weird breakage related to ssh a... [19:36:57] 10Release-Engineering-Team, 10Scap, 10Patch-For-Review: scap wikiversions-inuse and wikiversions-compile: Always use the staging directory - https://phabricator.wikimedia.org/T357581 (10CodeReviewBot) thcipriani merged https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/212 scap wikiversions-inu... [19:37:48] 10Release-Engineering-Team, 10Scap, 10Patch-For-Review: scap wikiversions-inuse and wikiversions-compile: Always use the staging directory - https://phabricator.wikimedia.org/T357581 (10dancy) 05Open→03Resolved [19:40:45] 10Release-Engineering-Team (Now this 🫠), 10Scap, 10Patch-For-Review: scap backport check prevents using it to fix an about-to-be-live branch - https://phabricator.wikimedia.org/T317795 (10CodeReviewBot) dancy updated https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/214 Add test case for backp... [19:43:53] 10Release-Engineering-Team (Now this 🫠), 10Scap, 10MW-on-K8s, 10SRE, 10serviceops: Scap should check errors coming from mw-on-k8s canaries during deployments - https://phabricator.wikimedia.org/T357402 (10dancy) p:05Triage→03Medium a:03dancy [20:13:58] 10Release-Engineering-Team (Priority Backlog 📥), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.42.0-wmf.18 deployment blockers - https://phabricator.wikimedia.org/T354436 (10brennen) [20:43:48] 10Release-Engineering-Team (Priority Backlog 📥), 10Release, 10Train Deployments, 10User-brennen: 1.42.0-wmf.18 deployment blockers - https://phabricator.wikimedia.org/T354436 (10thcipriani) [21:07:57] 10Continuous-Integration-Config, 10I18n, 10Language-Team (Language-2024-January-March), 10Language-Technical Support (Language-Technical Support (Current) ), and 2 others: Automatically allow tag in message translations - https://phabricator.wikimedia.org/T357670 (10Bugreporter) Note: the is... [21:27:51] 10Release-Engineering-Team (Seen), 10Scap: Run Swagger checks in Scap before exposing to prod MW traffic - https://phabricator.wikimedia.org/T239376 (10dancy) [21:29:08] 10Release-Engineering-Team, 10Scap: Improve scap canary check messages - https://phabricator.wikimedia.org/T142342 (10dancy) p:05Medium→03Low [21:49:42] 10Release-Engineering-Team, 10Scap, 10Patch-For-Review: Improve scap canary check messages - https://phabricator.wikimedia.org/T142342 (10CodeReviewBot) dancy opened https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/215 Mention how long we're waiting for canary traffic [21:51:16] 10Release-Engineering-Team, 10Scap, 10Patch-For-Review: Improve scap canary check messages - https://phabricator.wikimedia.org/T142342 (10CodeReviewBot) dancy updated https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/215 Mention how long we're waiting for canary traffic [21:52:54] 10Release-Engineering-Team, 10Scap, 10Patch-For-Review: Improve scap canary check messages - https://phabricator.wikimedia.org/T142342 (10CodeReviewBot) dancy merged https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/215 Mention how long we're waiting for canary traffic [21:53:52] 10Release-Engineering-Team, 10Scap, 10Patch-For-Review: Improve scap canary check messages - https://phabricator.wikimedia.org/T142342 (10dancy) [22:01:55] 10Release-Engineering-Team (Now this 🫠), 10Scap, 10Patch-For-Review: Improve scap canary check messages - https://phabricator.wikimedia.org/T142342 (10dancy) [22:12:59] 10Release-Engineering-Team (Priority Backlog 📥), 10Release, 10Train Deployments, 10User-brennen: 1.42.0-wmf.18 deployment blockers - https://phabricator.wikimedia.org/T354436 (10brennen) [22:15:55] 10Release-Engineering-Team (Priority Backlog 📥), 10Release, 10Train Deployments, 10User-brennen: 1.42.0-wmf.18 deployment blockers - https://phabricator.wikimedia.org/T354436 (10brennen) [22:22:01] 10Release-Engineering-Team (Priority Backlog 📥), 10Release, 10Train Deployments, 10User-brennen: 1.42.0-wmf.18 deployment blockers - https://phabricator.wikimedia.org/T354436 (10brennen) Noting here that there's a backcompat patch available for {T356368} if rolling wmf.18 back from group2 is necessary for... [22:23:56] 10Release-Engineering-Team (Priority Backlog 📥), 10Release, 10Train Deployments, 10User-brennen: 1.42.0-wmf.18 deployment blockers - https://phabricator.wikimedia.org/T354436 (10cscott) Yeah, just for clarity: 1003833 needs to be merged and deployed to wmf.17 before rolling back to wmf.17 or else we'll get...