[02:10:25] 10GitLab (Project Migration), 10Release-Engineering-Team: Create new GitLab project group: research - https://phabricator.wikimedia.org/T300074 (10Aklapper) Hi @fab, could you maybe [connect](https://phabricator.wikimedia.org/settings/panel/external/) your [SUL account](https://meta.wikimedia.org/wiki/User:FKa... [02:22:55] 10Phabricator, 10SecTeam-Processed, 10Security: Audit members of acl*security for more than x duration of no activity (Jan 2022) - https://phabricator.wikimedia.org/T299400 (10Aklapper) >>! In T299400#7649830, @Dsharpe wrote: > Because I couldn't think of a better way to do it, I added a personal calendar en... [03:07:45] 10Scap, 10Patch-For-Review: Make 'scap update-interwiki-cache' less scary - https://phabricator.wikimedia.org/T247107 (10Krinkle) [03:09:52] (03CR) 10Krinkle: [C: 03+1] updateinterwikicache: Do not commit+sync [tools/scap] - 10https://gerrit.wikimedia.org/r/757062 (https://phabricator.wikimedia.org/T247107) (owner: 10Majavah) [03:10:25] (03CR) 10Krinkle: [C: 03+1] "I no longer trust myself to (quickly) test and review scap code, but thanks for picking it up." [tools/scap] - 10https://gerrit.wikimedia.org/r/757062 (https://phabricator.wikimedia.org/T247107) (owner: 10Majavah) [05:26:43] 10Release-Engineering-Team (Next), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.38.0-wmf.19 deployment blockers - https://phabricator.wikimedia.org/T293960 (10Krinkle) Regression in wmf.19: {T300096} [07:11:45] 10Release-Engineering-Team (Next), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.38.0-wmf.19 deployment blockers - https://phabricator.wikimedia.org/T293960 (10Legoktm) [07:14:41] 10Release-Engineering-Team (Next), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.38.0-wmf.19 deployment blockers - https://phabricator.wikimedia.org/T293960 (10Legoktm) I added {T300100} as a UBN blocker, and identified the core change which should probably be reverted as an unann... [07:18:00] 10Release-Engineering-Team (Next), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.38.0-wmf.19 deployment blockers - https://phabricator.wikimedia.org/T293960 (10Legoktm) [08:02:21] (03CR) 10Kosta Harlan: [C: 03+2] Clean-up: remove old parallel_run [integration/quibble] - 10https://gerrit.wikimedia.org/r/587890 (owner: 10Awight) [08:02:39] (03PS5) 10Kosta Harlan: phpbench: Support aggregate reports [integration/quibble] - 10https://gerrit.wikimedia.org/r/741974 (https://phabricator.wikimedia.org/T291549) [08:10:25] (03PS1) 10Kosta Harlan: ci-full-run: Add an extension to full run [integration/quibble] - 10https://gerrit.wikimedia.org/r/757381 [08:14:26] (03CR) 10jerkins-bot: [V: 04-1] ci-full-run: Add an extension to full run [integration/quibble] - 10https://gerrit.wikimedia.org/r/757381 (owner: 10Kosta Harlan) [08:18:35] (03Merged) 10jenkins-bot: Clean-up: remove old parallel_run [integration/quibble] - 10https://gerrit.wikimedia.org/r/587890 (owner: 10Awight) [08:22:51] (03PS2) 10Kosta Harlan: ci-full-run: Add an extension to full run [integration/quibble] - 10https://gerrit.wikimedia.org/r/757381 [08:26:44] (03CR) 10jerkins-bot: [V: 04-1] ci-full-run: Add an extension to full run [integration/quibble] - 10https://gerrit.wikimedia.org/r/757381 (owner: 10Kosta Harlan) [08:27:42] (03CR) 10Awight: [C: 03+2] "Worth noting that our full-run will have to be either core or an extension, and the code paths diverge in many places. But adding an exte" [integration/quibble] - 10https://gerrit.wikimedia.org/r/757381 (owner: 10Kosta Harlan) [08:31:24] (03CR) 10jerkins-bot: [V: 04-1] ci-full-run: Add an extension to full run [integration/quibble] - 10https://gerrit.wikimedia.org/r/757381 (owner: 10Kosta Harlan) [08:42:37] (03PS3) 10Kosta Harlan: ci-full-run: Add an extension to full run [integration/quibble] - 10https://gerrit.wikimedia.org/r/757381 [08:49:02] (03CR) 10jerkins-bot: [V: 04-1] ci-full-run: Add an extension to full run [integration/quibble] - 10https://gerrit.wikimedia.org/r/757381 (owner: 10Kosta Harlan) [08:56:18] 10Release-Engineering-Team, 10Scap, 10serviceops: Deploy Scap version 4.2.0 - https://phabricator.wikimedia.org/T300058 (10JMeybohm) a:03JMeybohm [08:59:38] (03PS4) 10Kosta Harlan: ci-full-run: Add an extension to full run [integration/quibble] - 10https://gerrit.wikimedia.org/r/757381 [09:04:44] (03PS5) 10Kosta Harlan: ci-full-run: Add an extension to full run [integration/quibble] - 10https://gerrit.wikimedia.org/r/757381 [09:09:32] (03CR) 10jerkins-bot: [V: 04-1] ci-full-run: Add an extension to full run [integration/quibble] - 10https://gerrit.wikimedia.org/r/757381 (owner: 10Kosta Harlan) [09:14:06] (03PS6) 10Kosta Harlan: [WIP] ci-full-run: Add an extension to full run [integration/quibble] - 10https://gerrit.wikimedia.org/r/757381 [09:21:29] (03CR) 10jerkins-bot: [V: 04-1] [WIP] ci-full-run: Add an extension to full run [integration/quibble] - 10https://gerrit.wikimedia.org/r/757381 (owner: 10Kosta Harlan) [09:24:24] (03PS7) 10Kosta Harlan: [WIP] ci-full-run: Add extensions to full run [integration/quibble] - 10https://gerrit.wikimedia.org/r/757381 [09:31:21] 10Continuous-Integration-Infrastructure, 10Jenkins, 10Quibble: integration-quibble-funrun job console does not have collapsible sections - https://phabricator.wikimedia.org/T300112 (10hashar) [09:32:05] 10Quibble: Quibble concurrent capture tests fail on MacOS - https://phabricator.wikimedia.org/T299840 (10awight) p:05Triage→03Low [09:32:18] 10Continuous-Integration-Infrastructure, 10Jenkins, 10Quibble: integration-quibble-funrun job console does not have collapsible sections - https://phabricator.wikimedia.org/T300112 (10awight) p:05Triage→03Low [09:32:27] 10Release-Engineering-Team, 10Scap, 10serviceops: Deploy Scap version 4.2.0 - https://phabricator.wikimedia.org/T300058 (10JMeybohm) Canaries have been upgraded, did a `scap pull` from mwdebug1001 which seemed fine. Updating `A:restbase-canary` now as well, rest tomorrow. [09:33:36] 10Continuous-Integration-Infrastructure, 10Jenkins, 10Quibble: integration-quibble-funrun job console does not have collapsible sections - https://phabricator.wikimedia.org/T300112 (10hashar) a:03hashar The reason is that most of the jobs have Quibble to autodetect coloring and they end up with no coloring... [09:35:12] 10Quibble, 10Unplanned-Sprint-Work, 10User-awight, 10WMDE-TechWish-Sprint-2022-01-19: Use interruptable parallelism - https://phabricator.wikimedia.org/T234309 (10awight) 05Open→03Resolved a:03awight [09:36:14] 10Quibble, 10Unplanned-Sprint-Work, 10User-awight, 10WMDE-TechWish-Sprint-2022-01-19: Use interruptable parallelism - https://phabricator.wikimedia.org/T234309 (10awight) 05Resolved→03Open [09:36:26] 10Quibble, 10Unplanned-Sprint-Work, 10User-awight, 10WMDE-TechWish-Sprint-2022-01-19: Use interruptable parallelism - https://phabricator.wikimedia.org/T234309 (10awight) a:05awight→03None [09:38:10] (03CR) 10jerkins-bot: [V: 04-1] [WIP] ci-full-run: Add extensions to full run [integration/quibble] - 10https://gerrit.wikimedia.org/r/757381 (owner: 10Kosta Harlan) [09:52:26] 10Continuous-Integration-Infrastructure, 10Jenkins, 10Quibble: integration-quibble-funrun job console does not have collapsible sections - https://phabricator.wikimedia.org/T300112 (10hashar) 05Open→03Resolved Somehow `\u001B` is not recognized so I went with `.`, in the end I have applied: | Start | ^(... [09:52:40] 10Quibble: Switch QUnit tests to use Apache backend - https://phabricator.wikimedia.org/T299491 (10awight) Noting that some qunit tests e.g. in MediaWiki extensions are already using Apache, but at least the mediawiki/core repo is still launching a PHP standalone server. [10:09:22] (03CR) 10Awight: "This change is ready for review." [integration/config] - 10https://gerrit.wikimedia.org/r/757393 (https://phabricator.wikimedia.org/T299491) (owner: 10Awight) [10:10:39] 10Quibble, 10Patch-For-Review, 10Unplanned-Sprint-Work, 10WMDE-TechWish-Sprint-2022-01-19: Switch QUnit tests to use Apache backend - https://phabricator.wikimedia.org/T299491 (10awight) [10:10:58] (03CR) 10Kosta Harlan: [C: 03+1] Switch QUnit tests to use the apache backend [integration/config] - 10https://gerrit.wikimedia.org/r/757393 (https://phabricator.wikimedia.org/T299491) (owner: 10Awight) [10:12:27] 10Continuous-Integration-Config, 10Quibble, 10MediaWiki-Core-Tests, 10Code-Health, 10Patch-For-Review: Quibble: Run PHPUnit databaseless and database stages in parallel - https://phabricator.wikimedia.org/T235449 (10awight) a:05awight→03None [10:18:20] 10Quibble: Quibble should backup and delete LocalSettings.php - https://phabricator.wikimedia.org/T218647 (10awight) >>! In T218647#5640971, @Niedzielski wrote: > I've been away for awhile so please excuse this comment if it doesn't make sense, but did you consider a "`--deleteLocalSettings`" (or similar) flag?... [10:22:22] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 10Quibble, 10Patch-For-Review: Run linters before starting longer running jobs - https://phabricator.wikimedia.org/T297561 (10awight) Should this task be stalled, pending the discussion in https://gerrit.wikimedia.org/r/c/integration... [10:25:24] hashar: perhaps quibble gerrit should only appear in one channel (the other)? [10:25:42] awight: +1 [10:26:42] awight: the config was added by https://gerrit.wikimedia.org/r/c/labs/tools/wikibugs2/+/734561/3/channels.yaml [10:27:07] that is for Phabricator notifications [10:27:14] and the other filel https://gerrit.wikimedia.org/r/c/labs/tools/wikibugs2/+/734561/3/gerrit-channels.yaml is for gerrit notifications [10:27:20] releng has a catch all for integration/.* [10:27:33] and thus get notifications for integration/quibble [10:28:14] I am messing up with bash and trap SIGTERM :( [10:28:18] my trap does not work for some reason bah [10:37:14] hashar: ty, I'll patch those wikibugs2 configs shortly. [10:44:16] 10Quibble, 10Patch-For-Review, 10User-zeljkofilipin: Run Parsoid service in quibble - https://phabricator.wikimedia.org/T218534 (10awight) 05Open→03Resolved Looks like this is done. [11:09:21] 10Quibble: Quibble initialize step should only clone the target repository - https://phabricator.wikimedia.org/T211702 (10awight) 05Open→03Stalled Deploying a naive patch for this caused problems described in the revert, e19c2a96e998d. The implementation needs to be planned better before trying again. Mayb... [11:10:59] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 10Quibble, 10Patch-For-Review: Run linters before starting longer running jobs - https://phabricator.wikimedia.org/T297561 (10kostajh) >>! In T297561#7651941, @awight wrote: > Should this task be stalled, pending the discussion in ht... [11:11:04] 10Quibble: Quibble initialize step should only clone the target repository - https://phabricator.wikimedia.org/T211702 (10awight) It's still a motivating problem: cloning all dependencies takes 50 seconds of early startup, which must finish before any tests can be run. [11:14:54] 10Continuous-Integration-Infrastructure, 10Jenkins, 10Release-Engineering-Team, 10Upstream: Cancelled Jenkins build do not run postbuild script - https://phabricator.wikimedia.org/T299995 (10hashar) https://issues.jenkins.io/browse/JENKINS-26994 is from 2015 and looks very similar. A comment by one of Jenk... [11:22:34] (03PS1) 10Kosta Harlan: [WIP] Run post-dependency install, pre-test steps in parallel [integration/quibble] - 10https://gerrit.wikimedia.org/r/757411 [11:25:35] (03CR) 10jerkins-bot: [V: 04-1] [WIP] Run post-dependency install, pre-test steps in parallel [integration/quibble] - 10https://gerrit.wikimedia.org/r/757411 (owner: 10Kosta Harlan) [11:37:56] (03PS2) 10Kosta Harlan: [WIP] Run post-dependency install, pre-test steps in parallel [integration/quibble] - 10https://gerrit.wikimedia.org/r/757411 [11:42:43] (03CR) 10jerkins-bot: [V: 04-1] [WIP] Run post-dependency install, pre-test steps in parallel [integration/quibble] - 10https://gerrit.wikimedia.org/r/757411 (owner: 10Kosta Harlan) [11:50:08] (03PS3) 10Kosta Harlan: [WIP] Run post-dependency install, pre-test steps in parallel [integration/quibble] - 10https://gerrit.wikimedia.org/r/757411 [11:54:22] (03CR) 10jerkins-bot: [V: 04-1] [WIP] Run post-dependency install, pre-test steps in parallel [integration/quibble] - 10https://gerrit.wikimedia.org/r/757411 (owner: 10Kosta Harlan) [12:02:17] (03CR) 10Kosta Harlan: "The error seems to be:" [integration/quibble] - 10https://gerrit.wikimedia.org/r/757411 (owner: 10Kosta Harlan) [12:26:09] (03PS8) 10Awight: Sequence of commands as a command [integration/quibble] - 10https://gerrit.wikimedia.org/r/588083 [12:26:21] (03CR) 10Awight: "PS 8: manual rebase" [integration/quibble] - 10https://gerrit.wikimedia.org/r/588083 (owner: 10Awight) [12:50:07] 10Continuous-Integration-Infrastructure, 10Jenkins, 10Release-Engineering-Team, 10Upstream: Cancelled Jenkins build do not run postbuild script - https://phabricator.wikimedia.org/T299995 (10hashar) [12:55:12] (03PS9) 10Awight: Sequence of commands as a command [integration/quibble] - 10https://gerrit.wikimedia.org/r/588083 [12:55:26] 10Continuous-Integration-Infrastructure, 10Jenkins, 10Release-Engineering-Team, 10Upstream: Cancelled Jenkins build do not run postbuild script - https://phabricator.wikimedia.org/T299995 (10hashar) There are a few other builds such as https://integration.wikimedia.org/ci/job/mwgate-node12-docker/73918/con... [12:57:33] 10Continuous-Integration-Infrastructure, 10Jenkins, 10Release-Engineering-Team, 10Upstream: Cancelled Jenkins build do not run postbuild script - https://phabricator.wikimedia.org/T299995 (10hashar) I give up. I can't really reproduce by manually cancelling a build. It might be an issue related to the Gear... [13:02:27] 10Continuous-Integration-Infrastructure, 10Jenkins, 10Quibble: integration-quibble-funrun job console does not have collapsible sections - https://phabricator.wikimedia.org/T300112 (10awight) I think that `(?:[^:]+)?` can be simplified to `[^:]*`, now? [13:31:56] 10Release-Engineering-Team (Next), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.38.0-wmf.19 deployment blockers - https://phabricator.wikimedia.org/T293960 (10matmarex) [14:06:48] 10Release-Engineering-Team (Next), 10MediaWiki-extensions-General: Fix flapping tests in gated extensions as of 2019-06-06 - https://phabricator.wikimedia.org/T225162 (10WMDE-Fisch) [14:17:27] 10Continuous-Integration-Infrastructure: Create first CI agent with the new disk system - https://phabricator.wikimedia.org/T290783 (10aborrero) [14:23:39] 10GitLab (Infrastructure), 10GitLab-Test, 10Release-Engineering-Team (Radar), 10serviceops-radar, 10Cloud-VPS (Quota-requests): Request increased quota for devtools Cloud VPS project - https://phabricator.wikimedia.org/T299561 (10mdipietro) +1 [15:13:20] 10Beta-Cluster-Infrastructure: To testing - https://phabricator.wikimedia.org/T300151 (10AlPaD) [15:32:20] 10Beta-Cluster-Infrastructure: To testing things - https://phabricator.wikimedia.org/T300151 (10AlPaD) [15:57:50] 10GitLab (Infrastructure), 10GitLab-Test, 10Release-Engineering-Team (Radar), 10serviceops-radar, 10Cloud-VPS (Quota-requests): Request increased quota for devtools Cloud VPS project - https://phabricator.wikimedia.org/T299561 (10aborrero) hey, could you please elaborate a bit more on why you need the ad... [15:57:57] (03CR) 10Ahmon Dancy: [C: 03+2] scap update-wikiversions: Add --no-check flag [tools/scap] - 10https://gerrit.wikimedia.org/r/756035 (owner: 10Ahmon Dancy) [15:58:41] (03Merged) 10jenkins-bot: scap update-wikiversions: Add --no-check flag [tools/scap] - 10https://gerrit.wikimedia.org/r/756035 (owner: 10Ahmon Dancy) [16:00:43] 10Release-Engineering-Team (Next), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.38.0-wmf.19 deployment blockers - https://phabricator.wikimedia.org/T293960 (10brennen) Thanks all for blockers and patches. I'm juggling leading a couple of meetings this morning, but I'll do any pe... [16:26:22] 10GitLab (Infrastructure), 10GitLab-Test, 10Release-Engineering-Team (Radar), 10serviceops-radar, 10Cloud-VPS (Quota-requests): Request increased quota for devtools Cloud VPS project - https://phabricator.wikimedia.org/T299561 (10Jelto) >>! In T299561#7653272, @aborrero wrote: > hey, could you please ela... [16:45:42] !log integration: creating integration-agent-docker-1023 based on buster with new flavor `g3.cores8.ram24.disk20.ephemeral60.4xiops` # T290783 [16:45:43] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:45:44] T290783: Create first CI agent with the new disk system - https://phabricator.wikimedia.org/T290783 [16:46:05] 10Release-Engineering-Team (Next), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.38.0-wmf.19 deployment blockers - https://phabricator.wikimedia.org/T293960 (10Jdlrobson) [16:57:08] 10Continuous-Integration-Infrastructure: Create first CI agent with the new disk system - https://phabricator.wikimedia.org/T290783 (10hashar) Thank you @Krinkle for the work on T277078 and @Jdforrester-WMF for creating the first instance. I have successfully pooled a new Bullseye instance using the ephemeral di... [16:58:05] (03PS1) 10Majavah: Zuul: [mediawiki/extensions/WikimediaIncubator] Drop CentralAuth dependency [integration/config] - 10https://gerrit.wikimedia.org/r/757464 [17:03:21] 10Continuous-Integration-Infrastructure: Create first CI agent with the new disk system - https://phabricator.wikimedia.org/T290783 (10Jdforrester-WMF) >>! In T290783#7653512, @hashar wrote: > I think we can drop [[ https://horizon.wikimedia.org/project/instances/52eedb5b-f450-4b73-9ad6-39e426eab5eb/ | integrati... [17:03:50] 10Release-Engineering-Team: Should UI regressions (e.g. no fatals) with the non-default skins ever block the train? - https://phabricator.wikimedia.org/T300169 (10Jdlrobson) [17:04:42] James_F: thx ;) [17:05:33] one less task [17:05:36] 10Continuous-Integration-Infrastructure: Create first CI agent with the new disk system - https://phabricator.wikimedia.org/T290783 (10hashar) 05Open→03Resolved a:03Jdforrester-WMF It is done, assigning to @Jdforrester-WMF that did all the work, I just got a WMCS flavor with slightly more disk. Thank you! [17:05:38] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Seen): Move all Wikimedia CI (WMCS integration project) instances from stretch to buster/bullseye - https://phabricator.wikimedia.org/T252071 (10hashar) [17:06:53] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Seen): Move all Wikimedia CI (WMCS integration project) instances from stretch to buster/bullseye - https://phabricator.wikimedia.org/T252071 (10hashar) This is ready to be done and there is already an integration-agent-docker-1023. I have u... [17:24:20] hashar: :-) [17:24:38] 10GitLab (Infrastructure), 10GitLab-Test, 10Release-Engineering-Team (Radar), 10serviceops-radar, 10Cloud-VPS (Quota-requests): Request increased quota for devtools Cloud VPS project - https://phabricator.wikimedia.org/T299561 (10Dzahn) gitlab is copying the setup of gerrit and phabricator, which also ha... [17:26:56] 10GitLab (Infrastructure), 10GitLab-Test, 10Release-Engineering-Team (Radar), 10serviceops-radar, 10Cloud-VPS (Quota-requests): Request increased quota for devtools Cloud VPS project - https://phabricator.wikimedia.org/T299561 (10aborrero) Thanks for explaining. [17:27:17] 10GitLab (Infrastructure), 10serviceops: Migrate gitlab-test instance to puppet - https://phabricator.wikimedia.org/T297411 (10aborrero) [17:27:25] 10GitLab (CI & Job Runners), 10serviceops: upgrade gitlab-runners to bullseye - https://phabricator.wikimedia.org/T297659 (10aborrero) [17:27:43] 10GitLab (Infrastructure), 10GitLab-Test, 10Release-Engineering-Team (Radar), 10serviceops-radar, 10Cloud-VPS (Quota-requests): Request increased quota for devtools Cloud VPS project - https://phabricator.wikimedia.org/T299561 (10aborrero) 05Open→03Resolved a:03aborrero [17:29:51] (03CR) 10Ahmon Dancy: [C: 03+2] updateinterwikicache: Do not commit+sync [tools/scap] - 10https://gerrit.wikimedia.org/r/757062 (https://phabricator.wikimedia.org/T247107) (owner: 10Majavah) [17:30:33] (03Merged) 10jenkins-bot: updateinterwikicache: Do not commit+sync [tools/scap] - 10https://gerrit.wikimedia.org/r/757062 (https://phabricator.wikimedia.org/T247107) (owner: 10Majavah) [17:42:59] 10Release-Engineering-Team (Onboarding 🚀): Delete the wmf/1.37.0-wmf.* branches once we've fully deployed to 1.38.0-wmf.1 and finished with them - https://phabricator.wikimedia.org/T291501 (10thcipriani) [17:44:03] 10Continuous-Integration-Config, 10Release-Engineering-Team (Onboarding 🚀): Manage CI Jenkins views with JJB - https://phabricator.wikimedia.org/T256163 (10thcipriani) [17:44:21] 10Release-Engineering-Team (Onboarding 🚀): Automatically tag the release task in the commit message of the weekly branch cut - https://phabricator.wikimedia.org/T278994 (10thcipriani) [17:44:36] 10Release-Engineering-Team (Onboarding 🚀), 10MediaWiki-Release-Tools: Change deploy-promote to check previous groups have been promoted beforehand - https://phabricator.wikimedia.org/T269242 (10thcipriani) [17:46:56] 10Release-Engineering-Team (Onboarding 🚀), 10Scap, 10observability: Update Scap to emit ECS-compatible structured logs - https://phabricator.wikimedia.org/T279086 (10thcipriani) [17:47:20] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Onboarding 🚀), 10Release Pipeline: Have PipelineBot link directly to build console output on test - https://phabricator.wikimedia.org/T271107 (10thcipriani) [17:48:28] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Onboarding 🚀), 10Zuul, 10Release Pipeline: Separate zuul queue for pipelinelib publish jobs - https://phabricator.wikimedia.org/T292130 (10thcipriani) [17:51:26] 10Release-Engineering-Team (Onboarding 🚀), 10MediaWiki-Release-Tools, 10User-brennen: Clear out the old branches on mediawiki/core.git that aren't active any more - https://phabricator.wikimedia.org/T281939 (10thcipriani) [17:54:14] 10Release-Engineering-Team (Onboarding 🚀), 10Scap, 10SRE, 10Sustainability (Incident Followup), 10User-brennen: scap's logstash_checker.py is blissfully unaware of any logstash indexing latency - https://phabricator.wikimedia.org/T255197 (10thcipriani) [17:56:00] 10Release-Engineering-Team (Onboarding 🚀), 10Scap: Minor improvements to scap help output - https://phabricator.wikimedia.org/T243659 (10thcipriani) [17:59:03] 10Release-Engineering-Team (Onboarding 🚀), 10Scap, 10SRE: Remove trusty-specific hacks from logstash_checker.py - https://phabricator.wikimedia.org/T216380 (10thcipriani) [18:00:36] 10Release-Engineering-Team (Onboarding 🚀), 10Scap: Canary deployment continuation prompt should emit a timestamp to console - https://phabricator.wikimedia.org/T201635 (10thcipriani) [18:04:05] Hey, I've started hitting some issues with scap when using deploy-local - seems like overrides on the command-line are broken? https://phabricator.wikimedia.org/P19367 [18:04:43] This is how all puppet-defined scap services are installed so it might be a lot of ongoing breakage in future (just hit it while reimaging a server) [18:04:59] I can file a bug if it's a real issue and not a pebkac [18:05:35] hnowlan: I have to head to a meeting now but I will check it out in about an hour. Is that ok? If not, you can request a rollback on T300058 in case that is the cause. [18:05:36] T300058: Deploy Scap version 4.2.0 - https://phabricator.wikimedia.org/T300058 [18:07:27] dancy: no worries, removing the -D works around it for now [18:07:45] Please do file a bug in the meantime and I'll investigate [18:07:52] You can assign it to me. [18:09:17] ack, will do [18:12:09] 10Scap: scap overrides appear to be broken - https://phabricator.wikimedia.org/T300177 (10hnowlan) [18:13:01] 10Release-Engineering-Team (Next), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.38.0-wmf.19 deployment blockers - https://phabricator.wikimedia.org/T293960 (10Ammarpad) [18:14:27] 10Scap: scap overrides for deploy-local using -D parameter fail - https://phabricator.wikimedia.org/T300177 (10hnowlan) [18:17:01] !log integration: pooled in Jenkins a few Bullseye docker agent for T252071 [18:17:03] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:17:03] T252071: Move all Wikimedia CI (WMCS integration project) instances from stretch to buster/bullseye - https://phabricator.wikimedia.org/T252071 [18:46:33] 10GitLab (Infrastructure), 10serviceops: Migrate gitlab-test instance to puppet - https://phabricator.wikimedia.org/T297411 (10Dzahn) Thanks to @aborrero our quota increase request is resolved. We should now be able to have 2 additional temp. VMs at a time. So 2 of us could do a temp test at a time without to... [18:48:20] 10GitLab (Infrastructure), 10serviceops: Migrate gitlab-test instance to puppet - https://phabricator.wikimedia.org/T297411 (10Dzahn) I clicked "allocate IP to project" and added description "gitlab-ssh" and we got assigned 185.15.56.79 [18:56:00] !log integration: pooled in Jenkins a few more Bullseye docker agents for T252071 [18:56:03] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:56:03] T252071: Move all Wikimedia CI (WMCS integration project) instances from stretch to buster/bullseye - https://phabricator.wikimedia.org/T252071 [19:15:24] 10Beta-Cluster-Infrastructure: To testing things - https://phabricator.wikimedia.org/T300151 (10Zabe) Could you please elaborate what you want to test? [19:23:47] 10Beta-Cluster-Infrastructure: To testing things - https://phabricator.wikimedia.org/T300151 (10AlPaD) Yes I will try extensions like globallock, globalblock, globalrights and then undo them. I will be very careful when I do this and it won't be for regular users but tests accounts. [19:23:51] 10Scap: scap overrides for deploy-local using -D parameter fail - https://phabricator.wikimedia.org/T300177 (10dancy) [19:23:53] 10Release-Engineering-Team, 10Scap, 10serviceops: Deploy Scap version 4.2.0 - https://phabricator.wikimedia.org/T300058 (10dancy) [19:24:58] 10Release-Engineering-Team, 10Scap, 10serviceops: Deploy Scap version 4.2.0 - https://phabricator.wikimedia.org/T300058 (10dancy) @JMeybohm A bug has been detected in 4.2.0. Please roll back to 4.1.1 while I work on a fix. [19:30:06] (03PS1) 10Ahmon Dancy: deploy: Don't call dict() on self.arguments.defines [tools/scap] - 10https://gerrit.wikimedia.org/r/757510 (https://phabricator.wikimedia.org/T300177) [19:31:50] (03CR) 10Ahmon Dancy: [C: 03+2] deploy: Don't call dict() on self.arguments.defines [tools/scap] - 10https://gerrit.wikimedia.org/r/757510 (https://phabricator.wikimedia.org/T300177) (owner: 10Ahmon Dancy) [19:32:32] (03Merged) 10jenkins-bot: deploy: Don't call dict() on self.arguments.defines [tools/scap] - 10https://gerrit.wikimedia.org/r/757510 (https://phabricator.wikimedia.org/T300177) (owner: 10Ahmon Dancy) [19:38:58] (03PS1) 10Ahmon Dancy: Release 4.2.1-1 [tools/scap] - 10https://gerrit.wikimedia.org/r/757511 [19:39:15] (03CR) 10Ahmon Dancy: [C: 03+2] Release 4.2.1-1 [tools/scap] - 10https://gerrit.wikimedia.org/r/757511 (owner: 10Ahmon Dancy) [19:41:54] (03Merged) 10jenkins-bot: Release 4.2.1-1 [tools/scap] - 10https://gerrit.wikimedia.org/r/757511 (owner: 10Ahmon Dancy) [19:42:18] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Yak Shaving 🐃🪒), 10Security Team AppSec, 10Security-Team, and 2 others: Confirm with releng which docker images make the most sense to use - https://phabricator.wikimedia.org/T296805 (10sbassett) >>! In T296805#7650792, @thcipriani wrote: > Hopefully... [19:42:45] 10GitLab (CI & Job Runners), 10Security Team AppSec, 10Security-Team, 10Security: Create initial proof of concept application security pipeline repository - https://phabricator.wikimedia.org/T289293 (10sbassett) [19:43:16] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Yak Shaving 🐃🪒), 10Security Team AppSec, 10Security-Team, and 2 others: Confirm with releng which docker images make the most sense to use - https://phabricator.wikimedia.org/T296805 (10sbassett) 05Open→03Resolved p:05Triage→03Low [19:44:12] 10Release-Engineering-Team, 10Scap, 10serviceops: Deploy Scap version 4.2.1 - https://phabricator.wikimedia.org/T300058 (10dancy) [19:44:56] 10Release-Engineering-Team, 10Scap, 10serviceops: Deploy Scap version 4.2.1 - https://phabricator.wikimedia.org/T300058 (10dancy) >>! In T300058#7654121, @dancy wrote: > @JMeybohm A bug has been detected in 4.2.0. Please roll back to 4.1.1 while I work on a fix. Actually, please restart the process using t... [19:45:50] 10Release-Engineering-Team, 10Scap, 10serviceops: Deploy Scap version 4.2.1 - https://phabricator.wikimedia.org/T300058 (10dancy) [19:45:53] 10Release-Engineering-Team, 10Scap, 10Patch-For-Review: scap overrides for deploy-local using -D parameter fail - https://phabricator.wikimedia.org/T300177 (10dancy) 05Open→03In progress p:05Triage→03High [19:52:23] the oddity of the CI infrastructure [19:52:48] integration-agent-docker got repurposed to be dedicated to SonarQube run and only has the label "codehealth" [19:52:56] it is doing nothing https://integration.wikimedia.org/ci/computer/integration%2Dagent%2Ddocker%2D1014/load-statistics?type=hour [19:53:53] cause in the CI config we had a job tied to it from Sep 30 2019 to Oct 2nd 2019 [19:54:04] so essentially the instance has been sitting around for 2+ years :\ [19:55:11] !log deleting integration-agent-docker-1014 which only has the `codehealth` label. A short live experiment no more used since October 2nd 2019 - https://gerrit.wikimedia.org/r/c/integration/config/+/540362 - T234259 [19:55:13] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:55:13] T234259: Non concurrent mwcore-codehealth-master-non-voting cause Gearman executors to be locked - https://phabricator.wikimedia.org/T234259 [19:55:32] 10Release-Engineering-Team (Next), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.38.0-wmf.19 deployment blockers - https://phabricator.wikimedia.org/T293960 (10brennen) [20:15:22] Project mediawiki-core-doxygen-docker build #31410: 04FAILURE in 11 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen-docker/31410/ [20:27:24] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Seen): Move all Wikimedia CI (WMCS integration project) instances from stretch to buster/bullseye - https://phabricator.wikimedia.org/T252071 (10hashar) [20:27:44] 10Release-Engineering-Team (Next), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.38.0-wmf.19 deployment blockers - https://phabricator.wikimedia.org/T293960 (10jeena) [20:29:13] !log Completed migration of integration-agent-docker-XXXX instances from Stretch to Bullseye - T252071 [20:29:14] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:29:15] T252071: Move all Wikimedia CI (WMCS integration project) instances from stretch to buster/bullseye - https://phabricator.wikimedia.org/T252071 [20:41:47] and we won 2 more instances in the process after cleaning up obsolete stuff [20:42:46] hashar: Awesome! [20:42:56] * James_F hashar: Now to move it to sid! ;-) [20:50:33] James_F: thank you for all the preliminary work on that front :-] [20:50:53] I might have got to use Cinder to attach volumes but I felt like it was a lot of complication given the large fleet of vm [20:50:56] at lesat [20:51:06] I amanaged to rebuild all of them in my evening (over 3 or 4 hours) [20:51:09] which is not too bad [20:51:11] an dno errors! [20:55:34] Indeed. [20:59:32] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Seen): Move all Wikimedia CI (WMCS integration project) instances from stretch to buster/bullseye - https://phabricator.wikimedia.org/T252071 (10hashar) [20:59:36] 10Continuous-Integration-Infrastructure, 10Performance-Team: Provide one or more Qemu agents in CI that use a newer version than 2.x - https://phabricator.wikimedia.org/T284774 (10hashar) [21:15:44] Project mediawiki-core-doxygen-docker build #31411: 04STILL FAILING in 11 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen-docker/31411/ [21:32:38] (03PS4) 10Kosta Harlan: [WIP] Run post-dependency install, pre-test steps in parallel [integration/quibble] - 10https://gerrit.wikimedia.org/r/757411 [21:36:52] (03CR) 10jerkins-bot: [V: 04-1] [WIP] Run post-dependency install, pre-test steps in parallel [integration/quibble] - 10https://gerrit.wikimedia.org/r/757411 (owner: 10Kosta Harlan) [21:54:45] (03PS5) 10Kosta Harlan: [WIP] Run post-dependency install, pre-test steps in parallel [integration/quibble] - 10https://gerrit.wikimedia.org/r/757411 [21:57:36] (03PS6) 10Kosta Harlan: [WIP] Run post-dependency install, pre-test steps in parallel [integration/quibble] - 10https://gerrit.wikimedia.org/r/757411 [22:00:57] (03CR) 10jerkins-bot: [V: 04-1] [WIP] Run post-dependency install, pre-test steps in parallel [integration/quibble] - 10https://gerrit.wikimedia.org/r/757411 (owner: 10Kosta Harlan) [22:05:22] kostajh: what does parallel do by default with output, does it buffer them and flush in order as a whole? or interweave? (the former would be nice) [22:06:29] (03PS7) 10Kosta Harlan: [WIP] Run post-dependency install, pre-test steps in parallel [integration/quibble] - 10https://gerrit.wikimedia.org/r/757411 [22:09:15] (03CR) 10jerkins-bot: [V: 04-1] [WIP] Run post-dependency install, pre-test steps in parallel [integration/quibble] - 10https://gerrit.wikimedia.org/r/757411 (owner: 10Kosta Harlan) [22:16:34] Project mediawiki-core-doxygen-docker build #31412: 04STILL FAILING in 12 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen-docker/31412/ [22:58:54] (03CR) 10BryanDavis: "Recheck" [blubber] - 10https://gerrit.wikimedia.org/r/749569 (https://phabricator.wikimedia.org/T296046) (owner: 10BryanDavis) [23:17:03] Project mediawiki-core-doxygen-docker build #31413: 04STILL FAILING in 13 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen-docker/31413/ [23:22:52] I might be wrong but CI looks very unhappy [23:29:02] yeah, e.g. this semes unusually badly broken: https://integration.wikimedia.org/ci/job/mediawiki-core-php72-phan-docker/61638/console [23:34:41] 10Continuous-Integration-Infrastructure: 'No such file or directory' CI failures in multplie repos - https://phabricator.wikimedia.org/T300214 (10Zabe) [23:35:06] 10Continuous-Integration-Infrastructure: 'No such file or directory' CI failures in multiple repos - https://phabricator.wikimedia.org/T300214 (10Zabe) [23:35:11] https://sal.toolforge.org/log/M6UTmH4B8Fs0LHO5-eGs seems potentially suspicious [23:35:47] sure do [23:36:24] 10Continuous-Integration-Infrastructure: 'No such file or directory' CI failures in multiple repos - https://phabricator.wikimedia.org/T300214 (10bd808) Maybe related to {T252071}? >>! In T252071#7654453, @Stashbot wrote: > {nav icon=file, name=Mentioned in SAL (#wikimedia-releng), href=https://sal.toolforge.or... [23:37:35] 10Continuous-Integration-Infrastructure: 'No such file or directory' CI failures in multiple repos - https://phabricator.wikimedia.org/T300214 (10brennen) @hashar potentially fallout from T252071? [23:38:23] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10User-brennen: 'No such file or directory' CI failures in multiple repos - https://phabricator.wikimedia.org/T300214 (10brennen) [23:43:32] brennen: different error, but likely same cause is https://integration.wikimedia.org/ci/job/publish-to-doc/6697/console where the new https://openstack-browser.toolforge.org/server/integration-agent-docker-1028.integration.eqiad1.wikimedia.cloud instance has no rsync binary in the search path [23:46:29] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10User-brennen: 'No such file or directory' CI failures in multiple repos - https://phabricator.wikimedia.org/T300214 (10Zabe) p:05Triage→03High [23:54:26] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10User-brennen: 'No such file or directory' CI failures in multiple repos - https://phabricator.wikimedia.org/T300214 (10hashar) Looking at https://integration.wikimedia.org/ci/job/wmf-quibble-core-vendor-mysql-php72-docker/71027/consoleFul... [23:54:35] brennen: no idea what it could be :-\ [23:54:58] it looks like files are simply vanishing randomly