[00:06:43] (03PS1) 10Krinkle: make-deploy-notes: Fix internal confusion between VisualEditor ext and lib [tools/release] - 10https://gerrit.wikimedia.org/r/719387 (https://phabricator.wikimedia.org/T249553) [00:06:45] (03PS1) 10Krinkle: make-deploy-notes: Sort components in the "No changes" section [tools/release] - 10https://gerrit.wikimedia.org/r/719388 (https://phabricator.wikimedia.org/T249553) [00:07:31] (03CR) 10jerkins-bot: [V: 04-1] make-deploy-notes: Sort components in the "No changes" section [tools/release] - 10https://gerrit.wikimedia.org/r/719388 (https://phabricator.wikimedia.org/T249553) (owner: 10Krinkle) [00:07:34] (03CR) 10jerkins-bot: [V: 04-1] make-deploy-notes: Fix internal confusion between VisualEditor ext and lib [tools/release] - 10https://gerrit.wikimedia.org/r/719387 (https://phabricator.wikimedia.org/T249553) (owner: 10Krinkle) [00:10:04] (03PS2) 10Krinkle: make-deploy-notes: Fix internal confusion between VisualEditor ext and lib [tools/release] - 10https://gerrit.wikimedia.org/r/719387 (https://phabricator.wikimedia.org/T249553) [00:10:06] (03PS2) 10Krinkle: make-deploy-notes: Sort components in the "No changes" section [tools/release] - 10https://gerrit.wikimedia.org/r/719388 (https://phabricator.wikimedia.org/T249553) [00:21:27] (03PS3) 10Krinkle: make-deploy-notes: Fix internal confusion between VisualEditor ext and lib [tools/release] - 10https://gerrit.wikimedia.org/r/719387 (https://phabricator.wikimedia.org/T249553) [00:21:29] (03PS3) 10Krinkle: make-deploy-notes: Sort components in the "No changes" section [tools/release] - 10https://gerrit.wikimedia.org/r/719388 (https://phabricator.wikimedia.org/T249553) [02:34:40] (03CR) 10Jforrester: [C: 03+2] make-deploy-notes: Fix internal confusion between VisualEditor ext and lib [tools/release] - 10https://gerrit.wikimedia.org/r/719387 (https://phabricator.wikimedia.org/T249553) (owner: 10Krinkle) [02:34:42] (03CR) 10Jforrester: [C: 03+2] make-deploy-notes: Sort components in the "No changes" section [tools/release] - 10https://gerrit.wikimedia.org/r/719388 (https://phabricator.wikimedia.org/T249553) (owner: 10Krinkle) [02:39:48] (03Merged) 10jenkins-bot: make-deploy-notes: Fix internal confusion between VisualEditor ext and lib [tools/release] - 10https://gerrit.wikimedia.org/r/719387 (https://phabricator.wikimedia.org/T249553) (owner: 10Krinkle) [02:39:50] (03Merged) 10jenkins-bot: make-deploy-notes: Sort components in the "No changes" section [tools/release] - 10https://gerrit.wikimedia.org/r/719388 (https://phabricator.wikimedia.org/T249553) (owner: 10Krinkle) [03:06:24] 10Deployments, 10Release-Engineering-Team, 10Developer Productivity: Train changelog no longer excludes libraryupgrader - https://phabricator.wikimedia.org/T290530 (10Krinkle) 05Open→03Resolved [03:06:35] 10Deployments, 10Performance-Team, 10Developer Productivity: Train changelog no longer excludes libraryupgrader - https://phabricator.wikimedia.org/T290530 (10Krinkle) [05:08:31] 10Continuous-Integration-Infrastructure, 10Infrastructure-Foundations, 10Puppet: Puppet failure on integration-puppetmaster-02.integration.eqiad1.wikimedia.cloud - https://phabricator.wikimedia.org/T290422 (10hashar) Who knows? :] Thank you for the certificates regeneration! [07:14:03] (03PS1) 10JMeybohm: helm-linter: Add istioctl [integration/config] - 10https://gerrit.wikimedia.org/r/719454 [07:14:05] (03PS1) 10JMeybohm: jjb: update helm-linter job to releng/helm-linter:0.2.16 [integration/config] - 10https://gerrit.wikimedia.org/r/719455 [07:21:11] (03PS2) 10Jeena Huneidi: WIP add patch author as reviewer to promote patch [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/719319 (https://phabricator.wikimedia.org/T281392) [07:22:03] (03CR) 10jerkins-bot: [V: 04-1] WIP add patch author as reviewer to promote patch [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/719319 (https://phabricator.wikimedia.org/T281392) (owner: 10Jeena Huneidi) [07:22:05] (03CR) 10Jeena Huneidi: "still need to fix the unit tests" [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/719319 (https://phabricator.wikimedia.org/T281392) (owner: 10Jeena Huneidi) [07:59:06] 10Release-Engineering-Team (Radar), 10SRE, 10SRE-Access-Requests: Requesting exec access to pods in 'ci' namespace staging kubernetes - https://phabricator.wikimedia.org/T290360 (10hashar) [08:13:01] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Release Pipeline, 10bacula: CI backups on contint1001 generating 6GB of file metadata- not happening before- potentially slowing down or making impossible a recovery - https://phabricator.wikimedia.org/T290437 (10hashar) I thought it co... [08:17:21] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Release Pipeline, 10bacula, 10Patch-For-Review: CI backups on contint1001 generating 6GB of file metadata- not happening before- potentially slowing down or making impossible a recovery - https://phabricator.wikimedia.org/T290437 (10ha... [08:22:56] hashar: you have a minute for the usual "build helm-linter" request of mine? ;-) [09:31:50] (03CR) 10Hashar: [C: 03+2] helm-linter: Add istioctl [integration/config] - 10https://gerrit.wikimedia.org/r/719454 (owner: 10JMeybohm) [09:31:56] jayme: yup doing it now ;) [09:32:08] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Release Pipeline, 10bacula, 10Patch-For-Review: CI backups on contint1001 generating 6GB of file metadata- not happening before- potentially slowing down or making impossible a recovery - https://phabricator.wikimedia.org/T290437 (10jc... [09:32:53] (03Merged) 10jenkins-bot: helm-linter: Add istioctl [integration/config] - 10https://gerrit.wikimedia.org/r/719454 (owner: 10JMeybohm) [09:38:55] (03CR) 10Hashar: [C: 03+2] "job updated!" [integration/config] - 10https://gerrit.wikimedia.org/r/719455 (owner: 10JMeybohm) [09:40:02] (03Merged) 10jenkins-bot: jjb: update helm-linter job to releng/helm-linter:0.2.16 [integration/config] - 10https://gerrit.wikimedia.org/r/719455 (owner: 10JMeybohm) [09:46:45] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Doing), 10Release Pipeline, 10bacula, 10Patch-For-Review: CI backups on contint1001 generating 6GB of file metadata- not happening before- potentially slowing down or making impossible a recover... - https://phabricator.wikimedia.org/T290437 [09:47:54] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Seen), 10Cloud-VPS, 10Wikidata, and 3 others: Wikibase selenium tests timeout, seemingly due to "memory compaction" events on CI VMs - https://phabricator.wikimedia.org/T281122 (10hashar) [09:50:27] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Seen): Set up a Prometheus server under the integration project - https://phabricator.wikimedia.org/T205930 (10hashar) 05Open→03Declined Reflecting the reality, we are just following whatever is the default for WMCS projects. Right now w... [09:50:29] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Seen): Collect and expose Jenkins build metrics for visualization, reporting, and analysis - https://phabricator.wikimedia.org/T205927 (10hashar) [10:25:26] 10Release-Engineering-Team (Next), 10GitLab, 10serviceops, 10User-brennen: GitLab major version upgrade: 14.x - https://phabricator.wikimedia.org/T289802 (10Jelto) > @Jelto I'll coordinate with you tomorrow, but if you want to go ahead with the upgrade on gitlab2001 before I'm online, feel free. I upgrade... [10:42:04] thanks hashar! [13:26:10] gitlab upgrade to 14.x versions is underway; logging in #-operations. [13:34:36] 10Release-Engineering-Team (Next), 10GitLab, 10serviceops, 10Patch-For-Review, 10User-brennen: GitLab major version upgrade: 14.x - https://phabricator.wikimedia.org/T289802 (10Jelto) I updated `gitlab-runner` to `14.2.0` and `gitlab-ce` to `14.1.5-ce.0` on `apt1001`. [14:05:11] !log runner-1002.gitlab-runners: tested upgrade of gitlab-runner to 14.2.0, seemed to go fine, will do remaining runners (T289802) [14:05:14] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:05:14] T289802: GitLab major version upgrade: 14.x - https://phabricator.wikimedia.org/T289802 [14:51:29] (03CR) 10Hashar: [C: 03+2] "Indeed, sounds good to have it in the extension-gate and in turns mean the Standalone group can be used to prevent some specific tests fro" [integration/config] - 10https://gerrit.wikimedia.org/r/714328 (https://phabricator.wikimedia.org/T289465) (owner: 10Inductiveload) [14:52:57] 10Release-Engineering-Team (Next), 10GitLab, 10serviceops, 10User-brennen: GitLab major version upgrade: 14.x - https://phabricator.wikimedia.org/T289802 (10Jelto) I updated `gitlab-ce` to `14.2.3-ce.0` on `apt1001` [14:54:02] (03Merged) 10jenkins-bot: Add extension-gate to ProofreadPage [integration/config] - 10https://gerrit.wikimedia.org/r/714328 (https://phabricator.wikimedia.org/T289465) (owner: 10Inductiveload) [14:55:29] 10Continuous-Integration-Config, 10ProofreadPage, 10Patch-For-Review: Get ProofreadPage Lua tests working in CI - https://phabricator.wikimedia.org/T289465 (10Inductiveload) @hashar Thank you! For https://gerrit.wikimedia.org/r/c/mediawiki/extensions/ProofreadPage/+/699086, then does it need `@Standalone` o... [14:55:56] !log Reloading Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/714328/ [14:55:57] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:56:11] 10Release-Engineering-Team (Radar), 10SRE, 10SRE-Access-Requests: Requesting exec access to pods in 'ci' namespace staging kubernetes - https://phabricator.wikimedia.org/T290360 (10akosiaris) p:05Triage→03Medium Hi @dancy As in personal level access? We don't have user level accounts, so it would be so... [15:01:08] 10Release-Engineering-Team (Radar), 10SRE, 10SRE-Access-Requests: Requesting exec access to pods in 'ci' namespace staging kubernetes - https://phabricator.wikimedia.org/T290360 (10dancy) >>! In T290360#7339450, @akosiaris wrote: > Hi @dancy > > As in personal level access? We don't have user level account... [15:19:52] 10Release-Engineering-Team (Next), 10GitLab, 10serviceops, 10User-brennen: GitLab major version upgrade: 14.x - https://phabricator.wikimedia.org/T289802 (10brennen) 05Open→03Resolved a:03brennen We're at 14.2.3 on both gitlab2001 and gitlab1001. Shared runners are at 14.2.0 and restarted without in... [15:22:13] 10Release-Engineering-Team (Doing), 10GitLab, 10serviceops, 10User-brennen: GitLab major version upgrade: 14.x - https://phabricator.wikimedia.org/T289802 (10brennen) [15:34:16] 10Release-Engineering-Team (Radar), 10MobileFrontend, 10ci-test-error: Javascript test failures on REL1_31 / REL1_32 / REL1_33 - https://phabricator.wikimedia.org/T230454 (10Jdlrobson) Did this get fixed? [15:42:04] 10Release-Engineering-Team (Radar), 10Quality-and-Test-Engineering-Team (QTE), 10serviceops-radar, 10CommRel-Specialists-Support (Jul-Sep-2021), and 3 others: Expand the list of group 1 wikis to contain at least one (preferably 2) smaller "top ten size" wikis - https://phabricator.wikimedia.org/T286664 (10J... [15:43:08] 10Release-Engineering-Team (Radar), 10Quality-and-Test-Engineering-Team (QTE), 10serviceops-radar, 10CommRel-Specialists-Support (Jul-Sep-2021), and 3 others: Expand the list of group 1 wikis to contain at least one (preferably 2) smaller "top ten size" wikis - https://phabricator.wikimedia.org/T286664 (10J... [15:51:22] (03CR) 10Hashar: [C: 04-1] Quarry: remove old tests now that we're using the pipeline (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/719355 (owner: 10Andrew Bogott) [15:58:26] (03CR) 10Hashar: "A few minor things I have noticed :]" [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/719319 (https://phabricator.wikimedia.org/T281392) (owner: 10Jeena Huneidi) [16:02:16] (03CR) 10Ahmon Dancy: WIP add patch author as reviewer to promote patch (032 comments) [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/719319 (https://phabricator.wikimedia.org/T281392) (owner: 10Jeena Huneidi) [16:08:00] 10Release-Engineering-Team (Yak Shaving 🐃🪒), 10MediaWiki-Docker, 10dev-images: Remove unused stretch image definitions from dev-images - https://phabricator.wikimedia.org/T290532 (10thcipriani) random Fibonacci number for story points :) [16:10:01] hashar: fyi T290584 [16:10:01] T290584: CI builds fail with "Module prefix 'pi' is shared between ProofreadPage\Api\ApiQueryProofreadInfo and PageImages\ApiQueryPageImages" - https://phabricator.wikimedia.org/T290584 [16:15:20] 10Phabricator: "RuntimeException: Undefined index" trying to access task 60224 - https://phabricator.wikimedia.org/T290587 (10Aklapper) [16:15:48] (03CR) 10Urbanecm: "this caused https://phabricator.wikimedia.org/T290584. Not sure how to fix -- changing module prefix is going to be a breaking change i gu" [integration/config] - 10https://gerrit.wikimedia.org/r/714328 (https://phabricator.wikimedia.org/T289465) (owner: 10Inductiveload) [16:17:37] (03PS1) 10JMeybohm: helm-linter: Add a helmfile_log_sal stub [integration/config] - 10https://gerrit.wikimedia.org/r/719548 [16:17:39] (03PS1) 10JMeybohm: jjb: update helm-linter job to releng/helm-linter:0.2.17 [integration/config] - 10https://gerrit.wikimedia.org/r/719549 [16:20:41] 10Beta-Cluster-Infrastructure, 10Abstract Wikipedia team, 10function-evaluator, 10function-orchestrator: Work out how we're going to have "production-like" versions of the wikifunctions evaluator and orchestrator services in Beta Cluster - https://phabricator.wikimedia.org/T289315 (10Jdforrester-WMF) p:05... [16:21:17] hashar: sorry, need your support once more :) [16:22:03] (03CR) 10Ahmon Dancy: "I learned a little about JSON Schema today. Thanks. :-)" [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/719332 (https://phabricator.wikimedia.org/T225335) (owner: 10Dduvall) [16:22:36] (03PS2) 10Andrew Bogott: Quarry: remove old tox run now that we're using the pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/719355 [16:24:05] 10Release-Engineering-Team, 10ci-test-error (WMF-deployed Build Failure): CI builds fail with "Module prefix 'pi' is shared between ProofreadPage\Api\ApiQueryProofreadInfo and PageImages\ApiQueryPageImages" - https://phabricator.wikimedia.org/T290584 (10Urbanecm_WMF) Boldly tagging Releng since removing Proofr... [16:25:06] hashar: ^ can we get a revert on that? It's breaking all MediaWiki CI. [16:25:39] dduvall, does https://gerrit.wikimedia.org/r/c/integration/config/+/719355 require special magic to deploy or can I just +2 it? [16:26:27] tgr, hash.ar has extremely good work-hour discipline so you might need to find someone else here :) [16:27:05] andrewbogott: fair! whoever is willing to revert a zuul config patch, then [16:27:13] (03PS1) 10Urbanecm: Revert "Add extension-gate to ProofreadPage" [integration/config] - 10https://gerrit.wikimedia.org/r/719493 (https://phabricator.wikimedia.org/T290584) [16:27:23] revert uploaded, at least [16:27:57] we might be in a bit of a dead zone between euro and CA working hours [16:28:07] :/ [16:28:25] https://gerrit.wikimedia.org/r/admin/groups/af5fd59673a33a78f3fb79577d15c19be89d9e64,members has a list of people with +2 there [16:29:15] wasn't there some special releng event this week? [16:29:22] jeena, thcipriani, twentyafterfour, dancy, brennen anyone here? [16:30:15] If think Zuul needs a reload andrewbogott too [16:30:34] ok, that's what I thought [16:31:01] https://www.mediawiki.org/wiki/Continuous_integration/Zuul#Update_configuration is the special magic [16:31:29] there is a bit of a releng special event [16:31:33] we're doing a deep work week [16:31:33] https://www.mediawiki.org/wiki/Continuous_integration/Zuul#Deploy_configuration [16:31:46] majavah beat me [16:31:47] thcipriani: can you please merge https://gerrit.wikimedia.org/r/c/integration/config/+/719493 and unbreak MW CI for us? :-) [16:32:03] * thcipriani does [16:32:09] thanks! [16:32:35] (03CR) 10Thcipriani: [C: 03+2] Revert "Add extension-gate to ProofreadPage" [integration/config] - 10https://gerrit.wikimedia.org/r/719493 (https://phabricator.wikimedia.org/T290584) (owner: 10Urbanecm) [16:33:03] 10Release-Engineering-Team, 10Patch-For-Review, 10ci-test-error (WMF-deployed Build Failure): CI builds fail with "Module prefix 'pi' is shared between ProofreadPage\Api\ApiQueryProofreadInfo and PageImages\ApiQueryPageImages" - https://phabricator.wikimedia.org/T290584 (10Zabe) There is another proposed pat... [16:34:01] bah, I can't +op in this channel to add our deep work week to the topic [16:34:27] (03Merged) 10jenkins-bot: Revert "Add extension-gate to ProofreadPage" [integration/config] - 10https://gerrit.wikimedia.org/r/719493 (https://phabricator.wikimedia.org/T290584) (owner: 10Urbanecm) [16:34:32] thanks for the quick fix! [16:34:40] let's see if I can still deploy it :) [16:35:21] thcipriani: puppet says you're a member of contint-roots, so you should be :) [16:35:54] authorized, and maybe capable [16:37:16] !log Reloading zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/719493 [16:37:17] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:37:26] thanks thcipriani :) [16:37:32] {{done}} [16:37:37] thank you all for the quick fix and the ping [16:38:00] thcipriani: enjoy rest of your deep work :) [16:38:25] tsk: I'm a manager -- I never do deep work :) [16:38:37] j/k j/k [16:38:48] ty! [16:40:52] this is what we're doing this week (for the curious) https://www.mediawiki.org/wiki/Wikimedia_Release_Engineering_Team/Yak [16:48:33] 10Phabricator: "RuntimeException: Undefined index" trying to access task 60224 - https://phabricator.wikimedia.org/T290587 (10Zabe) Hmm, i can't reproduce this. [16:48:59] 10Phabricator: "RuntimeException: Undefined index" trying to access task 60224 - https://phabricator.wikimedia.org/T290587 (10Urbanecm) Works for me now, but I was able to reproduce few minutes ago. @Aklapper Does it work now? [16:49:54] thcipriani: kunal or Greg have topic perms [16:50:44] hm? [16:51:06] also I +2'd the core change to unbreak the tests [16:51:08] legoktm: could you modify topic here or grant me the magic to do so? [16:51:36] want to note releng is doing a deep work offsite this week: https://www.mediawiki.org/wiki/Wikimedia_Release_Engineering_Team/Yak [16:51:38] legoktm: well thcipriani kindly removed the extension from the gate, so no longer an issue (hopefully) [16:51:55] (and therefore less available than normal) [16:52:11] I don't have the ability to give out rights here (greg-g can), but let me add that to the topic [16:52:59] "Team members (+voiced)" seems a bit inaccurate too [16:53:43] * legoktm points to https://meta.wikimedia.org/wiki/IRC/Bots/ircservserv for easy channel management [16:53:49] <3 [16:54:47] thanks for the topic update [17:03:04] andrewbogott: i can merge/deploy https://gerrit.wikimedia.org/r/c/integration/config/+/719355 if it's ready [17:03:26] It's ready if it looks ready to you! [17:03:32] Sorry to interrupt the yak-shaving though [17:03:54] it's cool [17:04:03] (03CR) 10Dduvall: [C: 03+2] Quarry: remove old tox run now that we're using the pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/719355 (owner: 10Andrew Bogott) [17:05:21] (03Merged) 10jenkins-bot: Quarry: remove old tox run now that we're using the pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/719355 (owner: 10Andrew Bogott) [17:05:56] !log Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/719355 [17:05:58] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:06:09] thanks dduvall ! [17:06:16] np :) [17:06:34] Someday I will want to move the js tests into the pipeline but that can wait until the unlikely future day when I feel like writing js tests [17:06:55] understandable [17:11:31] 10Release-Engineering-Team, 10ci-test-error (WMF-deployed Build Failure): CI builds fail with "Module prefix 'pi' is shared between ProofreadPage\Api\ApiQueryProofreadInfo and PageImages\ApiQueryPageImages" - https://phabricator.wikimedia.org/T290584 (10Urbanecm_WMF) 05Open→03Resolved Jenkins looks to be h... [17:14:53] (03CR) 10Dduvall: Provide JSON schema for use in config validation (032 comments) [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/719332 (https://phabricator.wikimedia.org/T225335) (owner: 10Dduvall) [17:17:10] (03PS2) 10Dduvall: Provide JSON schema for use in config validation [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/719332 (https://phabricator.wikimedia.org/T225335) [17:17:12] (03PS2) 10Dduvall: WIP: Perform validation using JSON schema and ajv-cli [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/719382 (https://phabricator.wikimedia.org/T225335) [17:18:19] (03CR) 10jerkins-bot: [V: 04-1] WIP: Perform validation using JSON schema and ajv-cli [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/719382 (https://phabricator.wikimedia.org/T225335) (owner: 10Dduvall) [17:18:29] (03PS1) 10Tpt: Revert "Revert "Add extension-gate to ProofreadPage"" [integration/config] - 10https://gerrit.wikimedia.org/r/719496 [17:21:55] (03CR) 10jerkins-bot: [V: 04-1] Revert "Revert "Add extension-gate to ProofreadPage"" [integration/config] - 10https://gerrit.wikimedia.org/r/719496 (owner: 10Tpt) [17:24:59] 10Phabricator: Temporary "RuntimeException: Undefined index" trying to access task 60224 - https://phabricator.wikimedia.org/T290587 (10Aklapper) [17:25:25] 10Phabricator: Temporary "RuntimeException: Undefined index" trying to access task 60224 - https://phabricator.wikimedia.org/T290587 (10Aklapper) Huh, thanks. It works now, cannot reproduce either. For the records, `SELECT id FROM maniphest_task WHERE phid = "PHID-TASK-lmu25hphpaxreg3tprcd";` lists `T44473`. L... [17:32:49] (03PS2) 10Tpt: Revert Revert "Add extension-gate to ProofreadPage" [integration/config] - 10https://gerrit.wikimedia.org/r/719496 [17:37:21] (03CR) 10Ahmon Dancy: [C: 03+1] Provide JSON schema for use in config validation [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/719332 (https://phabricator.wikimedia.org/T225335) (owner: 10Dduvall) [17:39:20] 10Release-Engineering-Team (Yak Shaving 🐃🪒), 10MediaWiki-Docker, 10dev-images: Remove unused stretch image definitions from dev-images - https://phabricator.wikimedia.org/T290532 (10brennen) 05Open→03Resolved I see some stuff in [[https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/refs/he... [17:53:14] (03CR) 10Jeena Huneidi: WIP add patch author as reviewer to promote patch (033 comments) [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/719319 (https://phabricator.wikimedia.org/T281392) (owner: 10Jeena Huneidi) [17:56:00] (03CR) 10Ahmon Dancy: WIP add patch author as reviewer to promote patch (031 comment) [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/719319 (https://phabricator.wikimedia.org/T281392) (owner: 10Jeena Huneidi) [18:04:20] (03PS3) 10Dduvall: Provide JSON schema for use in config validation [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/719332 (https://phabricator.wikimedia.org/T225335) [18:05:57] (03CR) 10Dduvall: "Latest PS added some missing fields and is more strict about disallowing additional properties." [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/719332 (https://phabricator.wikimedia.org/T225335) (owner: 10Dduvall) [18:07:33] 10Release-Engineering-Team (Yak Shaving 🐃🪒), 10MediaWiki-Docker, 10dev-images: Remove unused stretch image definitions from dev-images - https://phabricator.wikimedia.org/T290532 (10brennen) [18:09:57] (03CR) 10Ahmon Dancy: [C: 03+1] Provide JSON schema for use in config validation [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/719332 (https://phabricator.wikimedia.org/T225335) (owner: 10Dduvall) [18:13:24] 10Release-Engineering-Team (Yak Shaving 🐃🪒), 10GitLab, 10serviceops, 10User-brennen: GitLab major version upgrade: 14.x - https://phabricator.wikimedia.org/T289802 (10thcipriani) [18:26:31] (03CR) 10Ahmon Dancy: [C: 03+2] dockerfiles: Provide ajv image for JSON schema based validation [integration/config] - 10https://gerrit.wikimedia.org/r/719350 (https://phabricator.wikimedia.org/T225335) (owner: 10Dduvall) [18:27:49] (03Merged) 10jenkins-bot: dockerfiles: Provide ajv image for JSON schema based validation [integration/config] - 10https://gerrit.wikimedia.org/r/719350 (https://phabricator.wikimedia.org/T225335) (owner: 10Dduvall) [18:28:50] !log Running ./fab deploy_docker to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/719350 [18:28:51] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:30:15] 10Release-Engineering-Team (Doing), 10Release, 10Train Deployments: 1.37.0-wmf.23 deployment blockers - https://phabricator.wikimedia.org/T281164 (10Jdlrobson) FYI Italian should be a group 1 wiki as part of this train for the first time (joining Catalan and Hebrew). This could lead to a higher volume of pot... [18:31:19] 10Release-Engineering-Team (Radar), 10Quality-and-Test-Engineering-Team (QTE), 10serviceops-radar, 10CommRel-Specialists-Support (Jul-Sep-2021), and 3 others: Expand the list of group 1 wikis to contain at least one (preferably 2) smaller "top ten size" wikis - https://phabricator.wikimedia.org/T286664 (10J... [18:32:02] 10Release-Engineering-Team (Radar), 10Quality-and-Test-Engineering-Team (QTE), 10serviceops-radar, 10CommRel-Specialists-Support (Jul-Sep-2021), and 3 others: Expand the list of group 1 wikis to contain at least one (preferably 2) smaller "top ten size" wikis - https://phabricator.wikimedia.org/T286664 (10J... [18:33:07] 10Release-Engineering-Team (Radar), 10Quality-and-Test-Engineering-Team (QTE), 10serviceops-radar, 10CommRel-Specialists-Support (Jul-Sep-2021), and 3 others: Expand the list of group 1 wikis to contain at least one (preferably 2) smaller "top ten size" wikis - https://phabricator.wikimedia.org/T286664 (10J... [18:33:09] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Doing), 10Release Pipeline, 10bacula: CI backups on contint1001 generating 6GB of file metadata- not happening before- potentially slowing down or making impossible a recovery - https://phabricator.wikimedia.org/T290437 (10hashar) @Dzahn... [18:33:49] dancy: dduvall: jeena: looks like pipelinelib is again leaking containers on contint1001 / contint2001 slowly filing disks :] [18:34:46] we had a task about it at some point which got fixed (some groovy patch got reverted) [18:34:52] i will file another task [18:39:12] hashar: containers or images? [18:40:45] dduvall: containers ;) [18:41:32] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Release Pipeline: Pipeline lib still leaks containers on contint1001 / contint2001 - https://phabricator.wikimedia.org/T290608 (10hashar) [18:41:41] and I found the task I was looking earlier [18:41:52] we had an issue in pipelinelib which was not reclaiming images https://gerrit.wikimedia.org/r/c/integration/pipelinelib/+/697825 [18:42:20] but this time it is containers that are left behind as if pipeline lib invokes `docker run` without `--rm` [18:42:20] https://gerrit.wikimedia.org/r/c/integration/pipelinelib/+/702778 May be the fix for the current problem. [18:42:25] anyway filed as T290608 ;) [18:42:26] T290608: Pipeline lib still leaks containers on contint1001 / contint2001 - https://phabricator.wikimedia.org/T290608 [18:44:05] dancy: lovely ;) [18:45:33] though I would expect the containers clean up to happen automatically (docker run --rm) or as a publisher step that happens after the build [18:45:59] the old jenkins jobs have a postbuildscript publisher that does the clean up [18:46:33] anyway, it is not urgent, it is just that the disk slowly fills up but it is easy to reclaim space ( docker container prune -f ) [18:47:12] !log contint1001 / contint2001: /srv/dev-images: git remote set-url origin 'https://gitlab.wikimedia.org/releng/dev-images.git' [18:47:14] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:48:31] 10Release-Engineering-Team (Yak Shaving 🐃🪒), 10GitLab, 10dev-images, 10Patch-For-Review, 10User-brennen: Migrate releng/dev-images to GitLab - https://phabricator.wikimedia.org/T290259 (10brennen) [18:53:34] brennen: that should probably be done via puppet as well. `git::clone` might well reset the url on the next puppet run ( modules/profile/manifests/local_dev/docker_publish.pp ) [18:53:39] 10Release-Engineering-Team (Yak Shaving 🐃🪒), 10MediaWiki-Docker, 10dev-images, 10mwcli, 10User-brennen: Create a PHP 8 dev image for use with mediawiki-docker-dev - https://phabricator.wikimedia.org/T283426 (10brennen) [18:53:59] 10Release-Engineering-Team (Yak Shaving 🐃🪒), 10GitLab, 10User-brennen: Migrate mediawiki/tools/release/ to GitLab - https://phabricator.wikimedia.org/T290260 (10brennen) [18:55:10] 10Release-Engineering-Team (Next), 10GitLab, 10User-brennen, 10Voice & Tone: Rename mainline Git branch from "master" to "main" on all WMF-hosted repositories during GitLab migration - https://phabricator.wikimedia.org/T281593 (10brennen) [18:55:21] 10Release-Engineering-Team (Next), 10GitLab, 10User-brennen: Establish a routine GitLab deployment / update window - https://phabricator.wikimedia.org/T287117 (10brennen) [18:55:37] 10Phabricator, 10Release-Engineering-Team (Doing), 10GitLab, 10GitLab-Test, 10User-brennen: Experiment with GitLab-Phabricator integration - https://phabricator.wikimedia.org/T265617 (10brennen) [19:06:52] Daimona: perfect pronunciation for "bonsoir" :] [19:07:50] 10Release-Engineering-Team (Yak Shaving 🐃🪒): Investigate password store options - https://phabricator.wikimedia.org/T290337 (10hashar) Spotted yesterday , I realized we already have a password management system! https://office.wikimedia.org/wiki/Password_Management :] [19:22:23] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Doing), 10Release Pipeline, 10bacula: CI backups on contint1001 generating 6GB of file metadata- not happening before- potentially slowing down or making impossible a recovery - https://phabricator.wikimedia.org/T290437 (10jcrespo) I can... [19:25:21] 10Release-Engineering-Team (Yak Shaving 🐃🪒), 10GitLab, 10User-brennen: GitLab project templates include issues - figure out if these can be customized or removed - https://phabricator.wikimedia.org/T290612 (10brennen) [19:25:29] 10Release-Engineering-Team (Yak Shaving 🐃🪒), 10GitLab, 10User-brennen: GitLab project templates include issues - figure out if these can be customized or removed - https://phabricator.wikimedia.org/T290612 (10brennen) [19:32:28] hashar: That's because I studied French... Even if I never ever speak/read it :D [19:34:30] 10Release-Engineering-Team (Yak Shaving 🐃🪒), 10GitLab, 10dev-images, 10Patch-For-Review, 10User-brennen: Migrate releng/dev-images to GitLab - https://phabricator.wikimedia.org/T290259 (10brennen) [20:00:32] something broke Beta Cluster [20:00:36] eg: https://en.wikipedia.beta.wmflabs.org/wiki/Special:Version [20:00:48] Original exception: [YTkWHSzBLgMIK4mlcvEsjwAAAAQ] /wiki/Special:Version TypeError: Argument 1 passed to ResourceLoaderSkinModule::getRelativeSizedLogo() must be of the type array, boolean given, called in /srv/mediawiki/php-master/includes/resourceloader/ResourceLoaderSkinModule.php on line 600 [20:09:33] https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/core/+/33ac451d3d170cef87e999df2beae95be1f31f5a [20:12:19] * thcipriani reverts [20:16:11] !log self +2 on https://gerrit.wikimedia.org/r/c/mediawiki/core/+/719500 to unbreak beta [20:16:13] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:29:36] 10Continuous-Integration-Infrastructure, 10cloud-services-team (Kanban): Puppet agent failure on integration-agent-qemu-1001 in project integration - https://phabricator.wikimedia.org/T290615 (10Bstorm) [20:33:44] 10Beta-Cluster-Infrastructure: Temporary interface admin right to beta commons for testing UploadWizard CSS gadget - https://phabricator.wikimedia.org/T290124 (10Aklapper) [20:33:51] thcipriani: greg-g just catching up [20:33:59] That sounds like bad configuration which could be far worse.. [20:34:01] these should never be booleans [20:34:30] Jdlrobson: makes sense, I actually went looking for mw-config changes rather than core changes [20:34:34] initially [20:34:42] https://www.irccloud.com/pastebin/bgg0w5YF/that [20:34:44] that's the issue [20:34:55] I assume it wants to unset default [20:35:02] how does beta cluster config unset [20:35:22] I guess you want an empty array now? [20:35:28] thcipriani: reverting that is likely to cause more problems as both Minerva and Vector merged patches that use that [20:35:43] thcipriani: empty array won't cut it.. it needs to be undefined [20:35:48] so isset check fails [20:37:23] sounds complicated, couldn't leave beta broken in the interim, hence the revert. Apologies for lack of context. [20:37:32] thcipriani: we got lucky here though [20:37:37] this should have thrown another exception [20:37:43] (reverting) [20:40:00] 10Release-Engineering-Team (Yak Shaving 🐃🪒), 10GitLab, 10User-brennen: GitLab project templates include issues - figure out if these can be customized or removed - https://phabricator.wikimedia.org/T290612 (10brennen) A couple of upstream docs & issues: - https://docs.gitlab.com/ee/user/admin_area/custom_pr... [20:42:33] Out of interest [20:42:40] "couldn't leave beta broken in the interim" what is the SLA for that? [20:44:10] "a general understanding that people will yell until someone does something" [20:44:22] Just I've seen the beta cluster broken for far longer periods than 40 mins :) and I have a fix up already. [20:44:41] releng is no longer the product owner for beta so I couldn't tell you, but it might have been better to say: I didn't feel like leaving it broken when a revert seemed possible [20:46:10] And what would have happened next if the revert hadn't worked? [20:46:25] file UBNs, or keep reverting further [20:46:28] ^ [20:46:38] probably filed an UBN [20:47:26] Reverting further seems like a dangerous path? [20:47:29] breaking beta is totally fine, because that's partially its purpose, to catch integration issues. but leaving it broken is problematic because it's a shared resource and will end up hiding other issues [20:47:40] Sure. [20:47:55] I feel like you're upset that I reverted you. [20:48:12] I'm not upset, I'm just a bit curious why I wasn't pinged as the patch author. [20:48:27] As I could have short circuited a lot of this [20:48:31] AFAICT, buggy patches were merged without making sure mediawiki-config was properly updated, immediately reverting seems like the correct course of action [20:48:37] would you like to be pinged instead of me reverting next time? [20:49:46] I think in general it's best for the people who introduce errors to deal with them if they're available as they might have useful information but I understand there needs to be a response time for that. [20:50:16] if there were skin patches that depended on the core patch, they should've been marked as Depends-On so it would be clear to anyone who needs to revert would know that it's not straightforward [20:50:44] Here it didn't turn out to be a problem but I was a little worried that skins trying to render missing variables would also throw exceptions (but it turns out https://phabricator.wikimedia.org/T128864 has not been implemented yet) [20:50:59] (which I think was the case, given https://gerrit.wikimedia.org/r/c/mediawiki/skins/MinervaNeue/+/717493) [20:51:47] legoktm: there was, yeh but llooks like it got stuck [20:52:15] and https://gerrit.wikimedia.org/r/c/mediawiki/skins/Vector/+/719599 wasn't tagged because I posted it after the merge [20:54:37] ok, so the revert looks perfectly safe to me, and https://en.wikipedia.beta.wmflabs.org/wiki/Special:Version works again [20:57:38] The issue is now I'm not sure how to this patch through with a preferred fix without causing the issue again or introducing tech debt. [21:00:19] In this case we've got the beta cluster working again, so job done, but I think in general it would be good practice to involve the patch submitter or reviewer in those decisions where possible given the complexity of our ecosystem and that's the only point I really wanted to make here. [21:00:51] I appreciate you listening. I appreciate I am not on the front lines of deploying and that may not be practical. [21:06:00] I think https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/719619 corrects the config [21:06:16] but I'm not sure how to test it [21:23:55] the core change should be compatible with both the current config and new config [21:24:13] and maybe wait a bit (like a day) for the core change to stick before merging the dependent skin changes [21:49:26] Timo just gave me a tip I wasn't aware of that the CI produces the config [21:50:08] that should be enough to be confident about cleaning up the tech debt, but yeh waiting some time is a good idea https://gerrit.wikimedia.org/r/c/mediawiki/core/+/719625 [21:50:15] (I'm not planning to land the config change today) [22:31:56] James_F: any idea what's going on here? https://integration.wikimedia.org/ci/job/mwgate-node12-docker/30130/console [22:32:20] https://www.irccloud.com/pastebin/IIomQdzO/ [22:32:30] (Minerva)