[04:50:36] (03PS2) 10Jforrester: jjb: Provide PHP 7.4 versions of wmf-quibble jobs [integration/config] - 10https://gerrit.wikimedia.org/r/734417 [04:52:58] (03CR) 10Jforrester: [C: 03+2] jjb: Provide PHP 7.4 versions of wmf-quibble jobs [integration/config] - 10https://gerrit.wikimedia.org/r/734417 (owner: 10Jforrester) [04:54:38] (03Merged) 10jenkins-bot: jjb: Provide PHP 7.4 versions of wmf-quibble jobs [integration/config] - 10https://gerrit.wikimedia.org/r/734417 (owner: 10Jforrester) [04:54:51] (03PS3) 10Jforrester: Zuul: Enable PHP74 jobs on gate-and-submit-wmf pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/734416 (https://phabricator.wikimedia.org/T293924) [04:57:04] (03CR) 10Jforrester: [C: 03+2] Zuul: Enable PHP74 jobs on gate-and-submit-wmf pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/734416 (https://phabricator.wikimedia.org/T293924) (owner: 10Jforrester) [04:58:46] (03Merged) 10jenkins-bot: Zuul: Enable PHP74 jobs on gate-and-submit-wmf pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/734416 (https://phabricator.wikimedia.org/T293924) (owner: 10Jforrester) [04:59:09] !log Zuul: Enable PHP74 jobs on gate-and-submit-wmf pipeline T293924 [04:59:12] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [04:59:13] T293924: Also run PHP 7.4 jobs on wmf branch patches - https://phabricator.wikimedia.org/T293924 [05:02:55] 10Release-Engineering-Team (Done by Thu 04 Nov), 10Release, 10Train Deployments: 1.38.0-wmf.6 deployment blockers - https://phabricator.wikimedia.org/T293947 (10Jdforrester-WMF) Heads-up that I've enabled PHP 7.4 voting jobs for merges on production branches (alongside the existing PHP 7.2 jobs), per T293924... [05:03:24] 10Continuous-Integration-Config, 10Patch-For-Review: Also run PHP 7.4 jobs on wmf branch patches - https://phabricator.wikimedia.org/T293924 (10Jdforrester-WMF) 05Open→03Resolved a:03Jdforrester-WMF [07:41:20] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Radar): Upgrade deployment-prep deploy* hosts to Buster - https://phabricator.wikimedia.org/T278689 (10hashar) [07:52:52] 10Release-Engineering-Team (Doing), 10Quibble: Establish communication channel for Quibble development (plot twist: Slack channel) - https://phabricator.wikimedia.org/T286770 (10hashar) I have created the IRC `#wikimedia-quibble` channel following the documentation at https://meta.wikimedia.org/wiki/IRC/Instru... [07:54:05] !log Created #wikimedia-quibble IRC channel for https://doc.wikimedia.org/quibble/ [07:55:18] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [08:21:47] 10Release-Engineering-Team (Done by Thu 04 Nov), 10Release, 10Train Deployments: 1.38.0-wmf.6 deployment blockers - https://phabricator.wikimedia.org/T293947 (10Urbanecm_WMF) >>! In T293947#7457548, @Jdforrester-WMF wrote: > Heads-up that I've enabled PHP 7.4 voting jobs for merges on production branches (al... [08:22:30] (03PS1) 10Urbanecm: Revert "Zuul: Enable PHP74 jobs on gate-and-submit-wmf pipeline" [integration/config] - 10https://gerrit.wikimedia.org/r/734325 (https://phabricator.wikimedia.org/T293924) [08:22:43] (03PS2) 10Urbanecm: Revert "Zuul: Enable PHP74 jobs on gate-and-submit-wmf pipeline" [integration/config] - 10https://gerrit.wikimedia.org/r/734325 (https://phabricator.wikimedia.org/T293924) [08:23:00] may i trouble someone to merge this for me please? maybe hashar? [08:23:17] urbanecm: SURE! [08:23:28] something broken under php 7.4? [08:23:39] (03CR) 10Hashar: [C: 03+2] Revert "Zuul: Enable PHP74 jobs on gate-and-submit-wmf pipeline" [integration/config] - 10https://gerrit.wikimedia.org/r/734325 (https://phabricator.wikimedia.org/T293924) (owner: 10Urbanecm) [08:23:46] yup, a test wants me to apply changes that will break my code for 7.2 (ie. WMF prod) [08:23:52] see https://phabricator.wikimedia.org/T293947#7457724 [08:24:06] thank you for spotting, reporting and reverting ! [08:24:11] any time :) [08:25:25] 10Continuous-Integration-Config, 10Patch-For-Review: Also run PHP 7.4 jobs on wmf branch patches - https://phabricator.wikimedia.org/T293924 (10hashar) 05Resolved→03Open Reverted due to some issue with GrowthExperiments at least T293947#7457724 [08:25:59] (03Merged) 10jenkins-bot: Revert "Zuul: Enable PHP74 jobs on gate-and-submit-wmf pipeline" [integration/config] - 10https://gerrit.wikimedia.org/r/734325 (https://phabricator.wikimedia.org/T293924) (owner: 10Urbanecm) [08:28:43] urbanecm: I have deployed the php7.4 removal [08:28:52] appreciated hashar [08:29:04] hashar: !log? [08:29:07] urbanecm: I am surprised the job breaks though. I thought we already run php7.4 job on CR+2 [08:29:25] !log zuul: reverted Enable PHP74 jobs on gate-and-submit-wmf pipeline" https://gerrit.wikimedia.org/r/734325 [08:29:26] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [08:30:24] we do run php 7.4 tests, but apparently not 7.4 phan [08:35:05] 10Release-Engineering-Team (Done by Thu 04 Nov), 10Release, 10Train Deployments: 1.38.0-wmf.6 deployment blockers - https://phabricator.wikimedia.org/T293947 (10Urbanecm_WMF) >>! In T293947#7457724, @Urbanecm_WMF wrote: >>>! In T293947#7457548, @Jdforrester-WMF wrote: >> Heads-up that I've enabled PHP 7.4 vo... [08:44:59] 10Release-Engineering-Team (Doing), 10Quibble, 10Patch-For-Review: Establish communication channel for Quibble development (plot twist: Slack channel) - https://phabricator.wikimedia.org/T286770 (10hashar) Got assistance from `#wikimedia-ops` to add a few helpful bots such as `ircserserv-wm` to manage channe... [09:03:43] 10Release-Engineering-Team (Doing), 10Quibble, 10Patch-For-Review: Establish communication channel for Quibble development (plot twist: Slack channel) - https://phabricator.wikimedia.org/T286770 (10hashar) 05Open→03In progress a:03hashar [09:25:10] 10Continuous-Integration-Config: Also run PHP 7.4 jobs on wmf branch patches - https://phabricator.wikimedia.org/T293924 (10Daimona) >>! In T293924#7457729, @hashar wrote: > Reverted due to some issue with GrowthExperiments at least T293947#7457724 We cannot really switch phan job to PHP 7.4: phan uses the PHP... [09:27:09] 10Release-Engineering-Team, 10Code-Stewardship-Reviews, 10Documentation: Document checklist steps to undeploy / sunset a codebase on WMF servers (not: archiving) - https://phabricator.wikimedia.org/T294329 (10Aklapper) [10:13:25] (03CR) 10Hashar: "It is due to T188375 , it took a while to restore the cache which is both bound to:" [integration/config] - 10https://gerrit.wikimedia.org/r/734412 (owner: 10Jforrester) [10:16:24] (03PS2) 10Hashar: jjb: Update caster users from 0.2.1 (2018) to 0.2.4 (2019) [integration/config] - 10https://gerrit.wikimedia.org/r/734412 (https://phabricator.wikimedia.org/T188375) (owner: 10Jforrester) [10:17:03] !log Updating all Jenkins job to remove `--compress` option from rsync when saving cache | https://gerrit.wikimedia.org/r/c/integration/config/+/734412 | T188375 [10:17:06] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [10:17:06] T188375: castor rsync's taking 3-5 minutes for mwgate-npm jobs - https://phabricator.wikimedia.org/T188375 [10:17:29] (03PS3) 10Hashar: jjb: Update castor users from 0.2.1 (2018) to 0.2.4 (2019) [integration/config] - 10https://gerrit.wikimedia.org/r/734412 (https://phabricator.wikimedia.org/T188375) (owner: 10Jforrester) [10:17:44] (03CR) 10Hashar: [C: 03+2] "Fixed typo in commit message and refreshed all jobs." [integration/config] - 10https://gerrit.wikimedia.org/r/734412 (https://phabricator.wikimedia.org/T188375) (owner: 10Jforrester) [10:19:57] (03Merged) 10jenkins-bot: jjb: Update castor users from 0.2.1 (2018) to 0.2.4 (2019) [integration/config] - 10https://gerrit.wikimedia.org/r/734412 (https://phabricator.wikimedia.org/T188375) (owner: 10Jforrester) [12:03:33] 10Continuous-Integration-Infrastructure, 10Jenkins, 10MediaWiki-extensions-WikibaseClient, 10Wikidata, and 5 others: wmf-quibble-selenium-php72-docker jobs failing repeatedly: Undefined index: rc_logid - https://phabricator.wikimedia.org/T293885 (10Urbanecm) Backport to wmf.5 is in progress, in case someon... [13:06:52] 10Release-Engineering-Team (Done by Thu 04 Nov), 10Release, 10Train Deployments: 1.38.0-wmf.6 deployment blockers - https://phabricator.wikimedia.org/T293947 (10hashar) I have synced 1.38.0-wmf.6 code to the MediaWiki servers. Next step will be to promote testwikis / rebuild l10n/cdb cache which should be fa... [13:32:32] 10Continuous-Integration-Infrastructure, 10Jenkins, 10MediaWiki-extensions-WikibaseClient, 10Wikidata, and 4 others: wmf-quibble-selenium-php72-docker jobs failing repeatedly: Undefined index: rc_logid - https://phabricator.wikimedia.org/T293885 (10hashar) Thank you @Urbanecm for fixing up the CI jobs! Th... [13:35:48] (03PS1) 10Zfilipin: selenium: Remove wdio-video-reporter npm package [integration/config] - 10https://gerrit.wikimedia.org/r/734639 (https://phabricator.wikimedia.org/T294341) [13:50:11] 10Continuous-Integration-Infrastructure, 10Jenkins, 10MediaWiki-extensions-WikibaseClient, 10Wikidata, and 4 others: wmf-quibble-selenium-php72-docker jobs failing repeatedly: Undefined index: rc_logid - https://phabricator.wikimedia.org/T293885 (10Urbanecm) No problem @hashar. Also deployed it, as I forgo... [13:51:47] 10Release-Engineering-Team (Done by Thu 04 Nov), 10Release, 10Train Deployments: 1.38.0-wmf.6 deployment blockers - https://phabricator.wikimedia.org/T293947 (10Urbanecm) >>! In T293947#7458294, @hashar wrote: > I have synced 1.38.0-wmf.6 code to the MediaWiki servers. Next step will be to promote testwikis... [14:20:46] (03CR) 10Hashar: "I got those log/rawSeleniumVideoGrabs removed since they filed the Jenkins CI server (was T249268)." [integration/config] - 10https://gerrit.wikimedia.org/r/734639 (https://phabricator.wikimedia.org/T294341) (owner: 10Zfilipin) [14:34:03] (03PS1) 10Lucas Werkmeister (WMDE): [Zuul]: [mediawiki/extensions/WikibaseLexeme] add mwext-doxygen-publish [integration/config] - 10https://gerrit.wikimedia.org/r/734654 [15:14:06] 10Project-Admins: Create project tag for wmfdb - https://phabricator.wikimedia.org/T294357 (10Kormat) [15:25:59] (03CR) 10Jforrester: "Yeah, let's deploy this patch in a year or so?" [integration/config] - 10https://gerrit.wikimedia.org/r/734639 (https://phabricator.wikimedia.org/T294341) (owner: 10Zfilipin) [15:40:57] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Doing): Replace deployment-imagescaler03 (stretch) with deployment-imagescaler04 (buster) - https://phabricator.wikimedia.org/T294148 (10dancy) Outstanding issues when running `puppet agent -t`: #1 ` Error: /Stage[main]/Thumbor/Package[python-logstash... [16:03:39] 10Release-Engineering-Team (Priority Backlog 🔥), 10Release Pipeline: "promote" step failure with multiple users per email - https://phabricator.wikimedia.org/T294300 (10dduvall) [16:15:18] 10Release-Engineering-Team (Priority Backlog 🔥), 10Release Pipeline: "promote" step failure with multiple users per email - https://phabricator.wikimedia.org/T294300 (10dduvall) A quick test in train-dev shows that reviewers labels can include user IDs instead of email addresses. This should be a straightforwa... [16:16:28] 10Release-Engineering-Team (Priority Backlog 🔥), 10Release Pipeline: "promote" step failure with multiple users per email - https://phabricator.wikimedia.org/T294300 (10dduvall) 05Open→03In progress p:05Triage→03High a:03dduvall [16:31:13] (03CR) 1020after4: deploy-promote: Only pass --proxy to curl when running on wmnet (031 comment) [tools/release] - 10https://gerrit.wikimedia.org/r/732757 (owner: 10Ahmon Dancy) [16:31:15] (03CR) 1020after4: [C: 03+2] deploy-promote: Only pass --proxy to curl when running on wmnet [tools/release] - 10https://gerrit.wikimedia.org/r/732757 (owner: 10Ahmon Dancy) [16:32:01] (03Merged) 10jenkins-bot: deploy-promote: Only pass --proxy to curl when running on wmnet [tools/release] - 10https://gerrit.wikimedia.org/r/732757 (owner: 10Ahmon Dancy) [16:33:46] (03PS2) 10Ahmon Dancy: deploy-promote: Set -o pipefail [tools/release] - 10https://gerrit.wikimedia.org/r/732750 [16:33:53] (03CR) 10Ahmon Dancy: [C: 03+2] deploy-promote: Set -o pipefail [tools/release] - 10https://gerrit.wikimedia.org/r/732750 (owner: 10Ahmon Dancy) [16:34:59] (03Merged) 10jenkins-bot: deploy-promote: Set -o pipefail [tools/release] - 10https://gerrit.wikimedia.org/r/732750 (owner: 10Ahmon Dancy) [16:35:45] (03PS1) 10Dduvall: Add chart patch reviewers by account ID instead of email [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/734695 (https://phabricator.wikimedia.org/T294300) [16:37:21] (03CR) 10Dduvall: "It wasn't clear from the Gerrit documentation that using anything other than an email address for the label value would work, but some loc" [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/734695 (https://phabricator.wikimedia.org/T294300) (owner: 10Dduvall) [17:00:46] 10Release-Engineering-Team (Done by Thu 04 Nov), 10Security-Team, 10GitLab (CI & Job Runners), 10User-brennen: Enable runners for projects under gitlab.wikimedia.org security group - https://phabricator.wikimedia.org/T294050 (10sbassett) Hey @brennen @dduvall - looks like we might have runners again? [[ h... [17:01:18] (03CR) 10Ahmon Dancy: [C: 03+1] "LGTM. Lemme know when you're ready for a +2." [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/734695 (https://phabricator.wikimedia.org/T294300) (owner: 10Dduvall) [17:02:14] (03CR) 10Dduvall: Add chart patch reviewers by account ID instead of email (031 comment) [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/734695 (https://phabricator.wikimedia.org/T294300) (owner: 10Dduvall) [17:10:17] (03CR) 10Ahmon Dancy: [C: 03+2] Add chart patch reviewers by account ID instead of email [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/734695 (https://phabricator.wikimedia.org/T294300) (owner: 10Dduvall) [17:10:56] (03Merged) 10jenkins-bot: Add chart patch reviewers by account ID instead of email [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/734695 (https://phabricator.wikimedia.org/T294300) (owner: 10Dduvall) [17:11:09] dancy: ty! [17:11:36] 👍🏾 [17:22:31] (03PS11) 10Jeena Huneidi: mw is deployable to k8s in traindev [tools/train-dev] - 10https://gerrit.wikimedia.org/r/726025 (https://phabricator.wikimedia.org/T287993) [17:54:29] 10Release-Engineering-Team (Done by Thu 04 Nov), 10Security-Team, 10GitLab (CI & Job Runners), 10User-brennen: Enable runners for projects under gitlab.wikimedia.org security group - https://phabricator.wikimedia.org/T294050 (10brennen) Not quite. We have runners reconfigured at the moment, trying to work... [18:12:34] (03CR) 10Ahmon Dancy: mw is deployable to k8s in traindev (0312 comments) [tools/train-dev] - 10https://gerrit.wikimedia.org/r/726025 (https://phabricator.wikimedia.org/T287993) (owner: 10Jeena Huneidi) [18:25:21] 10Release-Engineering-Team (Priority Backlog 🔥), 10Release Pipeline: "promote" step failure with multiple users per email - https://phabricator.wikimedia.org/T294300 (10nnikkhoui) that was so quick, thank you @dduvall ! [18:59:06] (03CR) 10Jeena Huneidi: mw is deployable to k8s in traindev (031 comment) [tools/train-dev] - 10https://gerrit.wikimedia.org/r/726025 (https://phabricator.wikimedia.org/T287993) (owner: 10Jeena Huneidi) [19:08:08] 10Gerrit: Gerrit Authentication failure - https://phabricator.wikimedia.org/T294384 (10EAkinloose) [19:09:20] 10Gerrit: Gerrit Authentication failure - https://phabricator.wikimedia.org/T294384 (10Dzahn) Hi, which user name(s) are you trying and what error do you get? Can you add error messages or screenshots and the details of the wikitech accounts? [19:17:32] 10Release-Engineering-Team (Doing), 10Security-Team, 10GitLab (CI & Job Runners), 10SecTeam-Processed, 10User-brennen: Limit GitLab shared runners to trusted contributors - https://phabricator.wikimedia.org/T292094 (10sbassett) [19:23:55] 10Release-Engineering-Team (Next), 10GitLab-Test, 10GitLab (Integrations), 10SecTeam-Processed, 10User-brennen: Experiment with package publishing workflows on GitLab - https://phabricator.wikimedia.org/T264131 (10sbassett) Untagging #security-team for now until this becomes a higher priority and more sp... [19:29:00] 10Gerrit: Gerrit Authentication failure - https://phabricator.wikimedia.org/T294384 (10hashar) https://logstash.wikimedia.org/app/discover#/doc/0fade920-6712-11eb-8327-370b46f9e7a5/ecs-1.7.0-5-default-2021.43?id=TtAHvnwB9aenX452HQVY Is an attempt to login as `Esther Akinloose` which gives: `counterexample com.g... [19:29:15] mutante: somehow that Gerrit account is disabled in Gerrit [19:32:42] (03CR) 10Ahmon Dancy: mw is deployable to k8s in traindev (031 comment) [tools/train-dev] - 10https://gerrit.wikimedia.org/r/726025 (https://phabricator.wikimedia.org/T287993) (owner: 10Jeena Huneidi) [19:33:16] 10Gerrit, 10Release-Engineering-Team: Gerrit Authentication failure - https://phabricator.wikimedia.org/T294384 (10hashar) [19:34:48] 10Gerrit, 10Release-Engineering-Team: Gerrit Authentication failure - https://phabricator.wikimedia.org/T294384 (10EAkinloose) https://wikitech.wikimedia.org/wiki/User:Esther_Akinloose. {F34712321} on Gerrit [19:35:50] 10Gerrit, 10Release-Engineering-Team: Gerrit Authentication failure - https://phabricator.wikimedia.org/T294384 (10EAkinloose) How do I go about reactivating the account @hashar ? [19:36:05] hashar: oh, interesting. oops. thanks for the warning [19:38:21] mutante: do we have an audit log of people we force logout / deactivate when their machine get stolen? [19:38:35] pretty sure we have a script to automagically logout/deactivate [19:39:34] hashar: well, deactivating shell accounts is a change in git logs [19:39:42] but I dont know about the emergency one [19:44:52] 10Release-Engineering-Team (Seen), 10GitLab (CI & Job Runners), 10SecTeam-Processed, 10Security, 10User-brennen: Document long-term requirements for GitLab job runners - https://phabricator.wikimedia.org/T286958 (10sbassett) [19:45:33] 10Release-Engineering-Team (Seen), 10GitLab (CI & Job Runners), 10SecTeam-Processed, 10Security, 10User-brennen: Document long-term requirements for GitLab job runners - https://phabricator.wikimedia.org/T286958 (10Mstyles) The security team would be more than happy to review the documentation when it ge... [19:45:35] 10Gerrit, 10Release-Engineering-Team: Gerrit Authentication failure - https://phabricator.wikimedia.org/T294384 (10Reedy) I suspect this might be fallout from when Esther's laptop was stolen, and I proactively banned her account on Wikitech in an abundance of caution. Which will have triggered the Gerrit API... [19:47:24] 10Gerrit, 10Release-Engineering-Team: Gerrit Authentication failure - https://phabricator.wikimedia.org/T294384 (10hashar) Found it! That user has been blocked with the reason "laptop stolen" from https://wikitech.wikimedia.org/wiki/Special:Log?type=&user=&page=User%3AEsther+Akinloose&wpdate=&tagfilter=&wpfilt... [19:48:19] Reedy: the account is not in the disabled policy, I guess it got manually disabled in Gerrit? [19:48:28] No [19:48:45] The magic hook in https://noc.wikimedia.org/conf/highlight.php?file=wikitech.php [19:48:56] $gerritUrl = 'https://gerrit.wikimedia.org'; [19:48:56] $username = strtolower( $block->getTargetName() ); [19:48:56] $ch = curl_init( [19:48:56] "{$gerritUrl}/r/a/accounts/" . urlencode( $username ) . '/active' [19:48:56] ); [19:48:57] curl_setopt( $ch, CURLOPT_RETURNTRANSFER, true ); [19:48:59] curl_setopt( $ch, CURLOPT_CUSTOMREQUEST, 'DELETE' ); [19:49:07] holy [19:49:29] I guess it's mostly designed that these accounts won't be "unblocked" [19:49:46] so we would need a hook when a user is unblocked or the block expires (if that is even possible) [19:50:04] mystery solved [19:50:07] We could, in theory [19:50:22] then how do we confirm the user is the actual user? ;D [19:50:24] I don't know if it's necessarily worth the effort (but it might be) for the amount of unblocks [19:50:39] Email their WMF email? :P [19:51:18] you can give it to SRE clinic duty as an access request if you want [19:51:32] they do this user verification thing for other tickets [19:52:40] oh my bad she is in QTE team :) [19:52:44] lol [19:52:46] by mailing and/or checking corp LDAP and/or asking HR. there is also the "quick hangout" method but it fails if you never met them [19:52:53] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Doing), 10Release Pipeline, 10Security-Team, and 3 others: Figure out the future of (or replacements for) PipelineLib in a GitLab world - https://phabricator.wikimedia.org/T287211 (10sbassett) >>! In T287211#7386319, @brennen wrote: > Tag... [19:53:09] does that change anything ? heh [19:54:04] I can do the verification with her tomorrow looping in one of the other person from QTE [19:54:17] or anyone really [19:54:27] I guess it is all about establishing a web of trust [19:54:29] ah :) [19:54:36] yea [19:54:53] optionally GPG signing party [19:55:30] E_KEY_EXPIRED [19:55:36] segmentation fault [19:55:49] Could not find key OxF999A3E [19:56:24] heh, yea, keyservers are annoying [19:56:32] we are now using our own repo for them though [19:58:36] 10Gerrit, 10Release-Engineering-Team: Gerrit Authentication failure - https://phabricator.wikimedia.org/T294384 (10hashar) @EAkinloose we need to verify that you are who you are. The easiest I guess is to do it over a video call with some other person from QTE that can validate that you are indeed the owner o... [19:58:53] commented about it on the task [19:58:59] mutante: Reedy thank you! [19:59:01] I am off ;) [19:59:37] enjoy dinner [20:03:09] 10Gerrit, 10Release-Engineering-Team: Gerrit Authentication failure - https://phabricator.wikimedia.org/T294384 (10EAkinloose) Thanks. I'll reach out to a member of my team for that. [20:18:30] (03PS1) 10Reedy: Attempt to setup CI for wikimedia/security/security-api [integration/config] - 10https://gerrit.wikimedia.org/r/734728 [20:23:40] (03CR) 10Reedy: [C: 03+2] Attempt to setup CI for wikimedia/security/security-api [integration/config] - 10https://gerrit.wikimedia.org/r/734728 (owner: 10Reedy) [20:25:30] (03Merged) 10jenkins-bot: Attempt to setup CI for wikimedia/security/security-api [integration/config] - 10https://gerrit.wikimedia.org/r/734728 (owner: 10Reedy) [20:26:19] !log Reloading Zuul to deploy https://gerrit.wikimedia.org/r/734728 [20:26:21] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:26:53] 10Gerrit, 10Release-Engineering-Team: Gerrit Authentication failure - https://phabricator.wikimedia.org/T294384 (10hashar) 05Open→03Resolved a:03hashar I have confirmed Esther over video, reactivated her account and she is in!!! [20:39:44] Reedy: mutante: Esther can access gerrit again. Thank you! [20:41:04] * hashar sleeps [20:41:53] 10MediaWiki-Releasing, 10AbuseFilter (Overhaul-2020), 10MW-1.38-release, 10Patch-For-Review: Bundle AbuseFilter extension with MediaWiki - https://phabricator.wikimedia.org/T191740 (10Daimona) [20:42:11] 10MediaWiki-Releasing, 10AbuseFilter (Overhaul-2020), 10MW-1.38-release, 10Patch-For-Review: Bundle AbuseFilter extension with MediaWiki - https://phabricator.wikimedia.org/T191740 (10Daimona) [20:44:32] (03PS12) 10Jeena Huneidi: mw is deployable to k8s in traindev [tools/train-dev] - 10https://gerrit.wikimedia.org/r/726025 (https://phabricator.wikimedia.org/T287993) [20:48:20] (03CR) 10Jeena Huneidi: mw is deployable to k8s in traindev (0311 comments) [tools/train-dev] - 10https://gerrit.wikimedia.org/r/726025 (https://phabricator.wikimedia.org/T287993) (owner: 10Jeena Huneidi) [20:55:33] (03PS13) 10Jeena Huneidi: mw is deployable to k8s in traindev [tools/train-dev] - 10https://gerrit.wikimedia.org/r/726025 (https://phabricator.wikimedia.org/T287993) [20:58:10] (03PS1) 10Reedy: Review access change [wikimedia/security/security-api] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/734685 [20:58:20] (03CR) 10Reedy: [V: 03+2 C: 03+2] Review access change [wikimedia/security/security-api] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/734685 (owner: 10Reedy) [20:58:37] hashar: If you're still about... Can you submit ^ please [20:58:51] yeah [20:59:43] (03PS2) 10Hashar: Change parent project from All-Projects to wikimedia/security [wikimedia/security/security-api] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/734685 (owner: 10Reedy) [20:59:51] (03CR) 10Hashar: [V: 03+2] Change parent project from All-Projects to wikimedia/security [wikimedia/security/security-api] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/734685 (owner: 10Reedy) [21:00:01] Reedy: done [21:00:14] merci! [21:00:27] https://gerrit.wikimedia.org/r/admin/repos/wikimedia/security/security-api,access [21:00:41] apparently Gerrit code is good enough to invalidate caches ;D [21:01:55] heh [21:02:24] sleeping time! have good hacks [21:04:47] (03CR) 10Ahmon Dancy: "I'm ready to +2 after this." [tools/train-dev] - 10https://gerrit.wikimedia.org/r/726025 (https://phabricator.wikimedia.org/T287993) (owner: 10Jeena Huneidi) [21:07:34] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Doing), 10Patch-For-Review: Replace deployment-imagescaler03 (stretch) with deployment-imagescaler04 (buster) - https://phabricator.wikimedia.org/T294148 (10dancy) `puppet agent -t` runs without errors on deployment-imagescaler04 now. Next up: Fig... [21:33:52] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Doing): Replace deployment-imagescaler03 (stretch) with deployment-imagescaler04 (buster) - https://phabricator.wikimedia.org/T294148 (10dancy) Using the python3-logstash package didn't work because thumbor itself is using python 2. [21:42:51] 10Release-Engineering-Team (Done by Thu 04 Nov), 10dev-images, 10mwcli, 10User-brennen: Add php-luasandbox to dev-images used by mwcli - https://phabricator.wikimedia.org/T286678 (10brennen) [21:48:30] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Doing): Replace deployment-imagescaler03 (stretch) with deployment-imagescaler04 (buster) - https://phabricator.wikimedia.org/T294148 (10dancy) [22:20:27] (03PS14) 10Jeena Huneidi: mw is deployable to k8s in traindev [tools/train-dev] - 10https://gerrit.wikimedia.org/r/726025 (https://phabricator.wikimedia.org/T287993) [22:21:00] (03CR) 10Jeena Huneidi: mw is deployable to k8s in traindev (033 comments) [tools/train-dev] - 10https://gerrit.wikimedia.org/r/726025 (https://phabricator.wikimedia.org/T287993) (owner: 10Jeena Huneidi) [22:27:17] (03CR) 10Ahmon Dancy: [C: 03+2] mw is deployable to k8s in traindev [tools/train-dev] - 10https://gerrit.wikimedia.org/r/726025 (https://phabricator.wikimedia.org/T287993) (owner: 10Jeena Huneidi) [22:27:43] (03Merged) 10jenkins-bot: mw is deployable to k8s in traindev [tools/train-dev] - 10https://gerrit.wikimedia.org/r/726025 (https://phabricator.wikimedia.org/T287993) (owner: 10Jeena Huneidi) [22:27:51] (03CR) 10Ahmon Dancy: "Very exciting!" [tools/train-dev] - 10https://gerrit.wikimedia.org/r/726025 (https://phabricator.wikimedia.org/T287993) (owner: 10Jeena Huneidi) [22:34:02] 10Release-Engineering-Team (Done by Thu 04 Nov), 10MW-on-K8s, 10MediaWiki Train Development Environment, 10Release Pipeline, and 2 others: Train-dev: Ability to deploy to k8s - https://phabricator.wikimedia.org/T287993 (10jeena) 05In progress→03Resolved [22:34:03] 10Release-Engineering-Team (Done by Thu 04 Nov), 10MW-on-K8s, 10Release Pipeline, 10User-brennen: Scap backport change_url command - https://phabricator.wikimedia.org/T287042 (10jeena) [22:47:10] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Doing): Replace deployment-imagescaler03 (stretch) with deployment-imagescaler04 (buster) - https://phabricator.wikimedia.org/T294148 (10Legoktm) Image scalers are still running stretch in production: {T216815}. If the goal is to do this in prep for up... [22:59:21] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Doing): Replace deployment-imagescaler03 (stretch) with deployment-imagescaler04 (buster) - https://phabricator.wikimedia.org/T294148 (10Legoktm) [22:59:43] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Doing): Replace deployment-imagescaler03 (stretch) with deployment-imagescaler04 (buster) - https://phabricator.wikimedia.org/T294148 (10dancy) >>! In T294148#7460168, @Legoktm wrote: > Image scalers are still running stretch in production: {T216815}.... [23:16:32] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Doing): Replace deployment-imagescaler03 (stretch) with deployment-imagescaler04 (buster) - https://phabricator.wikimedia.org/T294148 (10Legoktm) >>! In T294148#7460181, @dancy wrote: > Functionally yes. I don't think that means that the Debian releas... [23:22:35] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Doing): Replace deployment-imagescaler03 (stretch) with deployment-imagescaler04 (buster) - https://phabricator.wikimedia.org/T294148 (10dancy) >>! In T294148#7460212, @Legoktm wrote: > Well all of those seem available in stretch, so something else is... [23:23:05] (03PS1) 10Legoktm: debian: Remove build dependencies from scap binary package [tools/scap] - 10https://gerrit.wikimedia.org/r/734774 (https://phabricator.wikimedia.org/T294148) [23:23:07] (03PS1) 10Legoktm: debian: Remove outdated dependencies [tools/scap] - 10https://gerrit.wikimedia.org/r/734775 [23:23:26] (03PS2) 10Legoktm: debian: Remove outdated Suggests [tools/scap] - 10https://gerrit.wikimedia.org/r/734775 [23:24:09] dancy: ^^ [23:24:15] Taking a look [23:27:13] (03CR) 10Ahmon Dancy: [C: 03+2] debian: Remove build dependencies from scap binary package [tools/scap] - 10https://gerrit.wikimedia.org/r/734774 (https://phabricator.wikimedia.org/T294148) (owner: 10Legoktm) [23:27:19] (03CR) 10Ahmon Dancy: [C: 03+2] debian: Remove outdated Suggests [tools/scap] - 10https://gerrit.wikimedia.org/r/734775 (owner: 10Legoktm) [23:27:38] 10Release-Engineering-Team (Priority Backlog 🔥), 10Release Pipeline: "promote" step failure with multiple users per email - https://phabricator.wikimedia.org/T294300 (10dduvall) 05In progress→03Resolved >>! In T294300#7459458, @nnikkhoui wrote: > that was so quick, thank you @dduvall ! No problem! [23:27:58] (03Merged) 10jenkins-bot: debian: Remove build dependencies from scap binary package [tools/scap] - 10https://gerrit.wikimedia.org/r/734774 (https://phabricator.wikimedia.org/T294148) (owner: 10Legoktm) [23:29:05] I figured out the actual problem [23:29:27] Thanks for looking into that! A very simple solution. [23:29:31] thumbor needs/wants sphinx from stretch backports, but it wasn't fully backported? [23:29:37] (03Merged) 10jenkins-bot: debian: Remove outdated Suggests [tools/scap] - 10https://gerrit.wikimedia.org/r/734775 (owner: 10Legoktm) [23:29:47] but I'm not going to keep looking, since removing the dependencies fixes it too :) [23:30:11] but it wasn't fully backported...so when scap tries to install it, it runs into version incompatibilities [23:31:29] Any opinions on the disposition of T294148 then? Sounds like it is not worth persuing. [23:31:29] T294148: Replace deployment-imagescaler03 (stretch) with deployment-imagescaler04 (buster) - https://phabricator.wikimedia.org/T294148 [23:32:25] personally I would not embark down that journey [23:33:09] Maybe just retitle it as "latest scap not installable on deployment-imagescaler03" and we can call it resolved? [23:33:09] OK I will heed your wisdom. Really appreciate your help. [23:33:20] well, you just fixed that problem. [23:33:26] :)) yw [23:34:07] (ftr: a conversation is in the beginning stages re the future of thumbor, we'll see where it goes) [23:34:21] ((future stewardship, that is)) [23:34:40] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Doing), 10Patch-For-Review: Replace deployment-imagescaler03 (stretch) with deployment-imagescaler04 (buster) - https://phabricator.wikimedia.org/T294148 (10dancy) I was able to install the latest scap deb (build by beta-scap-build-deb) on deployment... [23:36:37] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Radar): Migrate deployment-prep away from Debian Stretch to Buster/Bullseye - https://phabricator.wikimedia.org/T278641 (10dancy) [23:36:39] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Doing), 10Patch-For-Review: Replace deployment-imagescaler03 (stretch) with deployment-imagescaler04 (buster) - https://phabricator.wikimedia.org/T294148 (10dancy) 05Open→03Declined The original motivation for this ticket has been resolved in a s... [23:36:57] dancy: one other question, is it OK that an older version of scap is on the thumbor servers? do we need to upgrade it to 4.0.2 now or leave it for the next release? [23:37:35] An older version on the thumbor servers is ok. Most scap changes are only relevant to the deploy server itself. [23:38:07] greg-g, good to hear, hope something comes out of it. [23:42:42] AntiComposite: me too! [23:44:07] 10Release-Engineering-Team (Done by Thu 04 Nov), 10Release Pipeline: "promote" step failure with multiple users per email - https://phabricator.wikimedia.org/T294300 (10thcipriani)