[00:03:16] Striker does stuff with that api when it creates a diffusion repo -- https://github.com/wikimedia/labs-striker/blob/master/striker/phabricator.py#L189-L210 [00:04:29] striker: a topic it is on my list to know something about this week. [00:05:17] brennen: happy to talk, especially if this ends up being related to T296893 [00:05:17] T296893: Replace Diffusion integration with Gitlab integration in Striker (toolsadmin) - https://phabricator.wikimedia.org/T296893 [00:06:08] yeah, that's the thing i need to get my head around [00:07:58] brennen: I think we could chat for 30 minutes and save you hours of trying to figure out wtf yourself :) [00:08:45] bd808: yeah, that'd be awesome. i just realized it's after 5pm here and i've got a toddler's birthday party to run to, assuming the car will start at 5°F, but if you've got 30 min to spare this week i'd happily grab it. [00:09:33] brennen: my calendar is up to date. pick a time when you are at your keyboard tomorrow and send me an invite. [00:09:41] cool, incoming [00:11:31] gah, wrong day [00:11:44] (03CR) 10Ahmon Dancy: "Tested ok. Just need to please the linter." [tools/scap] - 10https://gerrit.wikimedia.org/r/764910 (https://phabricator.wikimedia.org/T294466) (owner: 10Jeena Huneidi) [00:13:00] (fixed. i am good at calendars, i swear.) [00:14:09] brennen: +2. now get to that bday party :) [00:14:23] yep, outta here. have a good evening all. [00:16:22] happy to see you are talking about striker and repos :)) [00:16:39] because when I look at phab and sort by active.. yea.. a bunch of the tools repos [00:16:46] so that sounds cool, yay [00:17:19] meanwhile mailed some other random user, not striker / not releng about moving [00:21:06] mutante: I have not forgotten about your desire to kill of diffusion :) [00:21:12] *off [00:21:49] cool, thank you! cya [01:47:05] 10Release-Engineering-Team (Done by Feb 23🔥), 10Patch-For-Review, 10Release, 10Train Deployments: 1.38.0-wmf.23 deployment blockers - https://phabricator.wikimedia.org/T300199 (10Legoktm) >>! In T300199#7727027, @Legoktm wrote: > I added T210075 as a train blocker because https://gerrit.wikimedia.org/r/c/m... [04:07:57] 10Beta-Cluster-Infrastructure: Requesting access to deployment-prep for sbassett - https://phabricator.wikimedia.org/T302361 (10sbassett) [04:20:14] 10Release-Engineering-Team (Next), 10Release, 10Train Deployments: 1.38.0-wmf.24 deployment blockers - https://phabricator.wikimedia.org/T300200 (10Jdlrobson) [05:10:12] 10Beta-Cluster-Infrastructure, 10Beta-Cluster-reproducible: Beta cluster MediaWiki code not updating (again) - https://phabricator.wikimedia.org/T302364 (10Tgr) [05:15:20] 10Beta-Cluster-Infrastructure, 10Beta-Cluster-reproducible: Beta cluster MediaWiki code not updating (again) - https://phabricator.wikimedia.org/T302364 (10Tgr) Confirmed that scap is working; this is a Jenkins issue. [05:24:08] 10Beta-Cluster-Infrastructure, 10Beta-Cluster-reproducible: Beta cluster MediaWiki code not updating (again) - https://phabricator.wikimedia.org/T302364 (10Tgr) p:05Triage→03Unbreak! The Jenkins build queue GUI complains about executor starvation. There were neither successful not failed beta job runs in t... [06:19:44] 10Beta-Cluster-Infrastructure, 10Beta-Cluster-reproducible: Beta cluster MediaWiki code not updating (again) - https://phabricator.wikimedia.org/T302364 (10ArielGlenn) /srv/mediawiki-staging/php-master on deployment-deploy03 looks up to date now when I compare to a local pull of mediawiki-core. Maybe this has... [06:43:37] 10Release-Engineering-Team (Radar), 10MW-on-K8s, 10SRE, 10serviceops, and 2 others: The restricted/mediawiki-webserver image should include skins and resources - https://phabricator.wikimedia.org/T285232 (10Joe) 05Open→03Resolved a:03Joe [07:38:33] (03PS1) 10Robert Vogel: Add new "Tuleap*" skin and extensions to CI [integration/config] - 10https://gerrit.wikimedia.org/r/765192 [07:38:55] 10Beta-Cluster-Infrastructure: Requesting access to deployment-prep for sbassett - https://phabricator.wikimedia.org/T302361 (10Legoktm) You're already listed on https://openstack-browser.toolforge.org/project/deployment-prep so you probably already do have access :-) [08:37:28] !log Removing Stretch based integration-agent-qemu-1001 # T284774 [08:37:31] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [08:37:31] T284774: Provide one or more Qemu agents in CI that use a newer version than 2.x - https://phabricator.wikimedia.org/T284774 [09:04:46] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Seen): Move all Wikimedia CI (WMCS integration project) instances from stretch to buster/bullseye - https://phabricator.wikimedia.org/T252071 (10hashar) [10:21:28] !log Created Bullseye instance integration-agent-puppet-docker-1003 https://horizon.wikimedia.org/project/instances/96cf9ddc-daa3-4c9f-8c21-cdd58e95973e/ # T252071 [10:21:31] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [10:21:31] T252071: Move all Wikimedia CI (WMCS integration project) instances from stretch to buster/bullseye - https://phabricator.wikimedia.org/T252071 [10:29:03] (03PS2) 10Jeena Huneidi: Scap backport: legacy deployment [tools/scap] - 10https://gerrit.wikimedia.org/r/764910 (https://phabricator.wikimedia.org/T294466) [10:49:07] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Seen): Move all Wikimedia CI (WMCS integration project) instances from stretch to buster/bullseye - https://phabricator.wikimedia.org/T252071 (10hashar) I have created and pooled integration-agent-puppet-docker-1003 so we can put it online an... [10:49:38] 10Release-Engineering-Team (Deployment Training Requests): Deployment training request for Hannah Okwelum - https://phabricator.wikimedia.org/T302275 (10Hokwelum) [10:51:49] (03CR) 10Jeena Huneidi: "I've been able to test some things but haven't been able to test the building and pushing of the image because docker won't start on my vm" [tools/scap] - 10https://gerrit.wikimedia.org/r/763609 (https://phabricator.wikimedia.org/T297673) (owner: 10Ahmon Dancy) [11:06:35] 10Phabricator, 10Developer-Advocacy (Jan-Mar 2022): Followup fixes to weekly 2FA Phab cronjob - https://phabricator.wikimedia.org/T302385 (10Aklapper) p:05Triage→03Low [11:33:35] 10Phabricator, 10Developer-Advocacy (Jan-Mar 2022): Followup fixes to weekly 2FA Phab cronjob - https://phabricator.wikimedia.org/T302385 (10Aklapper) [12:00:47] 10Release-Engineering-Team (Done by Feb 23🔥), 10Patch-For-Review, 10Release, 10Train Deployments: 1.38.0-wmf.23 deployment blockers - https://phabricator.wikimedia.org/T300199 (10Nikerabbit) [12:05:55] 10Release-Engineering-Team (Done by Feb 23🔥), 10Patch-For-Review, 10Release, 10Train Deployments: 1.38.0-wmf.23 deployment blockers - https://phabricator.wikimedia.org/T300199 (10Reedy) [12:06:32] 10Release-Engineering-Team (Done by Feb 23🔥), 10Patch-For-Review, 10Release, 10Train Deployments: 1.38.0-wmf.23 deployment blockers - https://phabricator.wikimedia.org/T300199 (10Reedy) [12:29:36] hello! I'm having some issues rendering a template via scap and I was wondering is there a way to dump the values that are being injected into the template that are causing the error? I'm seeing this https://phabricator.wikimedia.org/P21371 but it seems like it's a problem with the variables [12:31:06] hnowlan: you'll find out later, meet meet? :-P [12:41:33] !log Depooling integration-agent-puppet-docker-1002 , pooling integration-agent-puppet-docker-1003 # T252071 [12:41:36] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [12:41:36] T252071: Move all Wikimedia CI (WMCS integration project) instances from stretch to buster/bullseye - https://phabricator.wikimedia.org/T252071 [12:43:07] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Seen): Move all Wikimedia CI (WMCS integration project) instances from stretch to buster/bullseye - https://phabricator.wikimedia.org/T252071 (10hashar) [12:44:00] 10Beta-Cluster-Infrastructure, 10Beta-Cluster-reproducible: Beta cluster MediaWiki code not updating (again) - https://phabricator.wikimedia.org/T302364 (10Zabe) Seems like it's working again. [14:54:16] Further to the above I'm a bit stumped - I've dumped the config and the varibles to be injected into it and neither of them contain other YAML documents or stray `---` entries or anything [15:05:31] I'm actually seeing that error when doing a regular deploy to an existing host that has previously had successful deploys to it also so it might be a larger issue [15:32:31] 10Beta-Cluster-Infrastructure, 10Beta-Cluster-reproducible: Beta cluster MediaWiki code not updating (again) - https://phabricator.wikimedia.org/T302364 (10Jdforrester-WMF) 05Open→03Resolved Let's mark this as Resolved for now, then. Probably just the usual CI backlog self-fixing. [15:40:00] hnowlan>: I will take a look [15:40:15] dancy: thanks! [15:40:42] you can replicate via `deployment-deploy03:/srv/deployment/restbase/deploy$ scap deploy --environment beta` [15:41:01] thx [15:46:24] The first thing I get when I run that is: `connection to deployment-restbase04.deployment-prep.eqiad.wmflabs failed ` [15:47:15] s/eqiad.wmflabs/eqiad1.wikimedia.cloud/ should fix that [15:53:58] good catch, those should be fixed [15:56:43] 10Release-Engineering-Team (Deployment Training Requests): Deployment training request for Hannah Okwelum - https://phabricator.wikimedia.org/T302275 (10thcipriani) >>! In T302275#7727828, @ArielGlenn wrote: > Hey Tyler and other folks, this is our co-dumps-maintainer in training, so to speak, so I'm getting her... [15:57:56] hnowlan: and restbase03 needs to still use wmflabs, it's an older host so that is what the entry in ssh_known_hosts is using [15:58:07] oh heh [16:00:49] 10Release-Engineering-Team (Deployment Training Requests): Deployment training request for Hannah Okwelum - https://phabricator.wikimedia.org/T302275 (10thcipriani) p:05Triage→03Medium @Hokwelum you should have an invite to the training on Feb 24th—let me know if anything seems amiss. [16:00:51] dancy: now it's failing with the "original" error [16:01:19] thx. digging in [16:01:36] 10Release-Engineering-Team (Deployment Training Requests): Deployment training request for mfossati - https://phabricator.wikimedia.org/T302204 (10thcipriani) p:05Triage→03Medium Hi @mfossati you should have an invite to the training on February 24. Adding @ArielGlenn as a heads-up. [16:05:58] hnowlan: in case this isn't on your radar yet, deployment-restbase04 isn't yet in the cassandra cluster [16:14:51] hnowlan: Found the problem. Doing some hacking for a few more minutes to make it easier to discover in the future. [16:15:19] It looks like you (or someone) already did some hacking in `/usr/lib/python3/dist-packages/scap/deploy.py` ? [16:15:51] (buggy hacking, I'll add) [16:17:15] The problem is in `/etc/restbase/config-vars.yaml` [16:17:53] There is a `---` in that file which start a new YAML document, which yaml.load() complains about [16:19:49] (03PS1) 10Cwhite: logstash-filter-verifier: install logstash-plugins [integration/config] - 10https://gerrit.wikimedia.org/r/765302 [16:19:49] dancy: ahhh nice (and yes, I did hack about with it badly :)) [16:19:50] `operations/puppet/modules/service/templates/node/config-vars.yaml.erb` has [16:19:50] ``` [16:19:50] <%- unless @deployment_vars.empty? -%> [16:19:50] # deployment variables [16:19:51] <%= @deployment_vars.to_yaml %> [16:19:51] <% end -%> [16:19:51] ``` [16:20:13] taavi: yep, wanted to get puppet passing first [16:20:19] That to_yaml seems to be generating a fresh yaml doc (by starting with `---`).. so somehow that needs to be fixed. [16:20:42] I'll improve scap so that it mentions the vars file when rendering a config file. [16:21:23] that seems to be a regression introduced by https://gerrit.wikimedia.org/r/c/operations/puppet/+/763362 [16:24:45] (03PS1) 10Ahmon Dancy: deploy: Mention what vars file is being used to render a config file [tools/scap] - 10https://gerrit.wikimedia.org/r/765304 [16:25:23] (03CR) 10jerkins-bot: [V: 04-1] deploy: Mention what vars file is being used to render a config file [tools/scap] - 10https://gerrit.wikimedia.org/r/765304 (owner: 10Ahmon Dancy) [16:26:26] (03PS2) 10Ahmon Dancy: deploy: Mention what vars file is being used to render a config file [tools/scap] - 10https://gerrit.wikimedia.org/r/765304 [16:27:52] aha, interesting. I'll follow up on that, breaks quite a bit of stuff [16:27:54] thanks both! [16:34:00] 10Gerrit, 10SRE, 10serviceops: replacement for gerrit2001 - https://phabricator.wikimedia.org/T243027 (10hashar) `gerrit2001.wikimedia.org` is a replica and can also be used as a spare to switch the primary service. It also serves repos over `gerrit-replica.wikimedia.org` which is used by various scripts an... [17:07:18] (03CR) 10Jeena Huneidi: "Just a couple nits but I think this is good to merge after the other patch is ready" [tools/train-dev] - 10https://gerrit.wikimedia.org/r/763846 (https://phabricator.wikimedia.org/T297673) (owner: 10Ahmon Dancy) [17:17:44] Project beta-scap-sync-world build #40387: 04FAILURE in 2 min 53 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/40387/ [17:18:52] 10Gerrit, 10Wikidata, 10Wikidata Query UI, 10wdwb-tech: wikidata-query-gui-build doesn’t work when latest commit is by dependabot (commit-msg hook adds Change-Id in wrong place) - https://phabricator.wikimedia.org/T295601 (10hashar) The hook in upstream is at https://gerrit.googlesource.com/gerrit/+/refs/h... [17:19:14] I thought we had a releng team meeting ! :D [17:19:49] It got emailed. :-) [17:19:59] I was very pleased to see a meeting disappear from the calendar. [17:20:38] oh great [17:20:46] so I am going to spend quality time with the kids instead [17:20:52] Excellent. [17:20:56] :-] [17:22:33] there is a docker image update pending for cwhite https://gerrit.wikimedia.org/r/c/integration/config/+/765302 :] [17:22:41] or I will do it later tonight [17:23:00] for now, I am going to revisite the multiplication tables of 2 and 5! [17:28:03] Project beta-scap-sync-world build #40388: 04STILL FAILING in 3 min 14 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/40388/ [17:28:38] ^ That is failing due to `ssh mwdeploy@deployment-parsoid12.deployment-prep.eqiad1.wikimedia.cloud: Permission denied (publickey)` [17:35:58] Project beta-scap-sync-world build #40389: 04STILL FAILING in 1 min 12 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/40389/ [17:46:00] Yippee, build fixed! [17:46:00] i can't access parsoid12 either :/ [17:46:00] Project beta-scap-sync-world build #40390: 09FIXED in 1 min 16 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/40390/ [17:46:21] and now i can [17:46:27] and build is also fixed [17:46:34] * urbanecm hides [17:47:55] thanks urbanecm for fixing it! you now own deployment-parsoid* [17:49:23] i literally just ran keyholder arm (which prompted me for analytics-deploy , dumpsdeploy, eventlogging and deploy-service passphrases) and then tried to ssh in the parsoid host [18:07:49] (03CR) 10Hnowlan: [C: 03+1] "lgtm, thanks!" [tools/scap] - 10https://gerrit.wikimedia.org/r/765304 (owner: 10Ahmon Dancy) [18:11:29] (03CR) 10Ahmon Dancy: [C: 03+2] deploy: Mention what vars file is being used to render a config file [tools/scap] - 10https://gerrit.wikimedia.org/r/765304 (owner: 10Ahmon Dancy) [18:12:12] (03Merged) 10jenkins-bot: deploy: Mention what vars file is being used to render a config file [tools/scap] - 10https://gerrit.wikimedia.org/r/765304 (owner: 10Ahmon Dancy) [18:19:25] (03PS3) 10Ahmon Dancy: Kill off build-mw-image-loop.py [tools/train-dev] - 10https://gerrit.wikimedia.org/r/763846 (https://phabricator.wikimedia.org/T297673) [18:19:35] 10GitLab (Infrastructure), 10DC-Ops, 10SRE, 10ops-eqiad, 10serviceops: Q3:(Need By: TBD) rack/setup/install gitlab100[2|3] and gitlab-runner100[2|3|4] - https://phabricator.wikimedia.org/T301177 (10RobH) [18:20:00] (03CR) 10Ahmon Dancy: Kill off build-mw-image-loop.py (032 comments) [tools/train-dev] - 10https://gerrit.wikimedia.org/r/763846 (https://phabricator.wikimedia.org/T297673) (owner: 10Ahmon Dancy) [18:21:41] 10GitLab (Infrastructure), 10DC-Ops, 10SRE, 10ops-codfw: Q3:(Need By: TBD) rack/setup/install gitlab200[2|3] and gitlab-runner200[2|3|4] - https://phabricator.wikimedia.org/T301183 (10RobH) [18:25:08] 10GitLab (Infrastructure), 10DC-Ops, 10SRE, 10ops-eqiad, 10serviceops: Q3:(Need By: TBD) rack/setup/install gitlab100[2|3] and gitlab-runner100[2|3|4] - https://phabricator.wikimedia.org/T301177 (10Dzahn) @RobH (cc: @Jelto ) gitlab1002 has existed as a VM in the past, when contractors used it but the... [18:26:50] (03PS1) 10Ahmon Dancy: Call self._compile_wikiversions() before self._check_fatals() [tools/scap] - 10https://gerrit.wikimedia.org/r/765318 [18:35:59] 10GitLab (Infrastructure), 10DC-Ops, 10SRE, 10ops-eqiad, 10serviceops: Q3:(Need By: TBD) rack/setup/install gitlab100[2|3] and gitlab-runner100[2|3|4] - https://phabricator.wikimedia.org/T301177 (10RobH) a:05Jclark-ctr→03LSobanski @lsobanski: Is it ok to shift these hostnames from gitlab100[23] to gi... [18:37:07] hnowlan: would you mind reviewing https://gerrit.wikimedia.org/r/c/operations/puppet/+/765317/? that fixes the yaml parsing/syntax issue [19:05:00] cwhite: hi, I am building that logstash CI image ;) [19:05:09] (03CR) 10Hashar: [C: 03+2] logstash-filter-verifier: install logstash-plugins [integration/config] - 10https://gerrit.wikimedia.org/r/765302 (owner: 10Cwhite) [19:06:40] Thanks! [19:06:56] (03Merged) 10jenkins-bot: logstash-filter-verifier: install logstash-plugins [integration/config] - 10https://gerrit.wikimedia.org/r/765302 (owner: 10Cwhite) [19:10:05] that whole process is a bit tedious [19:11:38] (03PS1) 10Hashar: jjb: update operations-puppet-logstash-filter-verifier [integration/config] - 10https://gerrit.wikimedia.org/r/765320 [19:13:58] (03PS2) 10Ahmon Dancy: Allow _check_fatals to work under more circumstances [tools/scap] - 10https://gerrit.wikimedia.org/r/765318 [19:14:42] (03CR) 10jerkins-bot: [V: 04-1] Allow _check_fatals to work under more circumstances [tools/scap] - 10https://gerrit.wikimedia.org/r/765318 (owner: 10Ahmon Dancy) [19:16:13] (03PS3) 10Ahmon Dancy: Allow _check_fatals to work under more circumstances [tools/scap] - 10https://gerrit.wikimedia.org/r/765318 [19:23:31] !log Built docker-registry.discovery.wmnet/releng/logstash-filter-verifier:0.0.3 [19:23:34] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:23:57] (03CR) 10Hashar: "I have updated the job to use the new image." [integration/config] - 10https://gerrit.wikimedia.org/r/765320 (owner: 10Hashar) [19:24:02] (03CR) 10Hashar: [C: 03+2] jjb: update operations-puppet-logstash-filter-verifier [integration/config] - 10https://gerrit.wikimedia.org/r/765320 (owner: 10Hashar) [19:24:21] cwhite: done and I have updated the jenkins job to point to the new image [19:25:50] (03Merged) 10jenkins-bot: jjb: update operations-puppet-logstash-filter-verifier [integration/config] - 10https://gerrit.wikimedia.org/r/765320 (owner: 10Hashar) [19:26:08] Thank you! [19:27:27] (03PS1) 10Kosta Harlan: jjb: Switch quibble-ci-fullrun jobs to use Apache backend [integration/config] - 10https://gerrit.wikimedia.org/r/765323 (https://phabricator.wikimedia.org/T299492) [19:27:54] https://wikitech.wikimedia.org/wiki/Phabricator#User_workflows | https://wikitech.wikimedia.org/wiki/Phabricator/pushing_over_https [19:28:34] ^ if you have a repo on Phabricator, please try this and push over https instead of ssh. separately consider moving it to gitlab in the long term. [19:30:27] 10Phabricator, 10Release-Engineering-Team (Next), 10serviceops: Deprecate git-ssh service on phabricator.wikimedia.org - https://phabricator.wikimedia.org/T296022 (10Dzahn) I created some docs how to git push over https (instead of ssh) to Phabricator. The idea is that we can give this to people who are sti... [19:32:36] (03PS2) 10Kosta Harlan: jjb: Switch quibble-ci-fullrun jobs to use Apache backend [integration/config] - 10https://gerrit.wikimedia.org/r/765323 (https://phabricator.wikimedia.org/T299492) [19:38:31] brennen: omg:) look at this: https://gitlab.wikimedia.org/dzahn/tool-cr-grants-team-metasync [19:38:40] guess what that is.. imported from phab [19:38:49] with history.. by just clicking [19:39:00] except my name should not be in the URL :P [19:39:13] one of the striker repos [19:39:49] guess I should really talk on the dedicated -gitlab channel. will move there [19:39:51] (aren't open and interoperable tools great?) [19:40:04] yea:) [19:40:13] it can also import phab tasks but you did not hear that [19:40:28] 10Release-Engineering-Team (Done by Feb 23🔥), 10MW-on-K8s, 10serviceops, 10Patch-For-Review: Build MediaWiki images for kubernetes on the deployment servers - https://phabricator.wikimedia.org/T297673 (10dancy) @joe I tried running this today but it failed: ` dancy@deploy1002$ sudo -u mwbuilder /usr/bin/ma... [19:41:14] why would anyone ever want to willingly switch from phab tasks to gitlab issues [19:41:48] Phab is pretty nice [19:41:59] so its gerrit too [19:42:04] N-1 tools [19:42:09] Agreed on all counts. [19:42:31] But apparently pushing to refs/for/ is too much to overcome. [19:43:26] (03CR) 10Ahmon Dancy: [C: 03+2] Add release_repo_build_and_push_images_cmd config parameter [tools/scap] - 10https://gerrit.wikimedia.org/r/763609 (https://phabricator.wikimedia.org/T297673) (owner: 10Ahmon Dancy) [19:44:08] (03Merged) 10jenkins-bot: Add release_repo_build_and_push_images_cmd config parameter [tools/scap] - 10https://gerrit.wikimedia.org/r/763609 (https://phabricator.wikimedia.org/T297673) (owner: 10Ahmon Dancy) [19:44:24] (03CR) 10Ahmon Dancy: [C: 03+2] Kill off build-mw-image-loop.py [tools/train-dev] - 10https://gerrit.wikimedia.org/r/763846 (https://phabricator.wikimedia.org/T297673) (owner: 10Ahmon Dancy) [19:44:48] (03Merged) 10jenkins-bot: Kill off build-mw-image-loop.py [tools/train-dev] - 10https://gerrit.wikimedia.org/r/763846 (https://phabricator.wikimedia.org/T297673) (owner: 10Ahmon Dancy) [19:47:32] (03PS1) 10Ahmon Dancy: Add mediawiki_image_extra_packages and mediawiki_image_extra_ca_cert [tools/scap] - 10https://gerrit.wikimedia.org/r/765325 [19:48:38] (03CR) 10Ahmon Dancy: [C: 03+2] Add mediawiki_image_extra_packages and mediawiki_image_extra_ca_cert [tools/scap] - 10https://gerrit.wikimedia.org/r/765325 (owner: 10Ahmon Dancy) [19:49:19] (03Merged) 10jenkins-bot: Add mediawiki_image_extra_packages and mediawiki_image_extra_ca_cert [tools/scap] - 10https://gerrit.wikimedia.org/r/765325 (owner: 10Ahmon Dancy) [19:49:30] (03PS1) 10Ahmon Dancy: update scap-prep-mw-branches to set up to run scap prep auto [tools/train-dev] - 10https://gerrit.wikimedia.org/r/765326 [19:49:32] (03PS1) 10Ahmon Dancy: scap.cfg: Inject debug packages and train-dev CA cert [tools/train-dev] - 10https://gerrit.wikimedia.org/r/765327 [19:51:29] (03PS2) 10Ahmon Dancy: update scap-prep-mw-branches to set up to run scap prep auto [tools/train-dev] - 10https://gerrit.wikimedia.org/r/765326 [19:51:31] (03PS2) 10Ahmon Dancy: scap.cfg: Inject debug packages and train-dev CA cert [tools/train-dev] - 10https://gerrit.wikimedia.org/r/765327 [19:52:02] (03CR) 10Ahmon Dancy: [C: 03+2] update scap-prep-mw-branches to set up to run scap prep auto [tools/train-dev] - 10https://gerrit.wikimedia.org/r/765326 (owner: 10Ahmon Dancy) [19:52:19] (03CR) 10Ahmon Dancy: [C: 03+2] scap.cfg: Inject debug packages and train-dev CA cert [tools/train-dev] - 10https://gerrit.wikimedia.org/r/765327 (owner: 10Ahmon Dancy) [19:52:26] (03Merged) 10jenkins-bot: update scap-prep-mw-branches to set up to run scap prep auto [tools/train-dev] - 10https://gerrit.wikimedia.org/r/765326 (owner: 10Ahmon Dancy) [19:52:43] (03Merged) 10jenkins-bot: scap.cfg: Inject debug packages and train-dev CA cert [tools/train-dev] - 10https://gerrit.wikimedia.org/r/765327 (owner: 10Ahmon Dancy) [20:00:48] (03PS1) 10QChris: Allow “Gerrit Managers” to import history [extensions/RegularTooltips] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/765329 [20:00:50] (03CR) 10QChris: [V: 03+2 C: 03+2] Allow “Gerrit Managers” to import history [extensions/RegularTooltips] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/765329 (owner: 10QChris) [20:00:52] (03PS1) 10QChris: Import done. Revoke import grants [extensions/RegularTooltips] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/765330 [20:00:54] (03CR) 10QChris: [V: 03+2 C: 03+2] Import done. Revoke import grants [extensions/RegularTooltips] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/765330 (owner: 10QChris) [20:10:49] !log Updating scap in beta [20:10:51] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:23:03] (03CR) 10Hashar: [C: 03+2] "INFO:jenkins_jobs.builder:Reconfiguring jenkins job integration-quibble-fullrun" [integration/config] - 10https://gerrit.wikimedia.org/r/765323 (https://phabricator.wikimedia.org/T299492) (owner: 10Kosta Harlan) [20:24:50] (03Merged) 10jenkins-bot: jjb: Switch quibble-ci-fullrun jobs to use Apache backend [integration/config] - 10https://gerrit.wikimedia.org/r/765323 (https://phabricator.wikimedia.org/T299492) (owner: 10Kosta Harlan) [20:29:38] (03CR) 10Kosta Harlan: "seems to not have worked :/ https://integration.wikimedia.org/ci/job/integration-quibble-fullrun-sqlite/9/console#console-section-10" [integration/config] - 10https://gerrit.wikimedia.org/r/765323 (https://phabricator.wikimedia.org/T299492) (owner: 10Kosta Harlan) [20:46:57] 10Beta-Cluster-Infrastructure: Requesting access to deployment-prep for sbassett - https://phabricator.wikimedia.org/T302361 (10sbassett) >>! In T302361#7730914, @Legoktm wrote: > You're already listed on https://openstack-browser.toolforge.org/project/deployment-prep so you probably already do have access :-)... [20:54:29] 10Beta-Cluster-Infrastructure: Requesting access to deployment-prep for sbassett - https://phabricator.wikimedia.org/T302361 (10Dzahn) @sbassett try this one instead to compare deployment-mediawiki12.deployment-prep that works for me, while the one you listed above I also get the error. I think it's just that... [20:55:05] 10Beta-Cluster-Infrastructure: Requesting access to deployment-prep for sbassett - https://phabricator.wikimedia.org/T302361 (10Zabe) It's deployment-deploy01.deployment-prep.eqiad1.wikimedia.cloud. You used eqiad instead of eqiad1. [20:55:07] 10Release-Engineering-Team (Done by Wed 24 Nov 🧟), 10Scap: Scap backport change_url: Implement “wait for my commits to be merged” - https://phabricator.wikimedia.org/T295494 (10jeena) a:03jeena [20:55:09] 10Beta-Cluster-Infrastructure: Requesting access to deployment-prep for sbassett - https://phabricator.wikimedia.org/T302361 (10Dzahn) @sbassett It's "deploy03" instead of "deploy01" meanwhile. https://horizon.wikimedia.org/project/instances/f72e5b18-6376-4ccd-9e59-64447759e53f/ [20:56:07] 10Release-Engineering-Team (Done by Feb 23🔥), 10Patch-For-Review, 10Release, 10Train Deployments: 1.38.0-wmf.23 deployment blockers - https://phabricator.wikimedia.org/T300199 (10dduvall) [21:11:47] 10Beta-Cluster-Infrastructure: Requesting access to deployment-prep for sbassett - https://phabricator.wikimedia.org/T302361 (10sbassett) 05Open→03Invalid Ah, thanks, @Zabe and @Dzahn. I haven't logged into these in a while. deploy03 and deployment-mediawiki12 with eqiad1 both work. [21:14:19] 10Phabricator, 10Release-Engineering-Team (Next), 10serviceops: Deprecate git-ssh service on phabricator.wikimedia.org - https://phabricator.wikimedia.org/T296022 (10Dzahn) @Osnard Ok, thank you! I have today tried to test import your repo and in principal it worked though under a namespace of my own user na... [21:26:33] (03PS1) 10Hashar: jjb: fix quibble-ci-fullrun entrypoint [integration/config] - 10https://gerrit.wikimedia.org/r/765337 [21:28:01] 10Phabricator, 10Release-Engineering-Team (Next), 10serviceops: Deprecate git-ssh service on phabricator.wikimedia.org - https://phabricator.wikimedia.org/T296022 (10Dzahn) [21:28:09] 10Phabricator, 10Release-Engineering-Team, 10serviceops: move "releng-secrets" git repo away from Phabricator - https://phabricator.wikimedia.org/T301170 (10Dzahn) [21:31:04] 10Phabricator, 10Release-Engineering-Team (Next), 10serviceops: Deprecate git-ssh service on phabricator.wikimedia.org - https://phabricator.wikimedia.org/T296022 (10Dzahn) [21:31:23] 10Phabricator, 10Release-Engineering-Team, 10serviceops: move "releng-secrets" git repo away from Phabricator (stop pushing over ssh to phab) - https://phabricator.wikimedia.org/T301170 (10Dzahn) 05Open→03Resolved a:03Dzahn So.. we have now tried the "push over https" to this repo and written docs for... [21:31:31] 10Phabricator, 10Release-Engineering-Team, 10serviceops: move "releng-secrets" git repo away from Phabricator (stop pushing over ssh to phab) - https://phabricator.wikimedia.org/T301170 (10Dzahn) [21:33:36] (03CR) 10Hashar: [C: 03+2] jjb: fix quibble-ci-fullrun entrypoint [integration/config] - 10https://gerrit.wikimedia.org/r/765337 (owner: 10Hashar) [21:35:22] (03Merged) 10jenkins-bot: jjb: fix quibble-ci-fullrun entrypoint [integration/config] - 10https://gerrit.wikimedia.org/r/765337 (owner: 10Hashar) [22:05:43] 10Release-Engineering-Team (Done by Feb 23🔥), 10MW-on-K8s, 10serviceops, 10Patch-For-Review: Build MediaWiki images for kubernetes on the deployment servers - https://phabricator.wikimedia.org/T297673 (10dancy) 05Open→03In progress p:05Medium→03High [22:10:53] (03CR) 10Thcipriani: [C: 03+2] "LGTM: one question inline" [tools/scap] - 10https://gerrit.wikimedia.org/r/765318 (owner: 10Ahmon Dancy) [22:11:34] (03Merged) 10jenkins-bot: Allow _check_fatals to work under more circumstances [tools/scap] - 10https://gerrit.wikimedia.org/r/765318 (owner: 10Ahmon Dancy) [22:32:55] (03PS1) 10Ahmon Dancy: sync-wikiversions: Remove redundant tasks.compile_wikiversions call [tools/scap] - 10https://gerrit.wikimedia.org/r/765345 [22:33:19] (03CR) 10Ahmon Dancy: Allow _check_fatals to work under more circumstances (031 comment) [tools/scap] - 10https://gerrit.wikimedia.org/r/765318 (owner: 10Ahmon Dancy) [22:35:19] (03PS2) 10Ahmon Dancy: sync-wikiversions: Remove redundant tasks.compile_wikiversions call [tools/scap] - 10https://gerrit.wikimedia.org/r/765345 [22:41:17] 10Phabricator, 10Release-Engineering-Team (Next), 10serviceops: Deprecate git-ssh service on phabricator.wikimedia.org - https://phabricator.wikimedia.org/T296022 (10Dzahn) This is a saved query to show all repos that are: - hosted on phabricator - active (however that is defined) - git (1 - 4 svn special c... [22:49:54] 10Beta-Cluster-Infrastructure: beta-build-scap-deb failing due to Error: lintian not available. - https://phabricator.wikimedia.org/T302460 (10dancy) [22:50:36] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team: beta-build-scap-deb failing due to Error: lintian not available. - https://phabricator.wikimedia.org/T302460 (10dancy) p:05Triage→03High [22:50:47] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team: beta-build-scap-deb failing due to Error: lintian not available. - https://phabricator.wikimedia.org/T302460 (10Majavah) p:05High→03Triage [22:51:06] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team: beta-build-scap-deb failing due to Error: lintian not available. - https://phabricator.wikimedia.org/T302460 (10Majavah) p:05Triage→03High [23:07:07] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team: beta-build-scap-deb failing due to Error: lintian not available. - https://phabricator.wikimedia.org/T302460 (10dancy) The end of /var/log/puppet.log has: ` Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install lint... [23:10:15] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team: beta-build-scap-deb failing due to Error: lintian not available. - https://phabricator.wikimedia.org/T302460 (10dancy) I ran `DEBIAN_FRONTEND=dialog dpkg --configure grub-pc` on integration-agent-pkgbuilder-1001/1002 and told it to install gr... [23:14:35] (03PS1) 10Jeena Huneidi: docker: don't use docker.io package [tools/train-dev] - 10https://gerrit.wikimedia.org/r/765350 [23:14:55] (03CR) 10jerkins-bot: [V: 04-1] docker: don't use docker.io package [tools/train-dev] - 10https://gerrit.wikimedia.org/r/765350 (owner: 10Jeena Huneidi) [23:16:00] (03PS2) 10Jeena Huneidi: docker: don't use docker.io package [tools/train-dev] - 10https://gerrit.wikimedia.org/r/765350 [23:18:13] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team: beta-build-scap-deb failing due to Error: lintian not available. - https://phabricator.wikimedia.org/T302460 (10dancy) 05Open→03Resolved a:03dancy Running `puppet agent --test` brought the systems back to the desired state. [23:21:22] !log Update beta cluster scap to 4.3.1-1+0~20220223231645.183~1.gbp8ddb60 [23:21:24] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [23:31:26] (03CR) 10Ahmon Dancy: docker: don't use docker.io package (032 comments) [tools/train-dev] - 10https://gerrit.wikimedia.org/r/765350 (owner: 10Jeena Huneidi) [23:43:16] (03PS1) 10Ahmon Dancy: Release 4.4.0-1 [tools/scap] - 10https://gerrit.wikimedia.org/r/765353 [23:43:18] (03CR) 10Ahmon Dancy: [C: 03+2] Release 4.4.0-1 [tools/scap] - 10https://gerrit.wikimedia.org/r/765353 (owner: 10Ahmon Dancy) [23:45:24] (03Merged) 10jenkins-bot: Release 4.4.0-1 [tools/scap] - 10https://gerrit.wikimedia.org/r/765353 (owner: 10Ahmon Dancy) [23:46:59] 10Release-Engineering-Team, 10Scap, 10serviceops: Deploy Scap version 4.4.0 - https://phabricator.wikimedia.org/T302464 (10dancy) [23:47:32] 10Release-Engineering-Team (Done by Feb 23🔥), 10MW-on-K8s, 10serviceops, 10Patch-For-Review: Build MediaWiki images for kubernetes on the deployment servers - https://phabricator.wikimedia.org/T297673 (10dancy) [23:47:38] 10Release-Engineering-Team, 10Scap, 10serviceops: Deploy Scap version 4.4.0 - https://phabricator.wikimedia.org/T302464 (10dancy)