[06:33:02] 10GitLab (Integrations), 10Phabricator (phabricator-next), 10Release-Engineering-Team (Yak Shaving πŸƒπŸͺ’), 10Wikimedia-Phabricator-Extensions: Phabricator gitlab panel is hard to read on narrow screens - https://phabricator.wikimedia.org/T362047#9716363 (10Aklapper) [06:38:46] 10Phabricator (phabricator-next), 10Release-Engineering-Team (Priority Backlog πŸ“₯), 10Wikimedia-Phabricator-Extensions: Improve error handling in "Escalate security issue" code - https://phabricator.wikimedia.org/T337654#9716374 (10Aklapper) [06:43:25] 10GitLab, 10Release-Engineering-Team (Radar), 06collaboration-services: GitLab archival by Software Heritage - https://phabricator.wikimedia.org/T362258#9716397 (10LSobanski) 05Openβ†’03Resolved a:03LSobanski Tyler responded via e-mail so I think we're OK to close this one. [07:38:02] 10Diffusion, 10Gerrit, 10Phabricator: Diffusion fails to push into Gerrit - https://phabricator.wikimedia.org/T362524#9716572 (10hashar) `refs/meta/config` holds the Gerrit configuration for the repository, it indeed can not be deleted. The ORES repositories being synced from GitHub to Gerrit using Phabri... [07:39:47] andre: ORES deployment model has been broken since its inception and Aaron Halfaker who was not willing to use Gerrit (nor use the Wikimedia org on Github) [07:40:15] while in production we had a clear limit: if something is deployed to production using git, it MUST be on Gerrit [07:40:33] so the alternative found is that Aaron tricked Mukunda to get Phabricator to fetch from github and push to gerrit [07:40:56] working around and bypassing the aforementioned limitation [07:41:15] (add in issues with syncing LFS thing) [07:41:18] something like that [07:41:45] I'd put the plug on it and say "sorry that never got supported, that is now causing troubles and we are not going to fix it" [07:43:22] hashar: +1. I think thcipriani wanted to talk with the ML folks about this anyway in context of T213246 [07:46:10] yeah [07:46:12] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Radar), 07Zuul, 06collaboration-services, 13Patch-For-Review: Test zuul under Bullseye with contint1003 - https://phabricator.wikimedia.org/T361224#9716592 (10hashar) 05Openβ†’03Resolved a:03hashar I have done the basic testing... [07:46:26] and refs/meta/config is Gerrit specific so you can't really delete it :) [07:46:33] (unless you are an Admin I guess) [07:46:58] I just want my logs to be less noisy but I keep running into issues :D [07:54:14] hehe [07:54:39] andre: that issue about refs/meta/config not being deletable/pushable probably always happened [07:54:56] eh eh [08:06:22] 10Release-Engineering-Team (Radar), 06collaboration-services: upgrade contint servers to bullseye - https://phabricator.wikimedia.org/T334517#9716627 (10hashar) [09:10:14] (03CR) 10Majavah: [C:03+2] zuul: Add DreamRimmer to the CI allowlist [integration/config] - 10https://gerrit.wikimedia.org/r/1019850 (owner: 10Dreamrimmer) [09:12:03] (03Merged) 10jenkins-bot: zuul: Add DreamRimmer to the CI allowlist [integration/config] - 10https://gerrit.wikimedia.org/r/1019850 (owner: 10Dreamrimmer) [09:12:36] !log reloading zuul for 1019850 [09:12:38] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [09:53:44] 06Release-Engineering-Team, 10Scap, 10MW-on-K8s, 06serviceops: Find a way to stage updated PHP packages on wikikube - https://phabricator.wikimedia.org/T362628 (10Clement_Goubert) 03NEW [10:01:06] 10Diffusion, 10Gerrit, 10Phabricator: Diffusion fails to push into Gerrit - https://phabricator.wikimedia.org/T362524#9717170 (10Aklapper) > Which repository(ies) has(ve) the issue? According to `aklapper@phab1004:~$ grep -r "RSA host key for IP address '2620:0:861:2:208:80:154:151' not in list of known hos... [10:14:19] 10Diffusion, 10Gerrit, 10Phabricator: Diffusion fails to push into Gerrit - https://phabricator.wikimedia.org/T362524#9717210 (10Aklapper) ...which means K19 fails. [10:16:20] 10Phabricator, 10wikimedia.biterg.io: Closed tickets in Bugzilla migrated without a closing date - https://phabricator.wikimedia.org/T107254#9717213 (10hashar) >>! In T107254#9710082, @hashar wrote: > As for {nav CLOSED}, I guess we did that to prevent further comments and/or to indicate the task resolution go... [10:24:31] (03PS1) 10Kevin Bazira: inference-services: add CI pipeline jobs for logo-detection model-server [integration/config] - 10https://gerrit.wikimedia.org/r/1019774 (https://phabricator.wikimedia.org/T362598) [10:32:16] (03CR) 10Lucas Werkmeister (WMDE): "It’s not my favorite option but I think we could live with it (I listed it in T362412#9710114)." [integration/config] - 10https://gerrit.wikimedia.org/r/1019298 (https://phabricator.wikimedia.org/T362412) (owner: 10Jforrester) [10:53:28] \o Morning [10:53:57] I have this change: https://gerrit.wikimedia.org/r/c/machinelearning/liftwing/inference-services/+/1020184 that I +2'd to make Jenkins do gate & submit, but nothing is happening. Did I miss some step? [11:00:30] 10GitLab-Application-Security-Pipeline, 06Security-Team: Craft more mediawiki-specific and php semgrep rule sets - https://phabricator.wikimedia.org/T342468#9717370 (10Aklapper) @sbassett: Hi, the `Due Date` set for this open task passed a while ago. Could you please either update or reset the Due Date (by cli... [11:01:59] Ah, I think this is because it triggers none of the actual pipelines (it's just a .gitignore change). So I V+2 and submitted it manually [11:09:54] (03CR) 10Hashar: [C:03+2] "INFO:jenkins_jobs.builder:Number of jobs generated: 4" [integration/config] - 10https://gerrit.wikimedia.org/r/1019774 (https://phabricator.wikimedia.org/T362598) (owner: 10Kevin Bazira) [11:10:44] klausman: yup ;) well found! [11:11:21] I had a CI at a previous job that would only run tests for code that had changed. Naturally, that missed some stuff _all the time_ :) [11:11:52] to workaround no jobs triggering we could add a `noop` job [11:12:09] that is a special job in Zuul which does not exist in Jenkins [11:12:19] it is "triggered" internally by Zuul and always succeed [11:12:34] Mh, might that not mask the case where there _should_ be a pipeline but it wasn't defined yet/correctly? [11:12:42] which fulfill the condition of one job being triggered and kicks in the Verified vote / Submit [11:13:03] also [11:13:15] we can have a small job that runs shellcheck from the root of the repo :) [11:13:32] hmm, not a bad idea [11:13:33] or run tox [11:13:34] or whatever [11:13:36] (03Merged) 10jenkins-bot: inference-services: add CI pipeline jobs for logo-detection model-server [integration/config] - 10https://gerrit.wikimedia.org/r/1019774 (https://phabricator.wikimedia.org/T362598) (owner: 10Kevin Bazira) [11:13:52] I'll float that idea within the team, thanks for the tip [11:14:07] you can add a tox job which is always triggered by CI [11:14:20] and then from the tox.ini add a virtualenv which runs shellcheck [11:15:05] !log Reloaded Zuul for https://gerrit.wikimedia.org/r/1019774 # T362598 [11:15:11] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [11:15:13] T362598: Prepare docker image for hosting the logo-detection model-server on LiftWing - https://phabricator.wikimedia.org/T362598 [11:16:45] klausman: an example is integration/config tox.ini. We run the commit-message-validator there and shellcheck as well https://gerrit.wikimedia.org/g/integration/config/+/refs/heads/master/tox.ini#33 [11:17:06] with some magic oneliner `/bin/bash -c 'shellcheck --color=always $(git ls-files "*.bash" "*.sh")'` [11:18:40] :+100: [11:32:33] !log Released Collapsible Console Section Jenkins plugin 1.9.0 # T362048 [11:32:35] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [11:32:36] T362048: Jenkins collapsible console section plugin no more shows table of content - https://phabricator.wikimedia.org/T362048 [13:04:20] 10Diffusion, 10Phabricator (phabricator-next): Enable git repo creation in Diffusion again - https://phabricator.wikimedia.org/T355644#9717883 (10Aklapper) [13:12:55] (03PS1) 10Jforrester: Zuul [mediawiki/extensions/Wikibase] Disable custom jobs on REL1_42 temporarily [integration/config] - 10https://gerrit.wikimedia.org/r/1020212 (https://phabricator.wikimedia.org/T362412) [13:14:45] (03CR) 10CI reject: [V:04-1] Zuul [mediawiki/extensions/Wikibase] Disable custom jobs on REL1_42 temporarily [integration/config] - 10https://gerrit.wikimedia.org/r/1020212 (https://phabricator.wikimedia.org/T362412) (owner: 10Jforrester) [13:15:41] (03PS2) 10Jforrester: Zuul [mediawiki/extensions/Wikibase] Disable custom jobs on REL1_42 temporarily [integration/config] - 10https://gerrit.wikimedia.org/r/1020212 (https://phabricator.wikimedia.org/T362412) [13:17:57] (03CR) 10Jforrester: [C:03+2] Zuul [mediawiki/extensions/Wikibase] Disable custom jobs on REL1_42 temporarily [integration/config] - 10https://gerrit.wikimedia.org/r/1020212 (https://phabricator.wikimedia.org/T362412) (owner: 10Jforrester) [13:20:10] (03Merged) 10jenkins-bot: Zuul [mediawiki/extensions/Wikibase] Disable custom jobs on REL1_42 temporarily [integration/config] - 10https://gerrit.wikimedia.org/r/1020212 (https://phabricator.wikimedia.org/T362412) (owner: 10Jforrester) [13:21:35] !log Zuul: [mediawiki/extensions/Wikibase] Disable custom jobs on REL1_42 temporarily for T362412 [13:21:38] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:21:39] T362412: Wikibase CI jobs wikibase-repo and wikibase-client broken on REL1_42 - https://phabricator.wikimedia.org/T362412 [13:29:07] (03PS4) 10Jforrester: [WIP] jjb: Specify PHP flavour in Wikibase special quibble jobs, add 81/83 [integration/config] - 10https://gerrit.wikimedia.org/r/1019298 (https://phabricator.wikimedia.org/T362412) [13:29:07] (03PS1) 10Jforrester: Zuul: [mediawiki/extensions/Wikibase] Run REL1_42 on PHP 8.1 jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1020216 (https://phabricator.wikimedia.org/T362412) [13:29:09] (03PS1) 10Jforrester: Zuul: [mediawiki/extensions/Wikibase] Run PHP 8.1 jobs by default [integration/config] - 10https://gerrit.wikimedia.org/r/1020217 [13:30:51] (03CR) 10CI reject: [V:04-1] Zuul: [mediawiki/extensions/Wikibase] Run REL1_42 on PHP 8.1 jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1020216 (https://phabricator.wikimedia.org/T362412) (owner: 10Jforrester) [13:31:29] (03CR) 10CI reject: [V:04-1] [WIP] jjb: Specify PHP flavour in Wikibase special quibble jobs, add 81/83 [integration/config] - 10https://gerrit.wikimedia.org/r/1019298 (https://phabricator.wikimedia.org/T362412) (owner: 10Jforrester) [13:31:30] (03CR) 10CI reject: [V:04-1] Zuul: [mediawiki/extensions/Wikibase] Run PHP 8.1 jobs by default [integration/config] - 10https://gerrit.wikimedia.org/r/1020217 (owner: 10Jforrester) [13:54:22] 10Release-Engineering-Team (Seen), 10MW-on-K8s, 06serviceops, 06SRE, 06Traffic: Rename X-Wikimedia-Debug k8s-experimental option - https://phabricator.wikimedia.org/T362662 (10Clement_Goubert) 03NEW [13:54:30] 10Release-Engineering-Team (Seen), 10MW-on-K8s, 06serviceops, 06SRE, 06Traffic: Rename X-Wikimedia-Debug k8s-experimental option - https://phabricator.wikimedia.org/T362662#9718155 (10Clement_Goubert) p:05Triageβ†’03Low [13:55:42] 10Continuous-Integration-Config, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10wmde-wikidata-tech, and 2 others: Make PHP 8.2 voting on development branch of Wikibase - https://phabricator.wikimedia.org/T360560#9718142 (10Lucas_Werkmeister_WMDE) There are a bunch of additional failures in [this... [14:00:26] 06Release-Engineering-Team, 06Infrastructure-Foundations, 06serviceops, 13Patch-For-Review: Deprecate buster-backports - https://phabricator.wikimedia.org/T362518#9718199 (10Clement_Goubert) The following images fail `docker-reporter` checks because they haven't been rebuilt on top of the new buster base i... [14:00:28] (03PS1) 10Lucas Werkmeister (WMDE): Zuul: [mediawiki/extensions/Wikibase] Remove stray experimental job [integration/config] - 10https://gerrit.wikimedia.org/r/1020223 [14:00:54] (03CR) 10Lucas Werkmeister (WMDE): "Small thing I noticed in [another change](https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Wikibase/+/1017816) when trying out PHP 8." [integration/config] - 10https://gerrit.wikimedia.org/r/1020223 (owner: 10Lucas Werkmeister (WMDE)) [14:18:55] 10Phabricator, 10Release-Engineering-Team (Radar), 03Wikimedia-Hackathon-2024: Phorge (Phabricator) Code Review Sprint - https://phabricator.wikimedia.org/T356384#9718318 (10brennen) [14:25:28] 10Phabricator (phabricator-next), 10Release-Engineering-Team (Now this 🫠), 06collaboration-services: Deploy Phabricator/Phorge 2024-04-16 - https://phabricator.wikimedia.org/T362666 (10brennen) 03NEW [14:30:19] 10Release-Engineering-Team (Radar), 06collaboration-services: upgrade contint servers to bullseye - https://phabricator.wikimedia.org/T334517#9718401 (10hashar) [14:31:33] 10GitLab-Application-Security-Pipeline, 06Security-Team: Craft more mediawiki-specific and php semgrep rule sets - https://phabricator.wikimedia.org/T342468#9718403 (10sbassett) [14:34:23] 10Phabricator (phabricator-next), 10Release-Engineering-Team (Now this 🫠), 06collaboration-services, 13Patch-For-Review: Deploy Phabricator/Phorge 2024-04-16 - https://phabricator.wikimedia.org/T362666#9718413 (10CodeReviewBot) brennen opened https://gitlab.wikimedia.org/repos/phabricator/deployment/-/merg... [14:39:06] (03CR) 10Jforrester: [C:03+2] "Oh, yes, good spot." [integration/config] - 10https://gerrit.wikimedia.org/r/1020223 (owner: 10Lucas Werkmeister (WMDE)) [14:41:12] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/Wikibase] Remove stray experimental job [integration/config] - 10https://gerrit.wikimedia.org/r/1020223 (owner: 10Lucas Werkmeister (WMDE)) [14:41:50] !log Zuul: [mediawiki/extensions/Wikibase] Drop quibble-vendor-mysql-php74-noselenium experimental job, already mainstream [14:41:52] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:01:47] andre: in https://meet.google.com/ccp-ijum-ksi [15:03:20] 10Phabricator (phabricator-next), 10Release-Engineering-Team (Now this 🫠), 06collaboration-services, 13Patch-For-Review: Deploy Phabricator/Phorge 2024-04-16 - https://phabricator.wikimedia.org/T362666#9718631 (10CodeReviewBot) brennen merged https://gitlab.wikimedia.org/repos/phabricator/deployment/-/merg... [15:03:59] 10Phabricator (phabricator-next), 10Release-Engineering-Team (Now this 🫠), 06collaboration-services, 13Patch-For-Review: Deploy Phabricator/Phorge 2024-04-16 - https://phabricator.wikimedia.org/T362666#9718634 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=6f68a9a5-b313-49f5-a348-5e307... [15:07:48] 10Release-Engineering-Team (Yak Shaving πŸƒπŸͺ’), 06collaboration-services, 06Security-Team, 07SecTeam-Processed, 07Security: Address Gerrit WMCS instance authenticating against LDAP (breaching WMCS policy) - https://phabricator.wikimedia.org/T330312#9718664 (10hashar) 05Openβ†’03Declined The instance g... [15:08:24] 10GitLab (Pipeline Services Migration🐀), 06collaboration-services: Move micro sites from Ganeti to Kubernetes and from Gerrit to GitLab - https://phabricator.wikimedia.org/T300171#9718668 (10Jelto) [15:25:38] mutante: so gerrit-prod-1001.devtools.eqiad1.wikimedia.cloud can be deleted and I am going to do it right now [15:26:00] gerrit-bullseye-test.devtools.eqiad1.wikimedia.cloud was created by you last year, I guess that was to verify puppet/scap/Gerrit worked when migrating the hosts from buster to bullseye [15:26:12] and that instance has served its purpose and can thus be deleted as well [15:26:33] 10Release-Engineering-Team (Priority Backlog πŸ“₯), 10Scap, 06Data-Platform-SRE, 10Wikidata, and 2 others: wdqs: replace git-fat with git-lfs - https://phabricator.wikimedia.org/T316876#9718810 (10bking) @dancy I can confirm that this is still on our radar. @Gehel is working on the archiva to gitlab migratio... [15:30:02] !log devtools: deleted obsolete Buster instances `gerrit-bullseye-test` and `gerrit-prod-1001` # T360964 [15:30:04] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:30:05] T360964: replace buster machines in devtools project - https://phabricator.wikimedia.org/T360964 [15:35:12] James_F: have we started migrating production to php 8.1? [15:35:33] hashar: Not seriously, it's pending the mw-on-k8s work. [15:35:36] But soon. [15:35:41] I have noticed we run wmf-quibble jobs with both 7.4 and 8.1 and my guess it is not needed [15:35:49] given the repositories already have a php 8.1 job anyway [15:36:11] hashar: Yeah, 8.1 is faster, we could drop 7.4 from test and only run in gate? [15:36:47] the thing is a change to mediawiki/core seems to trigger 20 jobs [15:37:02] Yes, we support too many versions of PHP right now. [15:37:07] and four wmf-quibble ones when two would be sufficient (solely for php 7.4) [15:37:11] REL1_42 is PHP 8.1–8.3. [15:37:12] that would reduce the load on the hosts [15:37:42] But master is (temporarily) PHP 7.4 + PHP 8.0 + REL1_42. [15:38:04] if php 8.0 has been dropped from REL1_42 we can drop it from master? [15:38:07] No. [15:38:20] * hashar looks puzzled [15:38:22] Well, maybe. [15:38:36] 10Release-Engineering-Team (Priority Backlog πŸ“₯), 10Scap, 06Data-Platform-SRE, 10Wikidata, and 2 others: wdqs: replace git-fat with git-lfs - https://phabricator.wikimedia.org/T316876#9718914 (10dancy) @bking Thanks for the update. I can help with converting a repo from git-fat to git-lfs if desired. [15:38:39] :) [15:38:54] so I drop the wmf-quibble php 8.1 jobs, they are too heavy weight [15:39:01] and drop php 8.0 from master [15:39:29] I'd drop the postgre one as well, but that is a different story *grin* [15:40:03] Drop the wmf-quibble php 7.4 jobs, they're heavier? [15:40:25] prod runs 7.4! [15:41:00] Yes, it'll catch changes in the other 7.4 jobs. [15:42:53] that is the same for phan running under 7.4 and 8.1 that is not needed [15:43:08] we can run it under php 8.1 and indicate in the config the minimal php version to support [15:43:44] Yes, but 8.1 runs faster than 7.4. [15:43:57] So if you have two CI jobs, run the 8.1 one. [15:44:20] then phan is 2/3 minutes so it is not the biggest offender [15:44:28] but yeah it should only run once [15:44:47] the devil is finding out how to have it instructed that we want to support 7.4 sitll while Phan itself runs under 8.1 [15:45:19] That's all sorted? [15:45:34] I don't know :) [15:46:04] (03PS1) 10Jforrester: Docker: Upgrade node18 and node20 images to latest [integration/config] - 10https://gerrit.wikimedia.org/r/1020271 [15:46:09] hashar: ^ is more fun. [15:52:25] (03PS2) 10Jforrester: Docker: Upgrade node18 and node20 images to latest [integration/config] - 10https://gerrit.wikimedia.org/r/1020271 [15:53:39] eek [15:53:44] well I Am off gotta prepare dinner :) [15:53:45] Yup. [15:53:48] * James_F grins. [15:54:36] hashar: thank you :) [15:55:26] (03CR) 10Jforrester: [C:03+2] Docker: Upgrade node18 and node20 images to latest [integration/config] - 10https://gerrit.wikimedia.org/r/1020271 (owner: 10Jforrester) [15:56:11] 10Diffusion, 10Phabricator (2024-04-16): Enable git repo creation in Diffusion again - https://phabricator.wikimedia.org/T355644#9719062 (10Aklapper) 05Openβ†’03Resolved a:03Aklapper Deployed on 2024-04-16 [15:56:43] 10Phabricator (2024-04-16), 10Release-Engineering-Team (Priority Backlog πŸ“₯): Remove script to convert Phabricator parent project into subproject - https://phabricator.wikimedia.org/T342275#9719066 (10Aklapper) 05Openβ†’03Resolved Deployed on 2024-04-16 [15:56:44] 10Phabricator (2024-04-16), 10Release-Engineering-Team (Now this 🫠), 06collaboration-services: Deploy Phabricator/Phorge 2024-04-16 - https://phabricator.wikimedia.org/T362666#9719059 (10Aklapper) 05Openβ†’03Resolved a:03Aklapper Deployed on 2024-04-16 [15:57:08] 10Phabricator (2024-04-16), 10Release-Engineering-Team (Priority Backlog πŸ“₯), 10Wikimedia-Phabricator-Extensions: Improve error handling in "Escalate security issue" code - https://phabricator.wikimedia.org/T337654#9719073 (10Aklapper) 05Openβ†’03Resolved Deployed on 2024-04-16 [15:57:09] 10GitLab (Integrations), 10Phabricator (2024-04-16), 10Release-Engineering-Team (Yak Shaving πŸƒπŸͺ’), 10Wikimedia-Phabricator-Extensions: Phabricator gitlab panel is hard to read on narrow screens - https://phabricator.wikimedia.org/T362047#9719070 (10Aklapper) 05In progressβ†’03Resolved a:03Aklapper D... [15:57:11] 10Phabricator (2024-04-16), 10Release-Engineering-Team (Now this 🫠), 06collaboration-services: Deploy Phabricator/Phorge 2024-04-16 - https://phabricator.wikimedia.org/T362666#9719075 (10Aklapper) a:05Aklapperβ†’03brennen [15:57:21] (03Merged) 10jenkins-bot: Docker: Upgrade node18 and node20 images to latest [integration/config] - 10https://gerrit.wikimedia.org/r/1020271 (owner: 10Jforrester) [15:57:23] brennen, andre: Any idea what's up with https://phabricator.wikimedia.org/T361395 ? [15:57:43] dancy, https://phabricator.wikimedia.org/T284397 [15:57:55] !log Docker: Upgrade node18 and node20 images to latest [15:57:57] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:58:14] dancy: could you file a new ticket, as that's still an issue, it seems? Thanks a lot! [15:58:20] Will do. [15:59:07] urgh, there are quite a few tasks it seems. I'll need to check this rather sooner [15:59:23] 10Phabricator (2024-04-16), 10Release-Engineering-Team (Now this 🫠), 06collaboration-services: Deploy Phabricator/Phorge 2024-04-16 - https://phabricator.wikimedia.org/T362666#9719079 (10cscott) This seems to have broken the train-blocker task for this week, T361395, which gives a RuntimeException when y... [16:00:15] andre, dancy - hrm, think this is new breakage from today's deploy? [16:00:17] brennen: Argh, we have an issue [16:00:43] https://phabricator.wikimedia.org/T362689 filed [16:01:04] Yes, it's a regression created by https://phabricator.wikimedia.org/T304540 [16:01:17] 10Phabricator: "Undefined offset: 5" error when visiting https://phabricator.wikimedia.org/T361395 - https://phabricator.wikimedia.org/T362689 (10dancy) 03NEW [16:01:23] kk, let's prep a revert of that. [16:01:27] 10Phabricator: "Undefined offset: 5" error when visiting https://phabricator.wikimedia.org/T361395 - https://phabricator.wikimedia.org/T362689#9719131 (10dancy) [16:01:55] brennen: I have another meeting right now :(( [16:02:04] 10Phabricator: "Undefined offset: 5" error when visiting https://phabricator.wikimedia.org/T361395 - https://phabricator.wikimedia.org/T362689#9719143 (10cscott) [16:02:09] 10Release-Engineering-Team (Priority Backlog πŸ“₯), 13Patch-For-Review, 05Release, 05Train Deployments: 1.43.0-wmf.1 deployment blockers - https://phabricator.wikimedia.org/T361395#9719144 (10cscott) [16:02:31] andre: no worries, i'll get a revert for that slung out. [16:02:34] thanks [16:02:35] 10Phabricator: "Undefined offset: 5" error when visiting https://phabricator.wikimedia.org/T361395 - https://phabricator.wikimedia.org/T362689#9719149 (10Aklapper) This is due to T304540 in https://gitlab.wikimedia.org/repos/phabricator/extensions/-/merge_requests/25 according to the logs [16:02:39] 10Phabricator: "Undefined offset: 5" error when visiting https://phabricator.wikimedia.org/T361395 - https://phabricator.wikimedia.org/T362689#9719163 (10Aklapper) p:05Triageβ†’03Unbreak! [16:02:50] (03PS1) 10Jforrester: jjb: Update all node18- and node20-based jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1020279 [16:03:12] 10Phabricator: "Undefined offset: 5" error when visiting https://phabricator.wikimedia.org/T361395 - https://phabricator.wikimedia.org/T362689#9719166 (10brennen) a:03brennen Prepping a revert. [16:06:09] 10Release-Engineering-Team (Priority Backlog πŸ“₯), 13Patch-For-Review, 05Release, 05Train Deployments: 1.43.0-wmf.1 deployment blockers - https://phabricator.wikimedia.org/T361395#9719187 (10cscott) [16:06:47] (03CR) 10Jforrester: "check experimental" [performance/fresnel] - 10https://gerrit.wikimedia.org/r/967518 (owner: 10Hashar) [16:07:11] 10Phabricator, 13Patch-For-Review: "Undefined offset: 5" error when visiting https://phabricator.wikimedia.org/T361395 - https://phabricator.wikimedia.org/T362689#9719182 (10CodeReviewBot) brennen opened https://gitlab.wikimedia.org/repos/phabricator/deployment/-/merge_requests/38 update extensions for revert... [16:08:01] 10Phabricator, 13Patch-For-Review: "Undefined offset: 5" error when visiting https://phabricator.wikimedia.org/T361395 - https://phabricator.wikimedia.org/T362689#9719197 (10CodeReviewBot) brennen merged https://gitlab.wikimedia.org/repos/phabricator/deployment/-/merge_requests/38 update extensions for revert... [16:12:04] 10Phabricator, 10Release-Engineering-Team (Priority Backlog πŸ“₯), 10Wikimedia-Phabricator-Extensions: Train blockers task throws exceptions on strange version numbers - https://phabricator.wikimedia.org/T304540#9719206 (10Aklapper) 05Resolvedβ†’03Open Reopening due to revert in T362689 [16:14:16] dancy, andre: fixed [16:14:18] brennen, thank you!! [16:14:27] Thanks! [16:14:52] 10Phabricator, 10Release-Engineering-Team (Priority Backlog πŸ“₯), 10Wikimedia-Phabricator-Extensions: Train blockers task throws exceptions on strange version numbers - https://phabricator.wikimedia.org/T304540#9719239 (10Aklapper) a:05brennenβ†’03None [16:15:31] sorry 'bout that. i had a moment of "hmm, is this gonna blow up" while looking that patch over yesterday. should have listened to my gut. [16:16:31] 06Release-Engineering-Team: Run a query to pull the number of Phabricator Tasks reported by community, resolved by WMF - https://phabricator.wikimedia.org/T362406#9719259 (10Aklapper) 05Openβ†’03Resolved [16:17:12] brennen: Sounds like it could use a little test suite [16:17:14] Argh, buster-backports. [16:17:17] Damn damn damn. [16:18:00] 10Continuous-Integration-Config, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10wmde-wikidata-tech, and 4 others: Make PHP 8.2 voting on development branch of Wikibase - https://phabricator.wikimedia.org/T360560#9719242 (10Lucas_Werkmeister_WMDE) It looks like the above three changes are enough t... [16:18:03] T362518 strikes. :-( [16:18:04] T362518: Deprecate buster-backports - https://phabricator.wikimedia.org/T362518 [16:18:34] ouch :( [16:18:39] 06Release-Engineering-Team, 06Infrastructure-Foundations, 06serviceops, 13Patch-For-Review: Deprecate buster-backports - https://phabricator.wikimedia.org/T362518#9719270 (10Jdforrester-WMF) This has also broken building CI images. Will have to migrate them to bullseye immediately, I suppose. [16:19:17] (03PS2) 10Jforrester: jjb: Update all node18- and node20-based jobs except Quibble ones [integration/config] - 10https://gerrit.wikimedia.org/r/1020279 [16:19:26] 10Phabricator, 06Release-Engineering-Team, 13Patch-For-Review: "Undefined offset: 5" error when visiting https://phabricator.wikimedia.org/T361395 - https://phabricator.wikimedia.org/T362689#9719265 (10brennen) 05Openβ†’03Resolved [16:19:39] dancy: no doubt [16:20:03] James_F: :( [16:20:09] brennen: Such is life. [16:20:13] I'll un-abandon my patch. [16:20:16] (03Restored) 10Jforrester: Docker: Provide bullseye-based Quibble images [integration/config] - 10https://gerrit.wikimedia.org/r/914316 (https://phabricator.wikimedia.org/T335766) (owner: 10Jforrester) [16:20:46] (03CR) 10Jforrester: "Doing this is now UBN due to buster-backports no longer existing. Will re-create." [integration/config] - 10https://gerrit.wikimedia.org/r/914316 (https://phabricator.wikimedia.org/T335766) (owner: 10Jforrester) [16:22:30] ugh. -backports being removed from apt is the thing that usually makes a Toolforge container image move from deprecated into abandoned mode. And it always sneaks up on me. [16:22:51] Yeah, me too. :-( [16:23:08] Is this why the cool kids switch tech jobs every 18 months? To hide from their pasts? [16:23:16] Indeed. [16:23:37] All bugs are just legacy tech debt if you migrate often enough. [16:24:56] (03PS3) 10Jforrester: jjb: Update all node18- and node20-based jobs except Quibble ones [integration/config] - 10https://gerrit.wikimedia.org/r/1020279 [16:33:02] (03CR) 10Jforrester: [C:03+2] "Deployed; seems fine." [integration/config] - 10https://gerrit.wikimedia.org/r/1020279 (owner: 10Jforrester) [16:35:15] (03Merged) 10jenkins-bot: jjb: Update all node18- and node20-based jobs except Quibble ones [integration/config] - 10https://gerrit.wikimedia.org/r/1020279 (owner: 10Jforrester) [16:47:46] 10Diffusion, 10Phabricator (Upstream), 07Upstream: Expose only canonical Clone URI in Diffusion when repo hosted externally and Differential disabled - https://phabricator.wikimedia.org/T361997#9719513 (10Aklapper) [16:47:50] 10Diffusion, 10Phabricator, 10Release-Engineering-Team (Seen), 13Patch-For-Review: Do not expose dead git-ssh.wikimedia.org service as repo Clone URLs (defined in diffusion.ssh-host setting) - https://phabricator.wikimedia.org/T347408#9719512 (10Aklapper) [16:48:21] 10Diffusion, 10Phabricator, 10Release-Engineering-Team (Seen), 13Patch-For-Review: Do not expose dead git-ssh.wikimedia.org service as repo Clone URLs (defined in diffusion.ssh-host setting) - https://phabricator.wikimedia.org/T347408#9719514 (10Aklapper) a:05Aklapperβ†’03None [16:49:09] 10Diffusion, 10Phabricator, 10Release-Engineering-Team (Seen), 13Patch-For-Review: Do not expose dead git-ssh.wikimedia.org service as repo Clone URLs (defined in diffusion.ssh-host setting) - https://phabricator.wikimedia.org/T347408#9719506 (10Aklapper) 05Openβ†’03Stalled I'd like to stall this on the... [16:51:44] 10Diffusion, 10Phabricator, 10Release-Engineering-Team (Priority Backlog πŸ“₯): Set security.require-https to true - https://phabricator.wikimedia.org/T347476#9719538 (10Aklapper) 05Openβ†’03Stalled I'd like to stall this on the outcome of T361997 in upstream which might be a way better approach. [16:51:48] 10Diffusion, 10Phabricator (Upstream), 07Upstream: Expose only canonical Clone URI in Diffusion when repo hosted externally and Differential disabled - https://phabricator.wikimedia.org/T361997#9719548 (10Aklapper) [16:51:49] 10Diffusion, 10Phabricator, 10Release-Engineering-Team (Priority Backlog πŸ“₯): Set security.require-https to true - https://phabricator.wikimedia.org/T347476#9719547 (10Aklapper) [16:55:05] 10Phabricator, 13Patch-For-Review, 07Technical-Debt: Remove unneeded custom changes in webroot/rsrc/css/phui/phui-property-list-view.css - https://phabricator.wikimedia.org/T352667#9719573 (10Aklapper) [16:56:33] 10Release-Engineering-Team (Now this 🫠), 06cloud-services-team, 06collaboration-services, 10VPS-project-devtools, and 2 others: Update devtools project puppetmaster - https://phabricator.wikimedia.org/T360470#9719580 (10Dzahn) puppetmaster-1003 is down again :/ tried to soft reboot it... [17:14:52] 10Release-Engineering-Team (Now this 🫠), 10Scap, 10MediaWiki-libs-Mime, 10MediaWiki-Platform-Team (Radar), and 2 others: Get rid of the /srv/mediawiki/php symbolic link - https://phabricator.wikimedia.org/T359643#9719698 (10CodeReviewBot) thcipriani merged https://gitlab.wikimedia.org/repos/releng/scap/-/m... [17:27:04] 10Release-Engineering-Team (Radar), 06collaboration-services: upgrade contint servers to bullseye - https://phabricator.wikimedia.org/T334517#9719748 (10Dzahn) 05Openβ†’03In progress a:03Dzahn [17:31:29] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 13Patch-For-Review: Migrate Quibble images from buster to bullseye - https://phabricator.wikimedia.org/T335766#9719765 (10Jdforrester-WMF) p:05Triageβ†’03High a:03Jdforrester-WMF Now urgent. following T362518. [17:31:37] 10Release-Engineering-Team (Radar), 06collaboration-services: upgrade contint servers to bullseye - https://phabricator.wikimedia.org/T334517#9719774 (10Dzahn) We had another meeting about this today and we said to: - reimage contint1002 any time - enable rsyncing of data in /srv/jenkins from contint2002 to c... [17:39:36] 10Release-Engineering-Team (Radar), 06collaboration-services, 13Patch-For-Review: upgrade contint servers to bullseye - https://phabricator.wikimedia.org/T334517#9719796 (10Dzahn) [17:40:08] 10Release-Engineering-Team (Radar), 06collaboration-services, 13Patch-For-Review: upgrade contint servers to bullseye - https://phabricator.wikimedia.org/T334517#9719799 (10Dzahn) zuul-merger on contint1002 has been masked. ` Loaded: masked (Reason: Unit zuul-merger.service is masked.) ` [17:40:45] (03PS6) 10Jforrester: Docker: Provide bullseye-based Quibble images [integration/config] - 10https://gerrit.wikimedia.org/r/914316 (https://phabricator.wikimedia.org/T335766) [17:40:45] (03PS1) 10Jforrester: jjb: [gearman-java-maven-java11-site-publish] Comment out broken ref [integration/config] - 10https://gerrit.wikimedia.org/r/1020298 [17:40:45] (03PS1) 10Jforrester: jjb: Migrate all quibble-based jobs from buster to bullseye [integration/config] - 10https://gerrit.wikimedia.org/r/1020299 (https://phabricator.wikimedia.org/T335766) [17:41:47] anyone, if you go to https://integration.wikimedia.org/ci/computer/contint1002/ do you see some kind of "STOP" or "disable" button for the jenkins agent there? [17:42:40] mutante: There's 'disconnect', 'mark temporarily offline', and 'delete agent'. [17:42:51] Depending on which task you want to do… [17:43:22] James_F: the instructions tell me: " [17:43:22] Disable the Jenkins agent https://integration.wikimedia.org/ci/computer/contint1002/ [17:43:36] temp offline ? [17:43:39] Yeah. [17:43:43] it's for reimaging the server [17:43:46] Which instructions? I thought I updated all those. [17:43:55] (03CR) 10Jforrester: [C:03+2] jjb: [gearman-java-maven-java11-site-publish] Comment out broken ref [integration/config] - 10https://gerrit.wikimedia.org/r/1020298 (owner: 10Jforrester) [17:43:56] check boxes on https://phabricator.wikimedia.org/T334517 [17:44:09] Ah. [17:44:25] I don't have privileges for this it seems [17:45:20] I just masked the zuul-merger, but that was in puppet [17:45:30] (03Merged) 10jenkins-bot: jjb: [gearman-java-maven-java11-site-publish] Comment out broken ref [integration/config] - 10https://gerrit.wikimedia.org/r/1020298 (owner: 10Jforrester) [17:45:57] I've marked it as offline. [17:46:26] (03CR) 10Jforrester: [C:03+2] Docker: Provide bullseye-based Quibble images [integration/config] - 10https://gerrit.wikimedia.org/r/914316 (https://phabricator.wikimedia.org/T335766) (owner: 10Jforrester) [17:46:26] thank you, James. going to downtime and reimage soon [17:47:25] 10Release-Engineering-Team (Radar), 06collaboration-services, 13Patch-For-Review: upgrade contint servers to bullseye - https://phabricator.wikimedia.org/T334517#9719843 (10Dzahn) [17:47:42] 10Release-Engineering-Team (Radar), 06collaboration-services, 13Patch-For-Review: upgrade contint servers to bullseye - https://phabricator.wikimedia.org/T334517#9719846 (10Dzahn) James marked the jenkis as "temp offline" on contint1002. Downtiming and reimaging now. [17:48:01] (03Merged) 10jenkins-bot: Docker: Provide bullseye-based Quibble images [integration/config] - 10https://gerrit.wikimedia.org/r/914316 (https://phabricator.wikimedia.org/T335766) (owner: 10Jforrester) [17:49:42] 10Release-Engineering-Team (Radar), 06collaboration-services, 13Patch-For-Review: upgrade contint servers to bullseye - https://phabricator.wikimedia.org/T334517#9719861 (10Dzahn) [17:50:01] (03PS2) 10Jforrester: jjb: Migrate all quibble-based jobs from buster to bullseye [integration/config] - 10https://gerrit.wikimedia.org/r/1020299 (https://phabricator.wikimedia.org/T335766) [17:50:01] (03PS1) 10Jforrester: Docker: [quibble-buster*] Back out today's changes, won't ever build [integration/config] - 10https://gerrit.wikimedia.org/r/1020300 [17:50:12] (03CR) 10Jforrester: [C:03+2] Docker: [quibble-buster*] Back out today's changes, won't ever build [integration/config] - 10https://gerrit.wikimedia.org/r/1020300 (owner: 10Jforrester) [17:52:09] 10Release-Engineering-Team (Radar), 06collaboration-services, 13Patch-For-Review: upgrade contint servers to bullseye - https://phabricator.wikimedia.org/T334517#9719885 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by dzahn@cumin2002 for host contint1002.wikimedia.org with OS bullseye [17:53:08] (03Merged) 10jenkins-bot: Docker: [quibble-buster*] Back out today's changes, won't ever build [integration/config] - 10https://gerrit.wikimedia.org/r/1020300 (owner: 10Jforrester) [18:04:56] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure: Migrate Quibble images from bullseye to bookworm - https://phabricator.wikimedia.org/T362705 (10Jdforrester-WMF) 03NEW [18:05:45] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 13Patch-For-Review: Migrate Quibble images from buster to bullseye - https://phabricator.wikimedia.org/T335766#9719944 (10Jdforrester-WMF) [18:09:48] 10Continuous-Integration-Config, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10wmde-wikidata-tech, and 4 others: Make PHP 8.2 voting on development branch of Wikibase - https://phabricator.wikimedia.org/T360560#9719952 (10Jdforrester-WMF) >>! In T360560#9719240, @Lucas_Werkmeister_WMDE wrote: >... [18:10:13] OK, we're good to cook, hopefully. [18:14:42] 06Release-Engineering-Team: Run a query to pull the number of Phabricator Tasks reported by community, resolved by WMF - https://phabricator.wikimedia.org/T362406#9719963 (10ARamirez_WMF) @Aklapper Can I get the google sheet report like the ones you sent amanda the previous times please? I need to be craft a... [18:22:15] Hmm, looks like I might need to fix T347380 for selenium somehow. :-( T347380 [18:22:15] T347380: VisualEditor unit tests will not run on Chromium snap - https://phabricator.wikimedia.org/T347380 [18:26:31] 10Release-Engineering-Team (Radar), 06collaboration-services, 13Patch-For-Review: upgrade contint servers to bullseye - https://phabricator.wikimedia.org/T334517#9720017 (10Dzahn) ` Error: Could not enable docker: Error: /Stage[main]/Profile::Ci::Docker/Service[docker]/enable: change from 'false' to 'true'... [18:27:54] (03CR) 10Jforrester: [C:03+2] "Deployed. Likely follow-ups in-repo." [integration/config] - 10https://gerrit.wikimedia.org/r/1020299 (https://phabricator.wikimedia.org/T335766) (owner: 10Jforrester) [18:29:36] (03Merged) 10jenkins-bot: jjb: Migrate all quibble-based jobs from buster to bullseye [integration/config] - 10https://gerrit.wikimedia.org/r/1020299 (https://phabricator.wikimedia.org/T335766) (owner: 10Jforrester) [18:31:37] Hmm. [18:32:11] Meh. [18:32:21] (03PS1) 10Jforrester: Revert "jjb: Migrate all quibble-based jobs from buster to bullseye" [integration/config] - 10https://gerrit.wikimedia.org/r/1020233 [18:32:24] (03CR) 10Jforrester: [C:03+2] Revert "jjb: Migrate all quibble-based jobs from buster to bullseye" [integration/config] - 10https://gerrit.wikimedia.org/r/1020233 (owner: 10Jforrester) [18:34:01] (03Merged) 10jenkins-bot: Revert "jjb: Migrate all quibble-based jobs from buster to bullseye" [integration/config] - 10https://gerrit.wikimedia.org/r/1020233 (owner: 10Jforrester) [18:37:32] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 13Patch-For-Review: Migrate Quibble images from buster to bullseye - https://phabricator.wikimedia.org/T335766#9720053 (10Jdforrester-WMF) Current state: * Quibble-buster images work but can't be updated * Quibble-bullseye images don't... [18:38:58] 10Release-Engineering-Team (Radar), 06collaboration-services, 13Patch-For-Review: upgrade contint servers to bullseye - https://phabricator.wikimedia.org/T334517#9720056 (10Dzahn) [18:44:16] 10Release-Engineering-Team (Radar), 06collaboration-services, 13Patch-For-Review: upgrade contint servers to bullseye - https://phabricator.wikimedia.org/T334517#9720093 (10Dzahn) >>! In T334517#9720017, @Dzahn wrote: > Error: '/usr/sbin/a2enmod php7.4' returned 1 instead of one of [0] > Error: /Stage[main... [18:44:30] 10Release-Engineering-Team (Radar), 06collaboration-services, 13Patch-For-Review: upgrade contint servers to bullseye - https://phabricator.wikimedia.org/T334517#9720095 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by dzahn@cumin2002 for host contint1002.wikimedia.org with OS bullseye... [18:45:46] 10Release-Engineering-Team (Priority Backlog πŸ“₯), 10MediaWiki-Docker, 07Regression: php-xdebug no longer works in MediaWiki-Docker - https://phabricator.wikimedia.org/T349266#9720100 (10brennen) p:05Triageβ†’03Medium [18:49:04] 10Release-Engineering-Team (Radar), 06collaboration-services, 13Patch-For-Review: upgrade contint servers to bullseye - https://phabricator.wikimedia.org/T334517#9720104 (10Dzahn) [18:59:15] 10Release-Engineering-Team (Radar), 06collaboration-services, 13Patch-For-Review: upgrade contint servers to bullseye - https://phabricator.wikimedia.org/T334517#9720117 (10Dzahn) verifying that: - /srv is a standalone partition! - /dev/mapper/vg0-srv on /srv type ext4 (rw,relatime) βœ… - Docker daemon is st... [19:09:14] 10Release-Engineering-Team (Radar), 06collaboration-services, 13Patch-For-Review: upgrade contint servers to bullseye - https://phabricator.wikimedia.org/T334517#9720153 (10Dzahn) [19:38:30] 06Release-Engineering-Team: Run a query to pull the number of Phabricator Tasks reported by community, resolved by WMF - https://phabricator.wikimedia.org/T362406#9720318 (10Aklapper) 05Resolvedβ†’03Open [19:39:11] 06Release-Engineering-Team: Run a query to pull the number of Phabricator Tasks reported by community, resolved by WMF - https://phabricator.wikimedia.org/T362406#9720321 (10Aklapper) I just don't remember what we/I did months ago. :D I'll check my mail and try to remember [19:45:42] 06Release-Engineering-Team: Run a query to pull the number of Phabricator Tasks reported by community, resolved by WMF - https://phabricator.wikimedia.org/T362406#9720340 (10Dzahn) The list of hardcoded user names in there seems kind of arbitrary? Is this only supposed to be for tickets with a certain tag? [19:49:51] 06Release-Engineering-Team: Run a query to pull the number of Phabricator Tasks reported by community, resolved by WMF - https://phabricator.wikimedia.org/T362406#9720353 (10Aklapper) >>! In T362406#9720340, @Dzahn wrote: > The list of hardcoded user names in there seems kind of arbitrary? No, it's accounts of... [19:51:38] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Radar), 07Zuul, 06collaboration-services, 13Patch-For-Review: Test zuul under Bullseye with contint1003 - https://phabricator.wikimedia.org/T361224#9720354 (10Dzahn) [19:51:45] 10Continuous-Integration-Infrastructure, 06collaboration-services, 06SRE, 10vm-requests, 13Patch-For-Review: Ganeti VM for contint migration - https://phabricator.wikimedia.org/T358237#9720355 (10Dzahn) [19:58:19] 10Release-Engineering-Team (Radar), 06collaboration-services, 13Patch-For-Review: upgrade contint servers to bullseye - https://phabricator.wikimedia.org/T334517#9720369 (10Dzahn) [19:59:14] James_F: https://integration.wikimedia.org/ci/computer/contint1002/ could now be enabled again [20:03:06] mutante: clicking "Bring this node back online" [20:05:39] mutante: agent seems to fail on ssh host key [20:06:27] brennen: ah, right, that is expected. " the ssh host key would need to be verified again since the reimaging causes the host key to change." [20:08:05] https://phabricator.wikimedia.org/P60692 [20:09:05] that is until we have our ssh host keys centrally managed instead of generated on each hosts :D [20:12:00] mutante: k, launched successfully this time i think [20:14:42] brennen: thanks! [20:16:41] hashar: Now you're just dreaming. :-) [20:19:02] I was completely fine using telnet over ipsec [20:19:12] or well .. rlogin [20:19:38] ssh-keygen -l -E md5 -f ssh_host_ed25519_key.pub [20:19:39] 256 MD5:a0:94:37:eb:06:8a:f1:a4:3f:f6:86:ca:e1:e4:62:a1 root@contint1002 (ED25519) [20:19:47] confirmed [20:20:25] enabling zuul-merger now [20:20:57] 06Release-Engineering-Team, 10Scap, 06serviceops: scap should optionally display helmfile diffs for review - https://phabricator.wikimedia.org/T362717 (10Scott_French) 03NEW [20:22:11] Active: active (running) since Tue 2024-04-16 20:21:54 UTC; 12s ago [20:24:33] 10Release-Engineering-Team (Radar), 06collaboration-services, 13Patch-For-Review: upgrade contint servers to bullseye - https://phabricator.wikimedia.org/T334517#9720437 (10Dzahn) [20:26:31] https://integration.wikimedia.org/ci/computer/contint1002/load-statistics :) [20:27:32] not very busy in general [20:30:30] 10Release-Engineering-Team (Radar), 06collaboration-services, 13Patch-For-Review: upgrade contint servers to bullseye - https://phabricator.wikimedia.org/T334517#9720449 (10Dzahn) {F47036284} [20:32:41] mutante: that is going to fail evenutually due to T157818 [20:32:42] T157818: zuul-merger fails when repository names overlaps - https://phabricator.wikimedia.org/T157818 [20:35:04] oh, interesting [20:35:54] there is a oneliner to clone the repos [20:36:09] which surely should be puppetized for whenever we first create a zuul merger [20:36:20] then that hit us only every two years or so :) [20:36:43] the error would show up in the log under /var/log/zuul/ [20:37:12] and that is reflected in gErrit with a message from jenkins-bot stating the change failed to merge [20:40:03] anyway, I am sleeping for real now [20:42:46] (03PS1) 10Jforrester: Zuul: [performance/fresnel] Use generic template, enable node20 (passing) [integration/config] - 10https://gerrit.wikimedia.org/r/1020357 [20:42:46] (03PS1) 10Jforrester: Zuul: [unicodejs] Use generic template, enable node20 (passing) [integration/config] - 10https://gerrit.wikimedia.org/r/1020358 [20:42:46] (03PS1) 10Jforrester: Zuul: [VisualEditor/VisualEditor] Use generic template, enable node20 (passing) [integration/config] - 10https://gerrit.wikimedia.org/r/1020359 [20:42:46] (03PS1) 10Jforrester: jjb: Drop specfic node jobs we don't need [integration/config] - 10https://gerrit.wikimedia.org/r/1020360 [20:42:47] (03PS1) 10Jforrester: jjb: Drop two unused phpunit-coverage-publish flavours [integration/config] - 10https://gerrit.wikimedia.org/r/1020361 [20:42:49] (03PS1) 10Jforrester: Zuul: Migrate default quibble-…-selenium jobs from php74 to php81 [integration/config] - 10https://gerrit.wikimedia.org/r/1020362 [20:42:53] (03PS1) 10Jforrester: Zuul: Make default noselenium job run php81, not php74 [integration/config] - 10https://gerrit.wikimedia.org/r/1020363 [20:42:57] (03PS1) 10Jforrester: Zuul: Make unused in-mediawiki-tarball target php81 [integration/config] - 10https://gerrit.wikimedia.org/r/1020364 [20:44:49] (03CR) 10Jforrester: [C:03+2] Zuul: [performance/fresnel] Use generic template, enable node20 (passing) [integration/config] - 10https://gerrit.wikimedia.org/r/1020357 (owner: 10Jforrester) [20:44:57] (03CR) 10Jforrester: [C:03+2] Zuul: [unicodejs] Use generic template, enable node20 (passing) [integration/config] - 10https://gerrit.wikimedia.org/r/1020358 (owner: 10Jforrester) [20:45:01] (03CR) 10Jforrester: [C:03+2] Zuul: [VisualEditor/VisualEditor] Use generic template, enable node20 (passing) [integration/config] - 10https://gerrit.wikimedia.org/r/1020359 (owner: 10Jforrester) [20:47:36] (03Merged) 10jenkins-bot: Zuul: [performance/fresnel] Use generic template, enable node20 (passing) [integration/config] - 10https://gerrit.wikimedia.org/r/1020357 (owner: 10Jforrester) [20:47:39] (03Merged) 10jenkins-bot: Zuul: [unicodejs] Use generic template, enable node20 (passing) [integration/config] - 10https://gerrit.wikimedia.org/r/1020358 (owner: 10Jforrester) [20:47:41] (03Merged) 10jenkins-bot: Zuul: [VisualEditor/VisualEditor] Use generic template, enable node20 (passing) [integration/config] - 10https://gerrit.wikimedia.org/r/1020359 (owner: 10Jforrester) [20:48:01] !log Zuul: Switch performance/fresnel, unicodejs, and VisualEditor/VisualEditor to generic node jobs. [20:48:03] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:49:29] (03CR) 10Jforrester: [C:03+2] jjb: Drop specfic node jobs we don't need [integration/config] - 10https://gerrit.wikimedia.org/r/1020360 (owner: 10Jforrester) [20:50:05] (03CR) 10Jforrester: [C:03+2] jjb: Drop two unused phpunit-coverage-publish flavours [integration/config] - 10https://gerrit.wikimedia.org/r/1020361 (owner: 10Jforrester) [20:51:30] (03Merged) 10jenkins-bot: jjb: Drop specfic node jobs we don't need [integration/config] - 10https://gerrit.wikimedia.org/r/1020360 (owner: 10Jforrester) [20:51:36] (03Merged) 10jenkins-bot: jjb: Drop two unused phpunit-coverage-publish flavours [integration/config] - 10https://gerrit.wikimedia.org/r/1020361 (owner: 10Jforrester) [20:55:42] 06Release-Engineering-Team, 10Scap, 06serviceops: scap should optionally display helmfile diffs for review - https://phabricator.wikimedia.org/T362717#9720501 (10Scott_French) p:05Triageβ†’03Low I have a proof-of-concept patch for this, the result of which is shown in P60688 (using train-dev). This front-... [21:02:58] 06Release-Engineering-Team: Run a query to pull the number of Phabricator Tasks reported by community, resolved by WMF - https://phabricator.wikimedia.org/T362406#9720545 (10Dzahn) Oh, I understand now, it's the staff who is known to not use wikimedia.org mail, ack, apologies. [21:32:05] 06Project-Admins: Create project tag for New-Engagement-Experiments - https://phabricator.wikimedia.org/T362721 (10RHo) 03NEW [21:48:38] 06Project-Admins: Create project tag for New-Engagement-Experiments - https://phabricator.wikimedia.org/T362721#9720671 (10Aklapper) [21:56:13] 06Release-Engineering-Team, 10Scap, 06serviceops: scap should optionally display helmfile diffs for review - https://phabricator.wikimedia.org/T362717#9720685 (10RLazarus) If we're really worried about that race condition, is it plausible to do this? - Run `helmfile diff` early, right after getting the scap...