[02:14:05] 10GitLab: Change my 'full name' in GitLab - https://phabricator.wikimedia.org/T329057 (10xcollazo) Thanks for looking into this folks. I understand this is not possible right now, and I do use gerrit quite often, so I'll close this. But before I do that: my user was created quite recently (~8 months ago), while... [06:36:23] 10GitLab: Change my 'full name' in GitLab - https://phabricator.wikimedia.org/T329057 (10hashar) >>! In T329057#8595763, @bd808 wrote: > Funny enough the main thing that has historically blown up after changing the CN of a developer account is Gerrit Gerrit requires a manual operation to associate the user with... [06:49:17] 10GitLab: Change my 'full name' in GitLab - https://phabricator.wikimedia.org/T329057 (10hashar) >>! In T329057#8596354, @xcollazo wrote: > Thanks for looking into this folks. I understand this is not possible right now, and I do use gerrit quite often, so I'll close this. > > But before I do that: my user was... [06:53:48] (03PS1) 10Hashar: dockerfiles: php-compile, fix typo CLFAGS > CFLAGS [integration/config] - 10https://gerrit.wikimedia.org/r/887634 [06:55:40] (03PS1) 10Hashar: jjb: fix typo in php-compile jobs [integration/config] - 10https://gerrit.wikimedia.org/r/887635 [06:55:53] (03CR) 10Hashar: [C: 03+2] dockerfiles: php-compile, fix typo CLFAGS > CFLAGS [integration/config] - 10https://gerrit.wikimedia.org/r/887634 (owner: 10Hashar) [06:57:05] (03Merged) 10jenkins-bot: dockerfiles: php-compile, fix typo CLFAGS > CFLAGS [integration/config] - 10https://gerrit.wikimedia.org/r/887634 (owner: 10Hashar) [07:04:47] (03CR) 10Hashar: [C: 03+2] dockerfiles: php-compile fix ambiguous env assignment (033 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/764820 (owner: 10Hashar) [07:05:21] (03CR) 10Hashar: [C: 03+2] "Solved!" [integration/config] - 10https://gerrit.wikimedia.org/r/887635 (owner: 10Hashar) [07:06:27] (03Merged) 10jenkins-bot: jjb: fix typo in php-compile jobs [integration/config] - 10https://gerrit.wikimedia.org/r/887635 (owner: 10Hashar) [07:54:37] 10GitLab: Change my 'full name' in GitLab - https://phabricator.wikimedia.org/T329057 (10Peachey88) >>! In T329057#8596506, @hashar wrote: > But immediately after: > >> Create your account in Wikimedia Phabricator and link it to your work wiki account which was created by ITS. Turn on Two-Factor Authentication... [08:39:26] 10Phabricator Antivandalism Extension, 10DBA: Switchover m3 master db1164 -> db1159 - https://phabricator.wikimedia.org/T329141 (10Marostegui) [08:39:48] 10Phabricator Antivandalism Extension, 10DBA: Switchover m3 master db1164 -> db1159 - https://phabricator.wikimedia.org/T329141 (10Marostegui) [08:39:58] 10Phabricator Antivandalism Extension, 10DBA: Switchover m3 master db1164 -> db1159 - https://phabricator.wikimedia.org/T329141 (10Marostegui) p:05Triage→03Medium [08:40:21] 10Phabricator Antivandalism Extension, 10DBA: Switchover m3 master db1164 -> db1159 - https://phabricator.wikimedia.org/T329141 (10Marostegui) The reason for this: T329013#8596654 [08:41:39] 10Phabricator Antivandalism Extension, 10DBA: Switchover m3 master db1164 -> db1159 - https://phabricator.wikimedia.org/T329141 (10Marostegui) [08:46:04] 10Phabricator Antivandalism Extension, 10DBA, 10Patch-For-Review: Switchover m3 master db1164 -> db1159 - https://phabricator.wikimedia.org/T329141 (10Marostegui) [08:48:56] o/, could someone with the correct rights visit https://integration.wikimedia.org/ci/computer/deployment-deploy03/launchSlaveAgent and attempt to get the node (re)started? [08:49:16] ref T329056 [08:49:17] T329056: Agent deployment-deploy03 is offline - https://phabricator.wikimedia.org/T329056 [08:49:29] 10Phabricator Antivandalism Extension, 10DBA, 10Patch-For-Review: Switchover m3 master db1164 -> db1159 - https://phabricator.wikimedia.org/T329141 (10Marostegui) [08:50:38] * RhinosF1 looks in hashar’s direction [08:52:30] 10Release-Engineering-Team, 10Scap: scap.cfg hostname config sections are not applied - https://phabricator.wikimedia.org/T329144 (10hashar) [08:52:45] TheresNoTime: RhinosF1 I am checking [08:53:35] [02/08/23 08:51:38] [SSH] Opening SSH connection to 172.16.4.233:22. [08:53:35] connect timed out [08:53:38] fun times [08:53:38] 10Beta-Cluster-Infrastructure, 10Continuous-Integration-Infrastructure, 10Jenkins, 10ci-test-error: Agent deployment-deploy03 is offline - https://phabricator.wikimedia.org/T329056 (10hashar) [08:53:43] from https://integration.wikimedia.org/ci/computer/deployment-deploy03/log [08:54:07] woo [08:54:57] checking whether the instance is up [08:55:05] 10Phabricator Antivandalism Extension, 10DBA, 10Patch-For-Review: Switchover m3 master db1164 -> db1159 - https://phabricator.wikimedia.org/T329141 (10Marostegui) [08:55:17] deployment-deploy03 is up, I'm currently SSH'd in [08:55:49] and it has the same IP 172.16.4.233/21 [08:57:04] but there is no ssh connection possible from contint2001 [08:57:05] fun [08:57:31] yup.. now this happened, I think, around the time of yesterday's network issue (trying to find the task I'm on about) [08:59:14] TheresNoTime: you mean the switch upgrade? [08:59:37] RhinosF1: ah, yes, might have been - I'm sure it was around then it started [08:59:55] I am capturing the stuff on the task [09:00:01] 10Beta-Cluster-Infrastructure, 10Continuous-Integration-Infrastructure, 10Jenkins, 10ci-test-error: Agent deployment-deploy03 is offline - https://phabricator.wikimedia.org/T329056 (10hashar) We can ssh from the bastion to the `deployment-deploy03.deployment-prep.eqiad1.wikimedia.cloud` instance when passi... [09:00:20] TheresNoTime: timing is suspect [09:00:22] 10Phabricator Antivandalism Extension, 10DBA, 10Patch-For-Review: Switchover m3 master db1164 -> db1159 - https://phabricator.wikimedia.org/T329141 (10Marostegui) [09:00:41] I’ll ping some people [09:01:00] thank you for the attention RhinosF1 hashar :-) [09:03:04] topranks: see conversation in here ^ [09:03:07] 10Beta-Cluster-Infrastructure, 10Continuous-Integration-Infrastructure, 10Jenkins, 10ci-test-error: Agent deployment-deploy03 is offline - https://phabricator.wikimedia.org/T329056 (10hashar) That is the ferm/iptables configuration on the instance. It is rejecting traffic originating from contint2001.wikim... [09:03:14] the instance lacks an iptable rule to allow the traffic [09:03:52] hashar: thanks :) [09:05:18] so well hmm [09:05:27] (03CR) 10Jakob: [C: 03+1] "Thanks, LGTM!" [integration/config] - 10https://gerrit.wikimedia.org/r/887340 (owner: 10Jakob) [09:05:30] I don't remember what kind of mess I have might have setup at the time to allow the traffic in [09:05:35] it must be some ferm rules maintained by Puppet [09:05:46] at least it is not on the WMCS infra since the traffic does reach the instance [09:05:52] * hashar looks at puppet log [09:06:07] hash/ar to the rescue! [09:06:27] maybe it is DNS https://xkcd.com/2259/ [09:07:25] puppet has no change [09:07:59] It is always DNS™ [09:09:03] pff [09:13:42] so yeah hmm [09:13:42] 10Beta-Cluster-Infrastructure, 10Continuous-Integration-Infrastructure, 10Jenkins, 10ci-test-error: Agent deployment-deploy03 is offline - https://phabricator.wikimedia.org/T329056 (10hashar) `/etc/ferm/conf.d` last touched at Feb 3rd 19:54. The last file touched is `00_defs` on January 31st 16:23. But... [09:13:59] ferm got started on Feb 3rd at 19:54 (/Stage[main]/Ferm/Service[ferm]/ensure) ensure changed 'stopped' to 'running' (corrective) [09:14:07] presumably it was not active before that [09:14:17] and I imagine the ssh connection was already established [09:14:32] yesterday, the WMCS network went done which must have terminated the ssh connnection [09:14:45] * TheresNoTime *silent screaming* [09:14:50] hashar: ferm only starts when a change is applied [09:14:57] once the network went back, the rule not being there anymore since "who knows how long", the ssh connection can't work [09:14:59] ah [09:15:07] so the instance initially doesn't have any rule applied? [09:15:54] hashar: I guess now to find why the rule was removed [09:15:59] !log deployment-prep: deployment-deploy03 `systemctl stop ferm` to allow jenkins to connect to the instance # T329056 [09:16:01] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [09:16:02] T329056: Agent deployment-deploy03 is offline - https://phabricator.wikimedia.org/T329056 [09:16:04] And revert that [09:16:35] so it is restored temporarily [09:16:47] I stopped ferm, connected the agent, started ferm again [09:17:29] the thing is the ferm service is enabled, so I would expect it to come up when the instance boot up [09:17:46] oor maybe Jenkins manages to connect before the ferm service starts [09:18:16] hashar: stopping ferm seems like a bad idea if it will break firewall rules. Some of them are probably there for good reason. [09:18:38] I restarted it after the connection got established [09:18:46] Ok [09:18:50] TheresNoTime: the beta jenkins job is running [09:19:08] hashar: thank you :) [09:19:20] it is not fixed though :D [09:19:32] I mean, I gotta ensure we have the proper firewall rule in place [09:19:39] fixed enough for me /s [09:21:00] Project beta-update-databases-eqiad build #64999: 04FAILURE in 3 min 49 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/64999/ [09:21:01] Project beta-update-databases-eqiad build #65000: 04STILL FAILING in 0.79 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/65000/ [09:21:10] oups [09:21:40] Yippee, build fixed! [09:21:41] Project beta-code-update-eqiad build #429994: 09FIXED in 5 min 58 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/429994/ [09:25:00] AH [09:25:22] Project beta-scap-sync-world build #89565: 04FAILURE in 3 min 40 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/89565/ [09:26:48] (^ keyholder issue, arming) [09:28:28] Project beta-scap-sync-world build #89566: 15ABORTED in 1 min 15 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/89566/ [09:29:26] 10Beta-Cluster-Infrastructure, 10Continuous-Integration-Infrastructure, 10Jenkins, 10ci-test-error: Agent deployment-deploy03 is offline - https://phabricator.wikimedia.org/T329056 (10hashar) We used to have a ferm rule to allow ssh connections. That got removed in June 2020 by https://gerrit.wikimedia.org... [09:31:56] hashar: I am in awe of your troubleshooting skills, thank you for figuring that out! [09:32:39] well [09:33:04] the root cause seems to be us removing the firewall rule that lets contint machines in [09:33:06] which was https://gerrit.wikimedia.org/r/c/operations/puppet/+/606737 [09:33:11] and I have signed-off that commit [09:33:20] cause for `integration` project it is fine (there are no ferm rules) [09:33:26] but somehow missed deployment-prep [09:33:40] and since the ocnnnection is already established removing the iptables rules does not magically terminate it [09:33:40] tl;dr blame you? /j [09:33:57] as to how Jenkins managed to establish connection for the last 2 years and a half, either the instance never had to reboot [09:34:09] or Jenkins is able to ssh in before ferm rules are applied (which is another bug probably) [09:34:27] I think the key there is I have been managing that infra for 12 years and have all the context/history etc [09:34:34] which surely helps finding the root cause faster :-] [09:34:38] :D [09:35:45] Yippee, build fixed! [09:35:46] Project beta-scap-sync-world build #89567: 09FIXED in 7 min 9 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/89567/ [09:37:50] * hashar wrestles with puppet [09:38:33] 10Beta-Cluster-Infrastructure, 10Continuous-Integration-Infrastructure, 10Jenkins, 10Patch-For-Review, 10ci-test-error: Agent deployment-deploy03 is offline - https://phabricator.wikimedia.org/T329056 (10TheresNoTime) p:05Unbreak!→03Triage Dropping from //UBN!//, immediate fault resolved by @hashar a... [09:46:12] Yippee, build fixed! [09:46:13] Project beta-update-databases-eqiad build #65001: 09FIXED in 10 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/65001/ [09:49:55] https://www.isbetabroken.com is all green again ✨ [09:50:53] * RhinosF1 disappears as his train is about to pull into where he will alight [09:51:05] it is official, I need some more monitors (ex: http://ad-exchange.fr/wp-content/uploads/2013/10/Trading_desk2-640x426.jpg ) [09:51:07] o/ thanks for your help RhinosF1 :) [09:51:16] RhinosF1: thanks for the warning ! :] [09:51:34] ahaha I wasn't aware of that status page for beta [09:52:07] we should add it everywhere :] [09:53:40] I can't OP myself here apparently :-( [09:53:44] nor can I change the topic [09:53:46] too bad [09:56:47] hashar: https://butreally.isbetabroken.com [09:59:01] >:D [10:04:27] 10Beta-Cluster-Infrastructure, 10Continuous-Integration-Infrastructure, 10Jenkins, 10Patch-For-Review, 10ci-test-error: Agent deployment-deploy03 is offline - https://phabricator.wikimedia.org/T329056 (10hashar) I have cherry picked https://gerrit.wikimedia.org/r/c/operations/puppet/+/887363 on `deployme... [10:04:30] 10Beta-Cluster-Infrastructure, 10Continuous-Integration-Infrastructure, 10Jenkins, 10Patch-For-Review, 10ci-test-error: Agent deployment-deploy03 is offline - https://phabricator.wikimedia.org/T329056 (10hashar) a:03hashar [10:04:40] so much mystery [10:04:50] on integration there are some ferm::rule and ferm::service [10:04:56] but somehow there is no ferm service running [10:05:06] I am choosing to ignore that issue [10:05:19] TheresNoTime: well done :-] [10:05:40] not that it would have helped here, but I don't have permissions to view the agent log/restart agents — is that something I can be granted? [10:05:54] maybe [10:06:13] ideally we would figure out a way for Jenkins to log all of that mess to some files / syslog [10:06:23] and funnel that up to logstash [10:06:51] yeah, seeing the failure to SSH to the agent would have at least pointed me in the right direction a little more [10:07:08] 10GitLab (Infrastructure), 10serviceops-collab: Automate GitLab version upgrade process - https://phabricator.wikimedia.org/T323569 (10ops-monitoring-bot) Cookbook cookbooks.sre.gitlab.upgrade was started by jelto@cumin1001 on GitLab host gitlab1003.wikimedia.org with reason: Test Upgrade GitLab Replica gitlab... [10:07:13] 10GitLab (Infrastructure), 10serviceops-collab: Automate GitLab version upgrade process - https://phabricator.wikimedia.org/T323569 (10ops-monitoring-bot) Cookbook cookbooks.sre.gitlab.upgrade started by jelto@cumin1001 executed with errors: on GitLab host gitlab1003.wikimedia.org with reason: Test Upgrade Git... [10:08:11] TheresNoTime: ther eis no specific permisission to view the agent logs :-( [10:09:04] :-(, and I guess `Agent/Configure`/`Agent/Connect` is a little sensitive (and not really something I want to be able to do without understanding how it all works) [10:10:06] 10GitLab (Infrastructure), 10serviceops-collab: Automate GitLab version upgrade process - https://phabricator.wikimedia.org/T323569 (10Jelto) [10:33:53] 10Release-Engineering-Team, 10Scap: scap.cfg hostname config sections are not applied - https://phabricator.wikimedia.org/T329144 (10hashar) a:03hashar [10:35:37] TheresNoTime: long term, I would love to move the Beta Cluster Jenkins jobs from https://integration.wikimedia.org/ci/ toward a dedicated Jenkins host [10:35:56] in order to slightly relieve the CI Jenkins [10:36:09] but to do so, we need a way to easily spin up and configure a new Jenkins instance [10:36:28] which is what jnuche and I have been working on recently (so one could then `scap deploy` a jenkins) [10:36:46] theorically, we can then easily setup a Jenkins directly in deployment-prep which will be independent from the CI jenkins [10:36:58] Interesting! Is it not as easy as puppetising the whole jenkins server/agents? [10:37:05] "easy" [10:37:13] and can be administered via yaml files and or the deployment-prep admins [10:37:19] yeah well kind of [10:37:34] for Puppet I did have a look at it ages ago but Jenkins was not as mature as of today [10:37:42] my first iteration was merely adding a bunch of xml files to Puppet [10:37:49] which caused several issues [10:37:54] (I say this with very little understanding of Jenkins *or* Puppet..!) [10:38:00] 1) I don't have merge rights to Puppet [10:38:16] why jenkins? could it just be done with a systemd timer? [10:38:17] 2) keeping the XML config files up to date as jenkins or its plugins are changing is quickly a maintenance burden [10:38:33] 3) we usually don't want to deploy changes with Puppet [10:39:15] since then, Jenkins learned to be configured from the cli and there is a configuration as code plugin that lets one define how a jenkins instnace is configured by using a bunch of yaml files [10:39:36] then the yaml definition is used to configure jenkins which writes the proper xml files by itself [10:39:51] and we go deeper into the Yaml Engineer field [10:40:33] taavi: we had a task about replacing Jenkins for deployment-prep to use cron jobs, but we needed some feedback loop to Gerrit and some kind of dashboard/easy to reach logs which Jenkins provides [10:40:58] well maybe that can be revisited by having the systemd timer scripts to write their output to something publicly available [10:41:04] then Jenkins does that all by itself rather easily [10:41:36] once we are done porting the release and CI jenkins toward yaml config, I am confident it will be reasonably easy to spin up a new Jenkins dedicated solely to deployment-prep [10:41:47] (plus that sounds like a fun project) [10:42:00] and maybe others could be interested in easily spinning up a Jenkins [10:42:31] (that was hashar's braindump of random thoughts) [11:13:02] 10Release-Engineering-Team, 10Scap: scap backport: Multiple changes found for Ifb0316256bdec5008acc48544ddd3e2bf71b6d41 - https://phabricator.wikimedia.org/T323277 (10Urbanecm) >>! In T323277#8596212, @Tgr wrote: > Or maybe just leave it to the user to backport the dependencies with separate `scap backport` ca... [11:22:13] 10Release-Engineering-Team, 10Scap: scap.cfg hostname config sections are not applied - https://phabricator.wikimedia.org/T329144 (10hashar) 05Open→03Resolved The issue is in docker-compose I have used: `hostname: deploy.localhost.` with a trailing `.`. It is the root of the DNS hierarchy and is definitely... [11:42:29] hashar: Maybe it doesn't need Jenkins? Eg a puppetised shell script that runs from cron on its own might suffice. I feel like this was explored a few years ago, maybe we found very good reason that makes this infeasible - I forgot in that case :) [11:44:55] Ah I see the rest now, right, I guess public output for debugging would be useful. Or syslog as MVP. It seems 90% of issues are specific to it connecting to the ssh agent or something getting stuck which would be a category of issues that simply don't exist and don't need debugging. Maybe for the rest we can ssh to deployment host and check logs [11:45:57] It's not clear to me at this point who would make that decision and own it. [12:03:43] 10GitLab (CI & Job Runners), 10serviceops-collab: Use dedicated volume for /var/lib/docker on Trusted Runners - https://phabricator.wikimedia.org/T329035 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by eoghan@cumin1001 for host gitlab-runner1002.eqiad.wmnet with OS bullseye [12:06:45] 10Phabricator Antivandalism Extension, 10DBA, 10Patch-For-Review: Switchover m3 master db1164 -> db1159 - https://phabricator.wikimedia.org/T329141 (10Marostegui) [12:07:23] 10Phabricator Antivandalism Extension, 10DBA, 10Patch-For-Review: Switchover m3 master db1164 -> db1159 - https://phabricator.wikimedia.org/T329141 (10Marostegui) [12:17:55] (03PS1) 10Genoveva Galarza: Zuul: Add Nik.xyz.in e-mail to CI allow list [integration/config] - 10https://gerrit.wikimedia.org/r/887775 [12:30:28] (03CR) 10Jforrester: [C: 03+2] Zuul: [mediawiki/extensions/Wikibase] Drop composer api-testing job (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/887340 (owner: 10Jakob) [12:31:33] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/Wikibase] Drop composer api-testing job [integration/config] - 10https://gerrit.wikimedia.org/r/887340 (owner: 10Jakob) [12:37:32] !log Zuul: [mediawiki/extensions/Wikibase] Drop composer api-testing job [12:37:34] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:03:29] 10GitLab (CI & Job Runners), 10serviceops-collab: Use dedicated volume for /var/lib/docker on Trusted Runners - https://phabricator.wikimedia.org/T329035 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by eoghan@cumin1001 for host gitlab-runner1002.eqiad.wmnet with OS bullseye completed: -... [13:17:14] 10Beta-Cluster-Infrastructure, 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 10Jenkins, 10Quality-and-Test-Engineering-Team (QTE): Move beta cluster automatic deployment to a dedicated infrastructure - https://phabricator.wikimedia.org/T256168 (10hashar) The releases Jenkins i... [13:18:11] Krinkle: taavi: about cronjob to update deployment-prep , that was the task https://phabricator.wikimedia.org/T188367 which I have declined in favor of creating a dedicated Jenkins https://phabricator.wikimedia.org/T256168 [13:19:33] 10Beta-Cluster-Infrastructure, 10Continuous-Integration-Infrastructure, 10Jenkins, 10Release-Engineering-Team (Kanban): Move the beta cluster jobs to a dedicated/standalone Jenkins instance - https://phabricator.wikimedia.org/T183164 (10hashar) For historical purpose, I have declined the task to move to a... [13:19:46] * hashar is still fighting with docker compose :\ [13:50:48] 10GitLab (CI & Job Runners), 10serviceops-collab: Use dedicated volume for /var/lib/docker on Trusted Runners - https://phabricator.wikimedia.org/T329035 (10Jelto) a:03eoghan @eoghan and I reimaged `gitlab-runner1002`. `/var/lib/docker` has a dedicated volume now with a bit under 500GB of space: ` $ df -h F... [13:52:28] 10GitLab (CI & Job Runners), 10serviceops-collab: Use dedicated volume for /var/lib/docker on Trusted Runners - https://phabricator.wikimedia.org/T329035 (10Jelto) [14:52:49] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Blocking 🧱), 10mwcli, 10User-brennen: Add registry.gitlab.com/dependabot-gitlab/dependabot to list of allowed images for gitlab runners - https://phabricator.wikimedia.org/T326507 (10Addshore) I can report it works, and Ill try to write up a guide at... [15:22:01] fun finding, I have been using `docker-compose` v1 which is end of life :/ [15:22:12] :( [15:23:27] so I gotta use the Docker compose plugin instead which is in /usr/libexec/docker/cli-plugins/docker-compose :] [15:24:14] solved! :] [16:20:13] FAILED tests/scap/test_git.py::GitTest::test_clean_tags - UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 1: ordinal not in range(128) [16:20:18] poor scap tests :) [16:40:03] 10GitLab, 10Release-Engineering-Team: Increase poll_timeout on kubernetes gitlab runners - https://phabricator.wikimedia.org/T329196 (10dancy) [16:40:35] 10GitLab: Change my 'full name' in GitLab - https://phabricator.wikimedia.org/T329057 (10bd808) >>! In T329057#8596506, @hashar wrote: > Which implies the Wikitech account has been created by ITS ahead of the onboarding. I guess **one should hint Office IT to create the Wikitech/LDAP account to use the full name... [16:49:46] 10Release-Engineering-Team, 10Scap: scap.cfg hostname config sections are not applied - https://phabricator.wikimedia.org/T329144 (10hashar) 05Resolved→03Open I have decided to fix it afterall :) [16:57:52] (03PS1) 10Subramanya Sastry: Update CSS to reflect bug fixes in the Cite CSS generating script [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/887808 [17:00:48] (03CR) 10Subramanya Sastry: Update CSS to reflect bug fixes in the Cite CSS generating script (031 comment) [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/887808 (owner: 10Subramanya Sastry) [17:24:41] (03CR) 10Subramanya Sastry: [C: 04-1] "Since we know that Safari doesn't support counter-style (and not sure when they will), *but* they support predefined counter styles (ex: h" [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/887808 (owner: 10Subramanya Sastry) [17:35:56] 10Continuous-Integration-Config, 10Accessibility, 10MinervaNeue (Tracking), 10User-zeljkofilipin: Save artifacts in Jenkins for selenium daily Minerva - https://phabricator.wikimedia.org/T328684 (10hashar) Looking at https://integration.wikimedia.org/ci/view/selenium-daily/job/selenium-daily-beta-Minerva/... [17:58:31] dancy: I have brain dumped a few things on https://wikitech.wikimedia.org/wiki/Podman ;) [17:58:44] Thanks hashar. [17:59:27] the reason I went with it is that the scap3-dev env spins up Jenkins using systemd [17:59:33] ENTRYPOINT ["/usr/lib/systemd"] [17:59:51] on Docker that requires the container to be privileged which to me sounds like running anything with full root [18:00:12] and eventually when I spinned it up on my Debian, the container ended replacing my host systemd [18:00:17] killing my x window etc [18:00:34] and showing me a login prompt such as: `jenkins-rel login:` [18:00:52] apparently an issue cause of Debian using cgroupv2 while docker does not support them yet or somethin gon those grounds [18:01:23] so that has been my excuse to try podman since it supports rootless containers and to my surprise it almost worked on the first try even with the outdated version from Debian [18:01:45] then, it does not support BuildKit as I understand it so I can't run the blubber/pipeline magic locally [18:01:56] anyway, we have a draft doc \o/ Thanks for calling about it! [18:11:04] !log Disable spam account https://phabricator.wikimedia.org/people/manage/35991/ [18:11:04] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:12:58] 10Release-Engineering-Team (GitLab IV: Mise En Place 🍱): Try deploying buildkitd as a GitLab CI service - https://phabricator.wikimedia.org/T329213 (10dduvall) [18:16:33] 10GitLab (CI & Job Runners), 10mwbot-rs: GitLab CI jobs failing with "You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" - https://phabricator.wikimedia.org/T329216 (10Legoktm) [18:26:06] 10GitLab (CI & Job Runners), 10Release-Engineering-Team, 10mwbot-rs: GitLab CI jobs failing with "You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" - https://phabricator.wikimedia.org/T329216 (10dancy) [18:39:56] 10GitLab, 10Release-Engineering-Team: buildkitd: Require use of the blubber frontend when running on trusted runners. - https://phabricator.wikimedia.org/T329220 (10dancy) [18:41:40] 10GitLab, 10Release-Engineering-Team: buildkitd: Require use of the blubber frontend when running on trusted runners. - https://phabricator.wikimedia.org/T329220 (10dancy) [18:41:44] 10Release-Engineering-Team: Kokkuri should allow dockerfile.v0 frontend - https://phabricator.wikimedia.org/T326569 (10dancy) [19:00:23] 10GitLab (CI & Job Runners), 10Release-Engineering-Team, 10mwbot-rs: GitLab CI jobs failing with "You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" - https://phabricator.wikimedia.org/T329216 (10Addshore) I also star... [19:25:51] (03CR) 10Subramanya Sastry: "No change after all that work in https://gerrit.wikimedia.org/r/c/mediawiki/services/parsoid/+/887818." [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/887808 (owner: 10Subramanya Sastry) [19:36:14] 10GitLab: Change my 'full name' in GitLab - https://phabricator.wikimedia.org/T329057 (10xcollazo) >>! In T329057#8598415, @bd808 wrote: >>>! In T329057#8596354, @xcollazo wrote: >> IIRC, I didn't create my LDAP user while onboarding. Do we know who this feedback should go to? > > I would love to know who creat... [19:43:26] 10GitLab: Change my 'full name' in GitLab - https://phabricator.wikimedia.org/T329057 (10bd808) >>! In T329057#8599245, @xcollazo wrote: > I do see at https://wikitech.wikimedia.org/wiki/Special:CreateAccount that there is a `Username` field. Would this be the one that maps to the `CN`? I may have very well inco... [19:43:52] 10GitLab, 10Release-Engineering-Team: Increase poll_timeout on kubernetes gitlab runners - https://phabricator.wikimedia.org/T329196 (10jeena) 05Open→03In progress a:03jeena [19:46:36] 10GitLab (CI & Job Runners), 10Release-Engineering-Team, 10mwbot-rs: GitLab CI jobs failing with "You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" - https://phabricator.wikimedia.org/T329216 (10Addshore) Another exa... [19:54:44] 10GitLab (CI & Job Runners), 10Release-Engineering-Team, 10mwbot-rs: GitLab CI jobs failing with "You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" - https://phabricator.wikimedia.org/T329216 (10Addshore) See also ht... [19:55:10] 10Release-Engineering-Team, 10Scap: Scap: Don't transmit "aborted" message to IRC if no prior announcement has been made - https://phabricator.wikimedia.org/T329228 (10dancy) [20:36:31] 10Continuous-Integration-Config, 10Wikimedia-Site-requests, 10User-Urbanecm: CI should ensure that wmf-config/logos.php matches logos/config.yaml - https://phabricator.wikimedia.org/T329231 (10Urbanecm) [20:38:42] (03PS1) 10Urbanecm: [Zuul] Add tox-docker as an experimental job for operations/mediawiki-config [integration/config] - 10https://gerrit.wikimedia.org/r/887829 (https://phabricator.wikimedia.org/T329231) [20:45:28] (03PS2) 10Urbanecm: [Zuul] Add tox-docker as an experimental job for operations/mediawiki-config [integration/config] - 10https://gerrit.wikimedia.org/r/887829 (https://phabricator.wikimedia.org/T329231) [20:55:46] 10Release-Engineering-Team (Priority Backlog 📥), 10Patch-For-Review, 10Release, 10Train Deployments: 1.40.0-wmf.22 deployment blockers - https://phabricator.wikimedia.org/T325585 (10Zabe) [21:00:14] (03CR) 10Hashar: [Zuul] Add tox-docker as an experimental job for operations/mediawiki-config (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/887829 (https://phabricator.wikimedia.org/T329231) (owner: 10Urbanecm) [21:05:28] (03PS3) 10Urbanecm: [Zuul] Add tox-docker as an experimental job for operations/mediawiki-config [integration/config] - 10https://gerrit.wikimedia.org/r/887829 (https://phabricator.wikimedia.org/T329231) [21:05:56] (03CR) 10Urbanecm: [Zuul] Add tox-docker as an experimental job for operations/mediawiki-config (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/887829 (https://phabricator.wikimedia.org/T329231) (owner: 10Urbanecm) [21:09:36] (03PS4) 10Urbanecm: [Zuul] Add tox-docker as an experimental job for operations/mediawiki-config [integration/config] - 10https://gerrit.wikimedia.org/r/887829 (https://phabricator.wikimedia.org/T329231) [21:20:21] 10MediaWiki-Releasing, 10MediaWiki-Installer, 10MediaWiki-Stakeholders-Group, 10Epic, 10MW-1.40-release: Expand the set of bundled extensions and skins in MediaWiki 1.40 - https://phabricator.wikimedia.org/T317146 (10matmarex) [21:28:24] 10Release-Engineering-Team, 10Scap: scap backport: Multiple changes found for Ifb0316256bdec5008acc48544ddd3e2bf71b6d41 - https://phabricator.wikimedia.org/T323277 (10jeena) 05Open→03In progress a:03jeena [21:30:19] (03CR) 10Hashar: [C: 03+2] "You are awesome :] And since I am not sleeping yet I have deployed the job:" [integration/config] - 10https://gerrit.wikimedia.org/r/887829 (https://phabricator.wikimedia.org/T329231) (owner: 10Urbanecm) [21:32:11] (03Merged) 10jenkins-bot: [Zuul] Add tox-docker as an experimental job for operations/mediawiki-config [integration/config] - 10https://gerrit.wikimedia.org/r/887829 (https://phabricator.wikimedia.org/T329231) (owner: 10Urbanecm) [21:32:43] !log Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/887829 # T329231 [21:32:45] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:32:45] T329231: CI should ensure that wmf-config/logos.php matches logos/config.yaml - https://phabricator.wikimedia.org/T329231 [21:32:48] urbanecm: Zuul reloaded :] [21:33:04] thank you very much hashar! and have a wonderful evening :) [21:33:31] thank you so much for tirelessly improving so many small things here and there (and for all the big things I am not aware of) [21:33:33] \o/ [21:33:42] we can promote the job to test/gate-and-submit tomorrow I guess [21:33:50] for now, I am going to sleep for real [21:45:39] 10Release-Engineering-Team, 10Scap: scap backport: Multiple changes found for Ifb0316256bdec5008acc48544ddd3e2bf71b6d41 - https://phabricator.wikimedia.org/T323277 (10jeena) >>! In T323277#8597192, @Urbanecm wrote: >>>! In T323277#8596212, @Tgr wrote: >> Or maybe just leave it to the user to backport the depen... [21:56:20] (03PS1) 10Jforrester: Zuul: Add Nik.xyz.in to CI allowlist [integration/config] - 10https://gerrit.wikimedia.org/r/887835 [22:07:50] James_F: fyi, that ^^ already exists as https://gerrit.wikimedia.org/r/c/integration/config/+/887775 [22:21:13] (03CR) 10Cwhite: "Deployed this change on beta-logs for testing." [releng/phatality] - 10https://gerrit.wikimedia.org/r/832004 (https://phabricator.wikimedia.org/T314098) (owner: 10Cwhite) [22:55:28] 10Release-Engineering-Team, 10Scap, 10Patch-For-Review: scap backport: Multiple changes found for Ifb0316256bdec5008acc48544ddd3e2bf71b6d41 - https://phabricator.wikimedia.org/T323277 (10jeena) MR above modifies the depends_on check to request dependecies for a change_id in a specific project & branch to avo... [23:04:56] 10Release-Engineering-Team, 10Scap: scap backport does not handle Depends-On header correctly - https://phabricator.wikimedia.org/T324275 (10jeena) I will add some extra output to clarify what is happening (checking for dependencies of the dependency). [23:11:44] (03CR) 10Tim Starling: dockerfiles: php-compile fix ambiguous env assignment (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/764820 (owner: 10Hashar) [23:13:55] urbanecm: Ha, I should talk to my teammate it appears. :-) [23:14:04] (03CR) 10Jforrester: [C: 03+2] Zuul: Add Nik.xyz.in e-mail to CI allow list [integration/config] - 10https://gerrit.wikimedia.org/r/887775 (owner: 10Genoveva Galarza) [23:14:16] (03Abandoned) 10Jforrester: Zuul: Add Nik.xyz.in to CI allowlist [integration/config] - 10https://gerrit.wikimedia.org/r/887835 (owner: 10Jforrester) [23:15:08] (03Merged) 10jenkins-bot: Zuul: Add Nik.xyz.in e-mail to CI allow list [integration/config] - 10https://gerrit.wikimedia.org/r/887775 (owner: 10Genoveva Galarza) [23:17:42] !log Zuul: Add Nik.xyz.in e-mail to CI allow list [23:17:43] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [23:21:25] 10Continuous-Integration-Config, 10MediaWiki-Documentation, 10Performance-Team, 10Patch-For-Review: MediaWiki core docs unavailable for MW 1.35 and later - https://phabricator.wikimedia.org/T317451 (10Krinkle) 05Open→03Resolved a:03Krinkle