[01:35:02] there was peach cobbler! [05:42:04] (03PS4) 10Hashar: Add job for integration/zuul/deploy [integration/config] - 10https://gerrit.wikimedia.org/r/940950 (https://phabricator.wikimedia.org/T342346) [06:15:13] (03CR) 10Hashar: [C: 03+2] Add job for integration/zuul/deploy [integration/config] - 10https://gerrit.wikimedia.org/r/940950 (https://phabricator.wikimedia.org/T342346) (owner: 10Hashar) [06:16:18] (03Merged) 10jenkins-bot: Add job for integration/zuul/deploy [integration/config] - 10https://gerrit.wikimedia.org/r/940950 (https://phabricator.wikimedia.org/T342346) (owner: 10Hashar) [06:25:19] (03CR) 10Hashar: "recheck" [integration/zuul/deploy] - 10https://gerrit.wikimedia.org/r/606242 (https://phabricator.wikimedia.org/T259611) (owner: 10Hashar) [06:25:37] (03CR) 10Hashar: "recheck" [integration/zuul/deploy] - 10https://gerrit.wikimedia.org/r/940166 (https://phabricator.wikimedia.org/T258630) (owner: 10Hashar) [06:25:49] (03CR) 10Hashar: "recheck" [integration/zuul/deploy] - 10https://gerrit.wikimedia.org/r/940167 (https://phabricator.wikimedia.org/T342346) (owner: 10Hashar) [06:37:50] 10Release-Engineering-Team, 10Fresh: Prepare a new release of Fresh - https://phabricator.wikimedia.org/T342601 (10hashar) [07:52:56] !log Deployed rev2 of sec patch for T340221 [07:52:57] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [08:43:31] 10GitLab (Auth & Access), 10Release-Engineering-Team (They Live 🕶️🧟), 10CAS-SSO, 10Infrastructure-Foundations, and 4 others: migrate gitlab away from the CAS protocol - https://phabricator.wikimedia.org/T320390 (10SLyngshede-WMF) I did a diff of the configurations for idp and idp-test, and they are basical... [09:37:11] 10Release-Engineering-Team, 10Fresh, 10Release: Prepare a new release of Fresh - https://phabricator.wikimedia.org/T342601 (10Reedy) [09:46:37] 10GitLab (Auth & Access), 10Release-Engineering-Team (They Live 🕶️🧟), 10CAS-SSO, 10Infrastructure-Foundations, and 4 others: migrate gitlab away from the CAS protocol - https://phabricator.wikimedia.org/T320390 (10Jelto) We restarted `idp1002` and `idp-test1002`. It seems the running configuration was not... [10:20:01] Project beta-update-databases-eqiad build #69000: 04FAILURE in 0.86 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/69000/ [10:56:57] Hm [11:07:42] Yippee, build fixed! [11:07:42] Project beta-update-databases-eqiad build #69001: 09FIXED in 10 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/69001/ [11:21:50] (03CR) 10Jaime Nuche: [C: 03+1] Link to git blames for each of the stacktrace frames (031 comment) [releng/phatality] - 10https://gerrit.wikimedia.org/r/940265 (https://phabricator.wikimedia.org/T342400) (owner: 10Dduvall) [11:45:54] 10GitLab (Auth & Access), 10Release-Engineering-Team (They Live 🕶️🧟), 10CAS-SSO, 10Infrastructure-Foundations, and 4 others: migrate gitlab away from the CAS protocol - https://phabricator.wikimedia.org/T320390 (10SLyngshede-WMF) We previously suspected that the issue was that CAS nested the attributes it... [12:24:40] (03PS2) 10Hashar: Add missing frozen requirements [integration/zuul/deploy] - 10https://gerrit.wikimedia.org/r/940167 (https://phabricator.wikimedia.org/T342346) [12:24:42] (03PS2) 10Hashar: Use gear from upstream and bump it to 0.16.0 [integration/zuul/deploy] - 10https://gerrit.wikimedia.org/r/940166 (https://phabricator.wikimedia.org/T258630) [13:22:27] (03PS3) 10Hashar: Add missing frozen requirements [integration/zuul/deploy] - 10https://gerrit.wikimedia.org/r/940167 (https://phabricator.wikimedia.org/T342346) [13:22:29] (03PS3) 10Hashar: Use gear from upstream and bump it to 0.16.0 [integration/zuul/deploy] - 10https://gerrit.wikimedia.org/r/940166 (https://phabricator.wikimedia.org/T258630) [13:34:31] (03PS2) 10Hashar: Add support to build with podman [integration/zuul/deploy] - 10https://gerrit.wikimedia.org/r/940168 (https://phabricator.wikimedia.org/T342346) [13:35:18] (03CR) 10Hashar: [C: 03+2] Add support to build with podman [integration/zuul/deploy] - 10https://gerrit.wikimedia.org/r/940168 (https://phabricator.wikimedia.org/T342346) (owner: 10Hashar) [13:35:54] (03Merged) 10jenkins-bot: Add support to build with podman [integration/zuul/deploy] - 10https://gerrit.wikimedia.org/r/940168 (https://phabricator.wikimedia.org/T342346) (owner: 10Hashar) [13:51:03] 10GitLab (Infrastructure), 10collaboration-services: Fix gitlab-runner registration token in gitlab-runners wmcs project - https://phabricator.wikimedia.org/T342654 (10Jelto) [14:37:37] (03PS2) 10Hashar: Add frozen-requirements.txt as a make target [integration/zuul/deploy] - 10https://gerrit.wikimedia.org/r/940170 (https://phabricator.wikimedia.org/T342346) [14:37:52] (03CR) 10Hashar: [C: 03+2] Add frozen-requirements.txt as a make target [integration/zuul/deploy] - 10https://gerrit.wikimedia.org/r/940170 (https://phabricator.wikimedia.org/T342346) (owner: 10Hashar) [14:38:30] (03Merged) 10jenkins-bot: Add frozen-requirements.txt as a make target [integration/zuul/deploy] - 10https://gerrit.wikimedia.org/r/940170 (https://phabricator.wikimedia.org/T342346) (owner: 10Hashar) [14:39:55] dduvall: is this me or gitlab? https://gitlab.wikimedia.org/andrew/gitlab-trusted-runner/-/jobs/123619 [14:40:18] (03CR) 10Hashar: [C: 03+2] Add missing frozen requirements [integration/zuul/deploy] - 10https://gerrit.wikimedia.org/r/940167 (https://phabricator.wikimedia.org/T342346) (owner: 10Hashar) [14:40:51] (03Merged) 10jenkins-bot: Add missing frozen requirements [integration/zuul/deploy] - 10https://gerrit.wikimedia.org/r/940167 (https://phabricator.wikimedia.org/T342346) (owner: 10Hashar) [14:42:06] (03PS5) 10Hashar: Reuse previous artifacts as a wheel cache [integration/zuul/deploy] - 10https://gerrit.wikimedia.org/r/606242 (https://phabricator.wikimedia.org/T259611) [14:42:45] (03PS2) 10Hashar: Output diff of the list of wheels held in artifacts [integration/zuul/deploy] - 10https://gerrit.wikimedia.org/r/940169 (https://phabricator.wikimedia.org/T342346) [14:43:01] (03CR) 10Hashar: [C: 03+2] Output diff of the list of wheels held in artifacts [integration/zuul/deploy] - 10https://gerrit.wikimedia.org/r/940169 (https://phabricator.wikimedia.org/T342346) (owner: 10Hashar) [14:43:06] (03CR) 10Hashar: [C: 03+2] Reuse previous artifacts as a wheel cache [integration/zuul/deploy] - 10https://gerrit.wikimedia.org/r/606242 (https://phabricator.wikimedia.org/T259611) (owner: 10Hashar) [14:45:51] (03Merged) 10jenkins-bot: Reuse previous artifacts as a wheel cache [integration/zuul/deploy] - 10https://gerrit.wikimedia.org/r/606242 (https://phabricator.wikimedia.org/T259611) (owner: 10Hashar) [14:45:53] (03Merged) 10jenkins-bot: Output diff of the list of wheels held in artifacts [integration/zuul/deploy] - 10https://gerrit.wikimedia.org/r/940169 (https://phabricator.wikimedia.org/T342346) (owner: 10Hashar) [14:48:01] (03PS4) 10Hashar: Use gear from upstream and bump it to 0.16.0 [integration/zuul/deploy] - 10https://gerrit.wikimedia.org/r/940166 (https://phabricator.wikimedia.org/T258630) [15:07:08] (03CR) 10Hashar: [C: 04-1] "This changes requires to update `artifacts/artifacts.buster.tar.gz` and I'd like it to reuse existing wheels rather than recreating the wh" [integration/zuul/deploy] - 10https://gerrit.wikimedia.org/r/940166 (https://phabricator.wikimedia.org/T258630) (owner: 10Hashar) [15:26:08] andrewbogott: The job didn't run because it has the 'trusted' tag, but your repo is not listed in https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/blob/main/projects.json (nor should it be). [15:27:27] ok -- I'm new to this, should I be following a totally different workflow? I was assuming fork, automated tests, PR [15:27:39] Should I just skip the middle step? [15:27:52] (Well, I mean, I didn't actually set up those tests anyway, it's just a side-effect of forking) [15:29:18] * bd808 assumes andrewbogott is in the fuzzy realm of "things somebody may have thought of, but we certainly do not have well established procedures for yet" [15:29:24] andrewbogott: you can still submit the MR even if the pipeline fails on your fork, i believe [15:29:45] yeah.. it's a side effect of forking at and this time I really don't know of a good way of not confusing people. [15:30:02] *and at this time. [15:30:06] Great, I'll just submit the MR then (and learn to say MR instead of PR) [15:30:10] but first... meetings! [15:30:15] you should submit those two changes together in the same MR, however, since the first is not correct (missing the comma) [15:30:24] dancy: would it be possible to add some filter to the pipeline so it does not on forks? [15:30:46] I'm not aware of any such setting but I'll do some googling . [15:32:04] i suppose we could test `CI_PROJECT_ID` in the workflow rules [15:32:14] Yeah, that seems to be the way. [16:02:31] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (They Live 🕶️🧟), 10Patch-For-Review: buildkitd: Require use of the blubber frontend when running on trusted runners. - https://phabricator.wikimedia.org/T329220 (10CodeReviewBot) dancy merged https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_req... [16:02:45] https://gitlab.com/gitlab-org/gitlab/-/issues/14508 is a 7-year old request to make a CI variable just for the fork status [16:09:03] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (They Live 🕶️🧟), 10Patch-For-Review: buildkitd: Require use of the blubber frontend when running on trusted runners. - https://phabricator.wikimedia.org/T329220 (10CodeReviewBot) dancy opened https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_req... [16:11:57] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (They Live 🕶️🧟), 10Patch-For-Review: buildkitd: Require use of the blubber frontend when running on trusted runners. - https://phabricator.wikimedia.org/T329220 (10CodeReviewBot) dancy merged https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_req... [16:13:19] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (They Live 🕶️🧟), 10Patch-For-Review: buildkitd: Require use of the blubber frontend when running on trusted runners. - https://phabricator.wikimedia.org/T329220 (10dancy) 05Open→03Resolved [16:13:23] 10Release-Engineering-Team (They Live 🕶️🧟), 10Patch-For-Review: Kokkuri should allow dockerfile.v0 frontend - https://phabricator.wikimedia.org/T326569 (10dancy) [16:21:18] 10Release-Engineering-Team (They Live 🕶️🧟), 10Patch-For-Review: Kokkuri should allow dockerfile.v0 frontend - https://phabricator.wikimedia.org/T326569 (10xcollazo) For completeness, we did start utilizing this mechanism via @Antoine_Quhen's work on T318346 for our `airflow-dags` project. We also want to migr... [16:31:05] dduvall: I believe that I have created a merge request! [16:57:48] andrewbogott: w00t! [17:01:31] andrewbogott: you're missing another comma. i suggested the change which you should be able to apply directly in the UI [17:16:30] dduvall: done, I think [17:42:20] what the... `This job is stuck because of one of the following problems. There are no active runners online, no runners for the protected branch, or no runners that match all of the job's tags: trusted` [17:42:49] it would be nice if GitLab would tell me which of those three scenarios is true [17:43:04] ostensibly it knows [17:46:10] So next I want to actually trigger a build -- for that I just push a tag, right? [17:48:42] andrewbogott: i don't see a `.gitlab-ci.yml` file in your repo. that's your next step [17:49:17] i wish we had a nice CI template available for folks in kokkuri but we don't yet [17:49:23] dduvall: confusingly everything I'm doing is on the 2023.1 branch [17:49:54] andrewbogott: oh, gotcha ok [17:50:59] then yeah, according to your rules it looks like pushing a protected tag is going to trigger your publish job [17:51:07] https://www.irccloud.com/pastebin/dJNkWSXm/ [17:52:15] What is a 'protected tag'? Is that a git concept or a gitlab concept? [17:52:25] it's a gitlab concept [17:53:12] tags/branches are protected according to the pattern matching set up in project settings [17:55:16] that confers some additional access control in your project. we also make assertions on the `protected_ref` claim of the JWT token on the registry side to ensure that only images built from protected refs are allowed in the prod registry [17:56:16] ok... I'm reading some gitlab docs now [17:57:00] dduvall: so I create a normal tag, and then use the gitlab ui to protect it [17:57:44] no. you configure a regex of tags to protect in the repo settings, and then push a tag matching that regex [17:57:50] you typically want to establish some kind of pattern for protected tags/branches. protect everything you want to build artifacts from i would say [17:58:00] what taavi said :) [17:58:29] looks like the UI supports either thing [17:58:35] but your way sounds easier [18:00:41] honestly i'm not sure what the use case is for unprotected tags. e.g. in blubber we protect all tags using `*` [18:05:00] I pushed a protected tag. Now I just wait a few hours for an email, or is there someplace to see that it's doing things? [18:07:07] https://gitlab.wikimedia.org/repos/cloud/cloud-vps/horizon/deploy/-/pipelines [18:08:57] thx [18:09:01] 'docker-registry.wikimedia.org/repos/releng/blubber:v0.17.0' is not an allowed gateway frontend [18:47:53] andrewbogott: `docker-registry.wikimedia.org/repos/releng/blubber/buildkit:v0.18.0` would be the latest (notice the `blubber/buildkit` part) [18:54:28] 10Beta-Cluster-Infrastructure, 10Scap, 10Technical-Debt: Reconcile "Scap: scap_source correct gid" Beta Cluster hot patch - https://phabricator.wikimedia.org/T342320 (10Krinkle) 05Open→03Resolved a:03dancy Thanks! [18:54:39] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Seen), 10Patch-For-Review: Beta puppetmaster cherry-pick process - https://phabricator.wikimedia.org/T135427 (10Krinkle) [18:57:20] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Seen), 10Patch-For-Review: Beta puppetmaster cherry-pick process - https://phabricator.wikimedia.org/T135427 (10Krinkle) As of now, we are down from 20+ to just six local patches. Two are fairly recent and being worked on in Gerrit. The other four ar... [19:05:41] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Seen), 10Patch-For-Review: Beta puppetmaster cherry-pick process - https://phabricator.wikimedia.org/T135427 (10Krinkle) @thcipriani I'm aware a better future always seems "just" six months away, but given the above, perhaps we can institute this pol... [19:47:12] dduvall: what's the difference between those two images? [19:48:27] taavi: we started publishing under the `/buildkit` suffix to be clear that it was the buildkit frontend build of blubber [20:02:03] puppet says "Notice: /Stage[main]/Profile::Openstack::Base::Horizon::Docker_deploy/Service::Docker[openstack-dashboard]/Exec[docker pull of docker-registry.wikimedia.org/wikimedia/openstack-horizon:2023-07-25-191000-dev for openstack-dashboard]/returns: executed successfully" which is good! [20:02:09] but syslog says "/usr/bin/docker: Error response from daemon: manifest for docker-registry.wikimedia.org/wikimedia/openstack-horizon:2023-07-25-191000-dev not found." which is bad [20:02:32] * andrewbogott still looking for the typo [20:04:21] andrewbogott: https://gitlab.wikimedia.org/repos/cloud/cloud-vps/horizon/deploy/-/jobs/124014 says `repos/cloud/cloud-vps/horizon/deploy`, which is quite different from `wikimedia/openstack-horizon` [20:05:07] yeah [20:05:17] so probably the thing that says it's successful is just wrong [20:05:37] https://docker-registry.wikimedia.org/repos/cloud/cloud-vps/horizon/deploy/tags/ supports that [20:31:09] Project beta-update-databases-eqiad build #69011: 04FAILURE in 11 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/69011/ [21:31:03] Project beta-update-databases-eqiad build #69012: 04STILL FAILING in 11 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/69012/ [22:31:00] Project beta-update-databases-eqiad build #69013: 04STILL FAILING in 10 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/69013/ [23:01:42] segfault [23:03:04] oof. [23:03:09] php[1055]: segfault at 7ffe01e6ef58 ip 00007fda92266807 sp 00007ffe01e6ef40 error 6 in libpcre2-8.so.0.7.1[7fda92223000+5d000] [23:05:14] well. I guess it's reproducable anyway. [23:23:29] > cannot open `/usr/lib/systemd/systemd-coredump' (No such file or directory) [23:23:53] well. that explains why it failed to create a coredump at least. [23:28:59] alright, lets see the next run if it generates a dump [23:30:48] Project beta-update-databases-eqiad build #69014: 04STILL FAILING in 10 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/69014/ [23:36:41] > /etc/systemd/coredump.conf:6: Failed to parse size value, ignoring: 20% [23:36:43] progress