[00:15:51] 10Release-Engineering-Team (GitLab III: GitLab in LA 🪃), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.40.0-wmf.10 deployment blockers - https://phabricator.wikimedia.org/T320515 (10brennen) [00:17:00] Krinkle: sorry if i closed T323153 too early [00:17:01] T323153: PHP Notice: Unexpected clearActionName after getActionName already called - https://phabricator.wikimedia.org/T323153 [00:19:27] brennen: nope, that's fine, I should file a new task for that. [00:20:54] cool cool. [01:48:38] 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-BlockAndNuke, 10Wikimedia-extension-review-queue: Consider installing BlockAndNuke on the Beta Cluster - https://phabricator.wikimedia.org/T176207 (10Kizule) 05Open→03Declined Closing as per {T299262}. [07:55:29] Krinkle: you are welcome. According to my testing the change to your Gerrit account should not have any effect ,) [08:37:53] 10GitLab: Set new owner in wmit-wikimedia GitLab group - https://phabricator.wikimedia.org/T323196 (10valerio.bozzolan) [08:38:36] 10Gerrit: Fix up Gerrit duplicate LDAP external ids - https://phabricator.wikimedia.org/T323135 (10hashar) And this morning I thought: what about upper case unicode characters? There are **three** such accounts found by matching against unicode characters having the property {nav letter + uppercase} which in pcr... [08:45:21] !log gerrit: deleted 192 LDAP accounts (scheme `gerrit:`) containing upper case characters which had an exact equivalent in an all lower case form # [08:45:22] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [08:45:42] !log gerrit: deleted 192 LDAP accounts (scheme `gerrit:`) containing upper case characters which had an exact equivalent in an all lower case form. `All-Users.git` commit is 5e5800ecc8fd5da591567e616898dd6df988c0c8 # T323135 [08:45:43] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [08:45:44] T323135: Fix up Gerrit duplicate LDAP external ids - https://phabricator.wikimedia.org/T323135 [08:46:13] !log gerrit: reindexed accounts `ssh -p 29418 gerrit.wikimedia.org -- gerrit index start accounts --force` # T323135 [08:46:15] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [09:15:20] 10Continuous-Integration-Infrastructure, 10Jenkins, 10Security: Jenkins plugins security advisory - 2022-11-15 - https://phabricator.wikimedia.org/T323054 (10MoritzMuehlenhoff) [09:15:37] 10Continuous-Integration-Infrastructure, 10Jenkins, 10Security: Jenkins plugins security advisory - 2022-11-15 - https://phabricator.wikimedia.org/T323054 (10MoritzMuehlenhoff) [09:16:19] 10Release-Engineering-Team (Priority Backlog 📥), 10Gerrit (Gerrit 3.5): Upgrade to Gerrit 3.5 - https://phabricator.wikimedia.org/T307334 (10hashar) [09:16:30] 10Gerrit: Fix up Gerrit duplicate LDAP external ids - https://phabricator.wikimedia.org/T323135 (10hashar) 05Open→03Resolved a:03hashar Upstream script `contrib/find-duplicate-usernames.sh` no more reports any duplicate against `username:`, `gerrit:` or `external:` scheme \o/ [09:18:04] 10GitLab: Set new owner in wmit-wikimedia GitLab group - https://phabricator.wikimedia.org/T323196 (10valerio.bozzolan) [09:48:45] (03PS1) 10Hashar: gerrit: remove Gerrit 3.5 obsolete @apply css statement [software/gerrit] (deploy/wmf/stable-3.5) - 10https://gerrit.wikimedia.org/r/857499 (https://phabricator.wikimedia.org/T315445) [09:49:26] (03CR) 10Hashar: [C: 03+2] gerrit: remove Gerrit 3.5 obsolete @apply css statement [software/gerrit] (deploy/wmf/stable-3.5) - 10https://gerrit.wikimedia.org/r/857499 (https://phabricator.wikimedia.org/T315445) (owner: 10Hashar) [09:49:53] (03Merged) 10jenkins-bot: gerrit: remove Gerrit 3.5 obsolete @apply css statement [software/gerrit] (deploy/wmf/stable-3.5) - 10https://gerrit.wikimedia.org/r/857499 (https://phabricator.wikimedia.org/T315445) (owner: 10Hashar) [09:50:17] 10GitLab (Infrastructure), 10serviceops-collab: Evaluate a high available GitLab architecture - https://phabricator.wikimedia.org/T323201 (10Jelto) [09:52:32] 10GitLab (Infrastructure), 10serviceops-collab, 10Patch-For-Review: bring new gitlab hardware servers into production - https://phabricator.wikimedia.org/T307142 (10Jelto) 05In progress→03Resolved All GitLab machines in codfw and eqiad are migrated to physical hosts. I created a follow-up task T323201 to... [09:53:41] 10GitLab (Infrastructure), 10serviceops-collab, 10Patch-For-Review: bring new gitlab hardware servers into production - https://phabricator.wikimedia.org/T307142 (10Jelto) [10:16:21] 10Gerrit, 10Upstream: Support posting screenshots in Gerrit - https://phabricator.wikimedia.org/T228084 (10hashar) >>! In T228084#5549908, @Paladox wrote: > You can either configure the inbuilt image server or use an external one, see https://gerrit.googlesource.com/plugins/imagare/+/refs/heads/master/src/main... [10:24:56] 10Gerrit: Gerrit email subject line should distinguish between "has posted comments on this change" and " I'd like you to reexamine a change." - https://phabricator.wikimedia.org/T70598 (10hashar) 05Open→03Declined Gerrit now has the attention set feature ( https://gerrit.wikimedia.org/r/Documentation/user-a... [10:32:30] 10Gerrit: Do not expand large files (like package-lock.json) by default when `Expand All` is clicked - https://phabricator.wikimedia.org/T262944 (10hashar) [10:33:41] 10Continuous-Integration-Infrastructure, 10phan, 10ci-test-error: Phan gets confused about the type of values coming from array fields, causing spurious "Call to undeclared method" errors. - https://phabricator.wikimedia.org/T323205 (10daniel) [10:34:27] 10Continuous-Integration-Infrastructure, 10phan, 10ci-test-error: Phan gets confused about the type of values coming from array fields, causing spurious "Call to undeclared method" errors. - https://phabricator.wikimedia.org/T323205 (10daniel) [12:34:16] 10Continuous-Integration-Infrastructure, 10Jenkins, 10Infrastructure-Foundations, 10SRE, 10puppet-compiler: compiler1002.puppet-diffs.eqiad.wmflabs disk is full - https://phabricator.wikimedia.org/T222072 (10jbond) [12:39:37] I guess there isn't anything more specific than 'this' month for 1.39.0? [12:42:28] 10Continuous-Integration-Infrastructure, 10phan, 10ci-test-error: Phan gets confused about the type of values coming from array fields, causing spurious "Call to undeclared method" errors. - https://phabricator.wikimedia.org/T323205 (10Daimona) > The assignment `$revRecord = $status->value['revision-record']... [13:05:07] hashar: you around? [13:52:49] hashar: no effect, then why do I feel so happy!!! [14:20:42] 10Continuous-Integration-Infrastructure, 10phan, 10ci-test-error: Phan gets confused about the type of values coming from array fields, causing spurious "Call to undeclared method" errors. - https://phabricator.wikimedia.org/T323205 (10Daimona) >>! In T323205#8399359, @Daimona wrote: >> For some reason, Phan... [14:36:06] arturo: yes back [14:49:16] hashar: o/ [14:49:27] \o [14:49:46] I'm having problems running ops/puppet.git `utils/run_ci_locally.sh` [14:50:05] 10Release-Engineering-Team (Priority Backlog 📥), 10Release, 10Train Deployments: 1.40.0-wmf.12 deployment blockers - https://phabricator.wikimedia.org/T320517 (10matthiasmullie) [14:50:06] but I'm suspicious of my local docker setup having something weird regarding permissions of the docker data root [14:50:30] this is what I get https://www.irccloud.com/pastebin/t3Q1pJvR/ [14:51:06] fun [14:51:19] any hint? [14:52:09] you can run it with bash -x to get the full command line used: `bash -x utils/run_ci_locally.sh` and from the script set `INTERACTIVE=yes` should drop you in a bash [14:52:17] the container should be running as the `nobody` user [14:52:20] which might be the issue [14:52:46] oh [14:53:09] it is running this [14:53:10] docker run --rm --env ZUUL_REF= --env RAKE_TARGET= --name puppet-tests-latest --volume /home/arturo/git/wmf/operations/puppet:/src docker-registry.wikimedia.org/releng/operations-puppet:latest [14:53:53] yeah so nothing special beside that it mounts some files from your home directory to /src [14:53:55] (03PS4) 10Stef Dunlap: WikiLambda: run e2e tests daily on betacluster [integration/config] - 10https://gerrit.wikimedia.org/r/856646 (https://phabricator.wikimedia.org/T294388) [14:54:16] the entrypoint is `execute_local` at https://gerrit.wikimedia.org/r/plugins/gitiles/integration/config/+/refs/heads/master/dockerfiles/operations-puppet/run.sh#58 [14:57:51] (03CR) 10Stef Dunlap: "I'm not sure if I did this right. My intention is to override the job defined in the WikiLambda (sub)project, so that it points to the wik" [integration/config] - 10https://gerrit.wikimedia.org/r/856646 (https://phabricator.wikimedia.org/T294388) (owner: 10Stef Dunlap) [14:58:32] -rw------- 1 nobody nogroup 1188 Sep 29 12:27 /srv/workspace/puppet/.bundle/ruby/2.5.0/gems/net-scp-3.0.0/net-scp-public_cert.pem [14:58:44] arturo: so yeah the file in the container is only readable by `nobody` [14:59:09] but that should be the user used to run the container [15:00:15] mmmm [15:00:20] I run docker as my user [15:00:33] I wonder if running it as root would make any difference [15:00:37] rather than the docker daemon running as root right? [15:00:59] $ docker run --rm -it --entrypoint=id docker-registry.wikimedia.org/releng/operations-puppet:latest [15:00:59] uid=65534(nobody) gid=65534(nogroup) groups=65534(nogroup) [15:01:24] same here: uid=65534(nobody) gid=65534(nogroup) groups=65534(nogroup) [15:01:32] * hashar is puzzled [15:02:33] can it reads the file? [15:02:35] docker run --rm -it --entrypoint=cat docker-registry.wikimedia.org/releng/operations-puppet:latest /srv/workspace/puppet/.bundle/ruby/2.5.0/gems/net-scp-3.0.0/net-scp-public_cert.pem [15:02:41] (gives me some certificate) [15:03:05] cat: /srv/workspace/puppet/.bundle/ruby/2.5.0/gems/net-scp-3.0.0/net-scp-public_cert.pem: Permission denied [15:03:58] ah [15:04:01] so that reproduces it [15:04:51] is this the same in your setup? [15:04:52] https://www.irccloud.com/pastebin/8AJKiK14/ [15:05:40] nope they are all owned by id 65534 [15:05:57] so I guess the entrypoint script runs as `nobody` but somehow the files on your machine are owned by your user [15:06:09] maybe there is some uid mapping happening? -:\ [15:06:15] * arturo nods [15:07:06] maybe you can teach the utils/run_ci_locally.sh script to run as a different user [15:07:58] probably by introducing an env variable [15:08:07] cause all arguments are passed to rake [15:10:21] thanks for the assistance [15:10:30] I will try now reinstalling my whole docker stack [15:19:07] hashar: new data point: I tried with podman, and it listed the right owners for the files :-( [15:19:31] right? [15:19:34] like nobody/nobody? [15:19:59] like `-rw------- 1 65534 65534 1188 Sep 29 12:27 net-scp-public_cert.pem` [15:24:49] hashar: the ci script works just fine with podman, so I'll stick with it :-) [15:24:55] again, thanks for the assistance, really appreciated [15:43:19] arturo: awesome! congratulations on fixing it :-] [15:43:27] 🎉 [15:44:11] arturo: and I think you can add to the ci_run_locally.sh script some parameter such as DOCKER_USER and/or OCI_USER which value would be passed as `--user` to the oci runtime [15:44:29] defaulting to well hmm who knows, nothing or hardcoded to `nobody` [15:47:49] * arturo nods [15:58:38] 10Release-Engineering-Team (Priority Backlog 📥), 10Gerrit (Gerrit 3.5): Upgrade to Gerrit 3.5 - https://phabricator.wikimedia.org/T307334 (10hashar) I have scheduled the upgrade for tomorrow Thursday November 17th at 9:00 UTC which is just after the backport window. https://wikitech.wikimedia.org/wiki/Deployme... [16:32:49] 10Release-Engineering-Team (Radar), 10MediaWiki-Vagrant, 10Parsoid (Tracking): Decide the fate of Parsoid/JS Puppet in Mediawiki-Vagrant - https://phabricator.wikimedia.org/T259988 (10Tgr) [16:34:42] 10Release-Engineering-Team (Radar), 10MediaWiki-Vagrant, 10Parsoid (Tracking): Decide the fate of Parsoid/JS Puppet in Mediawiki-Vagrant - https://phabricator.wikimedia.org/T259988 (10Tgr) > Add `_js` to the end of the name of all current puppet roles that need Parsoid/JS and create new roles that using Pars... [16:51:07] 10Continuous-Integration-Infrastructure, 10phan, 10ci-test-error: Phan gets confused about the type of values coming from array fields, causing spurious "Call to undeclared method" errors. - https://phabricator.wikimedia.org/T323205 (10daniel) But how does Phan get the idea that the type of $revRecord is Sta... [16:59:28] dduvall: should it still be possible to create a docker registry tag by pushing a tag to gerrit? It's been a 1 year and 8 months since you sorted that out for us, and I don't think I've tried since (https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/services/kask/+/6d7329308bff06474a9fd8d2e15b50048df5901f%5E%21/#F0). [17:01:28] urandom: it depends on the zuul configuration for that project. zuul is responsible for responding to gerrit events (including `ref-update` which fires for tag pushes) and kicking on the right jenkins jobs (in this case, the pipeline job that corresponds to your pipeline in `.pipeline/config.yaml`) [17:01:34] tl;dr let me check :) [17:03:09] the `publish` zuul pipeline is what fires for `ref-update`, and that looks configured for kask https://gerrit.wikimedia.org/r/plugins/gitiles/integration/config/+/refs/heads/master/zuul/layout.yaml#8657 [17:04:12] huh, I wonder why it didn't work [17:04:15] and it looks like kask has a project pipeline called "publish" https://gerrit.wikimedia.org/r/plugins/gitiles/integration/config/+/refs/heads/master/jjb/project-pipelines.yaml#242 [17:04:43] https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/services/kask/+/refs/heads/master/.pipeline/config.yaml#18 [17:06:12] looks like a job did fire for a tag push [17:06:13] https://integration.wikimedia.org/ci/job/trigger-kask-pipeline-publish/12/ [17:06:22] ah, sorry. i'm late for a meeting :) [17:06:34] i can look more into this in about 50 min! [17:08:09] > Triggered by reference: refs/tags/v1.0.10 [17:08:20] 10Continuous-Integration-Infrastructure, 10phan, 10ci-test-error: Phan gets confused about the type of values coming from array fields, causing spurious "Call to undeclared method" errors. - https://phabricator.wikimedia.org/T323205 (10Daimona) >>! In T323205#8400010, @daniel wrote: > But how does Phan get t... [17:11:03] dduvall: I'm not sure how I missed that, maybe because of the lexical sort? [17:12:39] dduvall: sorry for the noise; rest assured I am experiencing the appropriate level of shame & regret! :) [17:37:22] (03CR) 10Jforrester: "Looks reasonable. Normally we do these kinds of patches as two patches (one to jjb, the other to Zuul config), but this should be OK." [integration/config] - 10https://gerrit.wikimedia.org/r/856646 (https://phabricator.wikimedia.org/T294388) (owner: 10Stef Dunlap) [17:41:01] 10GitLab, 10Release-Engineering-Team (GitLab III: GitLab in LA 🪃): Provision Horizontal Pod Autoscaler (HPA) for GitLab cloud runners - https://phabricator.wikimedia.org/T323164 (10thcipriani) [17:41:09] 10Release-Engineering-Team (GitLab III: GitLab in LA 🪃): Try DigitalOcean registry for buildkit caching - https://phabricator.wikimedia.org/T323148 (10dancy) [17:41:18] 10Release-Engineering-Team (GitLab III: GitLab in LA 🪃): Try DigitalOcean object storage for buildkit caching - https://phabricator.wikimedia.org/T323147 (10dancy) [17:42:32] 10GitLab, 10Release-Engineering-Team (GitLab III: GitLab in LA 🪃): Try local directory export + GitLab cache for buildkit caching - https://phabricator.wikimedia.org/T323150 (10thcipriani) [17:42:49] 10GitLab, 10Release-Engineering-Team (GitLab III: GitLab in LA 🪃): Try DigitalOcean object storage for buildkit caching - https://phabricator.wikimedia.org/T323147 (10thcipriani) [17:43:25] 10GitLab, 10Release-Engineering-Team (GitLab III: GitLab in LA 🪃): Try Reggie for buildkit caching - https://phabricator.wikimedia.org/T323149 (10thcipriani) [17:43:30] 10GitLab, 10Release-Engineering-Team (GitLab III: GitLab in LA 🪃): Try DigitalOcean registry for buildkit caching - https://phabricator.wikimedia.org/T323148 (10thcipriani) [17:47:01] welcome back zeljkof :) [17:47:32] 👋 [17:48:15] 10GitLab (Infrastructure), 10serviceops-collab: ensure Gitlab logs end up in logstash - https://phabricator.wikimedia.org/T322261 (10brennen) [17:48:19] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (GitLab III: GitLab in LA 🪃): Try DigitalOcean registry for buildkit caching - https://phabricator.wikimedia.org/T323148 (10brennen) [17:48:22] 10GitLab (Auth & Access): Set new owner in wmit-wikimedia GitLab group - https://phabricator.wikimedia.org/T323196 (10brennen) [17:48:28] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (GitLab III: GitLab in LA 🪃): Provision Horizontal Pod Autoscaler (HPA) for GitLab cloud runners - https://phabricator.wikimedia.org/T323164 (10brennen) [17:48:45] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (GitLab III: GitLab in LA 🪃): Try DigitalOcean object storage for buildkit caching - https://phabricator.wikimedia.org/T323147 (10brennen) [17:48:50] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (GitLab III: GitLab in LA 🪃): Try Reggie for buildkit caching - https://phabricator.wikimedia.org/T323149 (10brennen) [17:49:17] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (GitLab III: GitLab in LA 🪃): Try local directory export + GitLab cache for buildkit caching - https://phabricator.wikimedia.org/T323150 (10brennen) [17:49:30] 10GitLab (Project Migration), 10Release-Engineering-Team (GitLab III: GitLab in LA 🪃), 10MediaWiki Train Development Environment: Migrate train-dev to GitLab - https://phabricator.wikimedia.org/T323225 (10thcipriani) [17:49:47] 10GitLab (Project Migration), 10Release-Engineering-Team (GitLab III: GitLab in LA 🪃), 10MediaWiki Train Development Environment: Migrate train-dev to GitLab - https://phabricator.wikimedia.org/T323225 (10brennen) [17:49:53] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Priority Backlog 📥), 10serviceops-collab: Build and import new release of jwt-authorizer (1.1.0) - https://phabricator.wikimedia.org/T322691 (10brennen) [17:49:57] 10Gerrit, 10Upstream: Support posting screenshots in Gerrit - https://phabricator.wikimedia.org/T228084 (10hashar) 05Open→03Declined I have asked the upstream authors and the plugin is indeed legacy. It would need a complete rewrite of the UI part and probably some addition to Gerrit UI itself. It was not... [17:50:05] 10GitLab (CI & Job Runners), 10Release-Engineering-Team, 10Data Pipelines, 10Data-Engineering-Planning, 10serviceops-collab: Experiencing pipeline failure due to disk-space issues - https://phabricator.wikimedia.org/T310593 (10brennen) [17:50:42] 10GitLab (CI & Job Runners), 10Release-Engineering-Team, 10serviceops-collab: Migrate GitLab Shared Runners from profile::gitlab::runner to role::gitlab_runner - https://phabricator.wikimedia.org/T322409 (10Jelto) p:05Triage→03Medium a:03Jelto I converted one of the Shared Runners `runner-1030.gitlab-r... [18:01:01] urandom: haha. good to hear :) seriously though, it's a complex system. very easy to miss the signal [18:01:20] i'm just glad it's working [18:01:39] same, and I'll pay more attention next time! :) [19:25:03] 10Release-Engineering-Team (Priority Backlog 📥), 10serviceops, 10Release Pipeline (Blubber): Buildkit erroring with "cannot reuse body, request must be retried" upon multi-platform push - https://phabricator.wikimedia.org/T322453 (10Jelto) [19:25:09] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Priority Backlog 📥), 10serviceops-collab: Build and import new release of jwt-authorizer (1.1.0) - https://phabricator.wikimedia.org/T322691 (10Jelto) 05Open→03Resolved a:03Jelto I built and published the new version `jwt-authorizer | 1.1.0-1` fo... [19:37:04] 10Release-Engineering-Team (GitLab III: GitLab in LA 🪃), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.40.0-wmf.10 deployment blockers - https://phabricator.wikimedia.org/T320515 (10brennen) [19:39:23] 10Release-Engineering-Team (GitLab III: GitLab in LA 🪃), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.40.0-wmf.10 deployment blockers - https://phabricator.wikimedia.org/T320515 (10brennen) Blocking for the moment - see T323184#8400543 for rationale. [20:37:31] 10Continuous-Integration-Config, 10css-sanitizer, 10PHP 8.1 support: Re-enable PHP 8.1 CI on css-sanitizer - https://phabricator.wikimedia.org/T311451 (10Umherirrender) 05Resolved→03Open It seems that is not fixed, because on https://gerrit.wikimedia.org/r/c/css-sanitizer/+/857096 the php8.1 fails with t... [20:53:21] !log restarting jenkins for update [20:53:23] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:58:32] 10Continuous-Integration-Infrastructure, 10Jenkins, 10Security: Jenkins plugins security advisory - 2022-11-15 - https://phabricator.wikimedia.org/T323054 (10thcipriani) 05Open→03Resolved a:03thcipriani > JUnit Plugin 1160.vf1f01a_a_ea_b_7f > Pipeline Utility Steps Plugin 2.13.1 and 2.13.2 > Script Sec... [21:00:22] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (GitLab III: GitLab in LA 🪃): Try local directory export + GitLab cache for buildkit caching - https://phabricator.wikimedia.org/T323150 (10jeena) This was done, but disabled due to poor performance. A summary of results is in this commit message: https:... [21:03:34] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (GitLab III: GitLab in LA 🪃): Provision Horizontal Pod Autoscaler (HPA) for GitLab cloud runners - https://phabricator.wikimedia.org/T323164 (10jeena) a:03jeena