[07:57:18] 10GitLab: Consider enabling distributed caching for GitLab runners - https://phabricator.wikimedia.org/T328516 (10Chlod) [08:27:10] 10GitLab (Infrastructure), 10Release-Engineering-Team, 10serviceops-collab, 10Security: GitLab Security Release: 15.8.1, 15.7.6, and 15.6.7 - https://phabricator.wikimedia.org/T328518 (10Jelto) [08:27:20] 10GitLab (Infrastructure), 10Release-Engineering-Team, 10serviceops-collab, 10Security: GitLab Security Release: 15.8.1, 15.7.6, and 15.6.7 - https://phabricator.wikimedia.org/T328518 (10Jelto) p:05Triageβ†’03High [08:53:28] 10Gerrit: [wm-checks-api] error: changeMessage.real_author is undefined - https://phabricator.wikimedia.org/T328484 (10hashar) a:03hashar That is definitely on me, the code is not very robust. The plugin process all of the messages and pass each of them to the various robots implementations which then returns... [08:59:21] 10Gerrit: [wm-checks-api] error: changeMessage.real_author is undefined - https://phabricator.wikimedia.org/T328484 (10hashar) The `Change has been successfully merged by jenkins-bot` message does not have a real_author: ` 3: Object { id: "22d93b9b657a158da273f6146f40287c71778637", date: "2016-08-11 00:41:44.000... [09:11:23] review sprint time! [09:25:44] how hard it would be to run https://doc.wikimedia.org/mediawiki-core/master/phpmetrics/ on branches too? [09:27:48] god I love docker, I just pull the image from releng and run it myself [09:35:40] not too hard Amir1 :] [09:35:53] I can't do it today though [09:36:03] no worries [09:36:27] I think the jobs are only triggering on the master branch, there must be a filter in zuul/layout.yaml [09:36:28] something like [09:36:30] - job: [09:36:35] name: '*phpmetric*' [09:36:39] branch: 'master' [09:36:42] I saw in jjb it even has branch: -master, we could add something like REL1_39 but I don't know how koshar is to add hard-coded branches [09:36:55] yeah [09:37:13] then the destinatio on doc.wm.o would have to take in account the branch the job got triggered for [09:37:22] then the above url already has the branch in it [09:37:31] so I think it is just about removing the branch filter from the job [09:37:42] then retriggering it for each of the branches we still support [09:37:55] would that make it run on all branches? That might be a lot of branches [09:37:55] then the idea of that report was for duesen and working on decoupling mediawiki/core [09:38:23] it might not be of much interest for release branches since I think all that works is only for development / master and carries little purpose for the release branches [09:38:25] I'm using it for the refactoring I'm doing on the rdbms library [09:38:51] if you refactor, that is in master isn't it? [09:38:52] so I want to track the progress, I used wayback machine but that's not very sustainable [09:39:09] ah [09:39:59] so rather than having reports per branches, you are looking for historical data. Maybe it is easier to generate those locally [09:40:40] yeah, already doing so [09:41:00] $ docker run -v /home/amir/core:/src -v /home/amir/core/metrics:/phpmetrics-report 18c84fa00f25 --report-html=/phpmetrics-report --report-violations=/log/violations-phpmetrics.xml /src [09:43:07] Heyy, we're currently blocked on deploying to the WDQS GUI and the Wikidata Query Builder, because of that git "dubious ownership" thingy. When you have time, could you have a look on whether the solution I proposed at https://gerrit.wikimedia.org/r/c/integration/config/+/885378 (T328441) is any good? [09:43:08] T328441: git fatal: detected dubious ownership in repository at '/srv/jenkins/workspace/wikidata-query-builder-build/src' - https://phabricator.wikimedia.org/T328441 [10:03:21] MichaelG_WMDE: I will update the jobs [10:04:01] that fix is wrong iin some regard since it would allow someone to inject some git config settings which would end up being executed on the Jenkins agent as the jenkins user [10:04:14] the real fix would be to run all of those commands in a container [10:04:19] but that is arguably a lot more work :D [10:09:29] (03CR) 10Hashar: "Well done! That is a corner case in the stack for sure (see inline comment for a few details)." [integration/config] - 10https://gerrit.wikimedia.org/r/885378 (https://phabricator.wikimedia.org/T328441) (owner: 10Michael Große) [10:09:42] MichaelG_WMDE: I have updated both jobs. May you trigger a build and verify? [10:09:59] I have left a comment that those build commands should probably be moved out of the jenkins job to the wikidata repos [10:10:42] Thank you! I will check right away! [10:23:42] @hashar It worked, thank you! [10:24:52] 10GitLab, 10serviceops-collab: Investigate incremental backups for GitLab - https://phabricator.wikimedia.org/T324506 (10Jelto) >>! In T324506#8575734, @Arnoldokoth wrote: > Yes, you are absolutely right about this. I performed the incremental backup on gitlab1003 and it pretty much maxed out the disk usage. T... [10:25:04] I'll create a follow-up task for us to move those jenkins shell-scripts into some script in our respective repositories [10:47:18] MichaelG_WMDE: thanks ;) [11:05:44] 10Continuous-Integration-Config, 10Wikidata, 10Wikidata Query Builder, 10Wikidata Query UI, and 2 others: Move build scripts from CI to the repository - https://phabricator.wikimedia.org/T328543 (10Michael) [11:11:54] 10Continuous-Integration-Config, 10Wikidata, 10Wikidata Query Builder, 10Wikidata Query UI, and 3 others: Move build scripts from CI to the repository - https://phabricator.wikimedia.org/T328543 (10Michael) [11:14:21] 10Continuous-Integration-Config, 10Wikidata, 10Wikidata Query Builder, 10Wikidata Query UI, and 3 others: git fatal: detected dubious ownership in repository at '/srv/jenkins/workspace/wikidata-query-builder-build/src' - https://phabricator.wikimedia.org/T328441 (10Michael) 05Openβ†’03Resolved a:03Micha... [11:15:48] (03Abandoned) 10Michael Große: Allow our own repo as safe to git [integration/config] - 10https://gerrit.wikimedia.org/r/885378 (https://phabricator.wikimedia.org/T328441) (owner: 10Michael Große) [11:20:45] 10GitLab: Consider enabling distributed caching for GitLab runners - https://phabricator.wikimedia.org/T328516 (10Chlod) [11:23:26] FTR: the follow-up task: T328543 [11:23:27] T328543: Move build scripts from CI to the repository - https://phabricator.wikimedia.org/T328543 [11:52:04] 10Continuous-Integration-Config, 10Wikidata, 10Wikidata Query Builder, 10Wikidata Query UI, and 3 others: Move build scripts from CI to the repository - https://phabricator.wikimedia.org/T328543 (10Lucas_Werkmeister_WMDE) [12:05:04] 10GitLab (Infrastructure), 10serviceops-collab: ensure Gitlab logs end up in logstash - https://phabricator.wikimedia.org/T322261 (10eoghan) So it looks like gitlab logs are already configured to be included in logstash, someone beat us to it in 2021! They're configured here ([[ https://gerrit.wikimedia.org/r/... [12:30:48] 10Release-Engineering-Team (Radar), 10Scap, 10MediaWiki-Internationalization, 10Performance-Team, 10Patch-For-Review: Use static php array files for l10n cache at WMF (instead of CDB) - https://phabricator.wikimedia.org/T99740 (10Ladsgroup) I can't find it but I had it somewhere that serviceops mentioned... [12:45:07] 10GitLab (Infrastructure), 10serviceops-collab: ensure Gitlab logs end up in logstash - https://phabricator.wikimedia.org/T322261 (10Jelto) It seems logstash integration has been implemented and tracked in T274462 already. Somehow it stopped working. The change https://gerrit.wikimedia.org/r/c/operations/pupp... [14:57:40] 10GitLab, 10Release-Engineering-Team, 10serviceops-collab, 10Patch-For-Review: Align the GitLab runner tags - https://phabricator.wikimedia.org/T325069 (10Jelto) 05Openβ†’03Resolved So the last tag which isn't settled yet is `cloud`. I don't see big benefits in migrating from tag `cloud` to `public-cloud... [15:50:59] 10GitLab (CI & Job Runners), 10Release-Engineering-Team: Consider enabling distributed caching for GitLab runners - https://phabricator.wikimedia.org/T328516 (10brennen) [17:34:02] 10Gerrit, 10Release-Engineering-Team, 10Patch-For-Review: Remove plugins/.eslintrc.json - https://phabricator.wikimedia.org/T328134 (10hashar) p:05Triageβ†’03Low [17:34:35] 10Gerrit, 10Release-Engineering-Team (Seen), 10Patch-For-Review: Remove plugins/.eslintrc.json - https://phabricator.wikimedia.org/T328134 (10hashar) [17:35:40] 10Continuous-Integration-Config, 10Release-Engineering-Team: mediawiki/libs/metrics-platform CI jobs files filters have no match for some change: no CI job running - https://phabricator.wikimedia.org/T327301 (10hashar) 05Openβ†’03Resolved a:03hashar That got fixed by updating the jobs files filters in Zuul... [17:35:59] 10Release-Engineering-Team, 10Scap: Scap fails on debian bullseye targets - https://phabricator.wikimedia.org/T326668 (10jnuche) a:05dancyβ†’03jnuche [17:37:47] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Radar), 10SRE, 10Patch-For-Review: git: detected dubious ownership in repository at '/srv/mediawiki-staging' - https://phabricator.wikimedia.org/T325128 (10hashar) [17:38:50] 10GitLab (CI & Job Runners), 10Release-Engineering-Team: Consider enabling distributed caching for GitLab runners - https://phabricator.wikimedia.org/T328516 (10thcipriani) [17:38:53] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Priority Backlog πŸ“₯), 10serviceops-collab, 10User-brennen: Provision untrusted instance-wide GitLab job runners to handle user-level projects and merge requests from forks - https://phabricator.wikimedia.org/T297426 (10thcipriani) [17:39:39] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Priority Backlog πŸ“₯): Consider enabling distributed caching for GitLab runners - https://phabricator.wikimedia.org/T328516 (10thcipriani) We're working on moving shared runners to a space where we have this enabled, so the hope is it will Just Workβ„’ once... [17:40:08] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Radar), 10serviceops-collab: Cleanup and attach volumes for gitlab-runners WMCS project - https://phabricator.wikimedia.org/T328283 (10thcipriani) [17:41:49] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Blocking 🧱), 10CirrusSearch, 10Discovery-Search: Consider adding "official" docker.elastic.co images to the list of allowed images for gitlab runners - https://phabricator.wikimedia.org/T327519 (10thcipriani) a:03brennen [17:42:37] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Blocking 🧱), 10mwcli: Add registry.gitlab.com/dependabot-gitlab/dependabot to list of allowed images for gitlab runners - https://phabricator.wikimedia.org/T326507 (10thcipriani) a:03brennen [17:44:40] 10Release-Engineering-Team: Kokkuri should allow dockerfile.v0 frontend - https://phabricator.wikimedia.org/T326569 (10thcipriani) [17:44:52] 10Release-Engineering-Team, 10Scap: `scap backport` should rebase patches to the same repository on top of each other - https://phabricator.wikimedia.org/T326175 (10hashar) If you have three pending changes A, B, C and Code-Review +2 them in that order, our CI test them as if the previous ones already got merg... [17:45:28] 10Release-Engineering-Team: Runner runner-1030.gitlab-runners.eqiad1.wikimedia.cloud out of space - https://phabricator.wikimedia.org/T326558 (10thcipriani) [17:45:30] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Radar), 10serviceops-collab: Cleanup and attach volumes for gitlab-runners WMCS project - https://phabricator.wikimedia.org/T328283 (10thcipriani) [17:45:39] 10Release-Engineering-Team (Seen): Runner runner-1030.gitlab-runners.eqiad1.wikimedia.cloud out of space - https://phabricator.wikimedia.org/T326558 (10thcipriani) [17:46:38] 10GitLab, 10Release-Engineering-Team, 10MediaWiki-extensions-Gadgets, 10Security-Team, 10Security: Allow Javascript files from Wikimedia GitLab to be loaded as scripts in Wikimedia wikis - https://phabricator.wikimedia.org/T321458 (10brennen) [17:50:19] 10GitLab (Administration, Settings & Policy), 10Release-Engineering-Team, 10Upstream: Gitlab make public the default fork option - https://phabricator.wikimedia.org/T324016 (10brennen) p:05Triageβ†’03Low I haven't looked but I'm guessing this is a question for upstream. [17:51:31] 10GitLab (Administration, Settings & Policy), 10Release-Engineering-Team (Seen), 10Upstream: Gitlab make public the default fork option - https://phabricator.wikimedia.org/T324016 (10brennen) [17:52:03] 10Gerrit, 10Release-Engineering-Team (Seen), 10Scap: `scap backport` should rebase patches to the same repository on top of each other - https://phabricator.wikimedia.org/T326175 (10thcipriani) p:05Triageβ†’03Low [17:52:34] 10Continuous-Integration-Config, 10Release-Engineering-Team (Seen), 10SRE, 10serviceops: operations/docker-images/production-images has no CI - https://phabricator.wikimedia.org/T283855 (10hashar) [17:52:40] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Radar): Move doc.wikimedia.org storage to a distributed storage - https://phabricator.wikimedia.org/T310617 (10hashar) [17:53:25] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Radar): Move doc.wikimedia.org storage to a distributed storage - https://phabricator.wikimedia.org/T310617 (10hashar) 05Openβ†’03Stalled This is merely a place holder pending the realization of the //[[ https://docs.google.com/document/d/1... [17:54:35] 10GitLab (Project Migration), 10Release-Engineering-Team (Radar), 10serviceops-collab: Create new GitLab project group: wm-juniors-il - https://phabricator.wikimedia.org/T313750 (10brennen) [18:35:15] 10GitLab (Project Migration), 10Release-Engineering-Team (Radar), 10serviceops-collab: Create new GitLab project group: wm-juniors-il - https://phabricator.wikimedia.org/T313750 (10Dzahn) 05In progressβ†’03Stalled setting to stalled due to lack of response [19:13:07] 10Phabricator (Upstream), 10Release-Engineering-Team (Seen), 10serviceops-collab, 10Upstream: Consider migrating from Phabricator to Phorge once that new upstream is ready for such a migration - https://phabricator.wikimedia.org/T302528 (10Dzahn) We have considered it and have plans to create a phorge inst... [19:14:08] 10Phabricator (Upstream), 10Release-Engineering-Team (Seen), 10serviceops-collab, 10Upstream: Consider migrating from Phabricator to Phorge once that new upstream is ready for such a migration - https://phabricator.wikimedia.org/T302528 (10Dzahn) 05Openβ†’03Resolved a:03Dzahn Also https://we.phorge.it/... [19:19:40] 10Phabricator (Upstream), 10Release-Engineering-Team (Seen), 10serviceops-collab, 10Upstream: Consider migrating from Phabricator to Phorge once that new upstream is ready for such a migration - https://phabricator.wikimedia.org/T302528 (10Aklapper) [19:44:49] 10Release-Engineering-Team (GitLab IV: Mise En Place 🍱), 10Patch-For-Review, 10Release, 10Train Deployments: 1.40.0-wmf.21 deployment blockers - https://phabricator.wikimedia.org/T325584 (10dancy) [21:53:11] 10Continuous-Integration-Config, 10Release-Engineering-Team, 10Fundraising-Backlog, 10MediaWiki-extensions-DonationInterface, and 2 others: Update CI config for new fundraising/REL1_39 branch of mediawiki/core under php7.4 - https://phabricator.wikimedia.org/T326372 (10Ejegg) Thanks again, @Jdforrester-WMF... [21:53:23] 10Continuous-Integration-Config, 10Release-Engineering-Team, 10Fundraising-Backlog, 10MediaWiki-extensions-DonationInterface, and 2 others: Update CI config for new fundraising/REL1_39 branch of mediawiki/core under php7.4 - https://phabricator.wikimedia.org/T326372 (10Ejegg) 05Openβ†’03Resolved [22:11:07] 10GitLab (Infrastructure), 10serviceops-collab: Migrate gitlab-test instance to bullseye - https://phabricator.wikimedia.org/T318521 (10Dzahn) >>! In T318521#8552699, @hashar wrote: > Puppet fails on the instance `gitlab-prod-1002`, from today email: This is fixed now. Puppet run does not fail any longer: No... [22:57:38] I just noticed that the logstash error log checking happens before the changes is synced to k8s canary, I guess it doesn't really matter currently, but isn't it the wrong order in general [23:03:38] 10Release-Engineering-Team (GitLab IV: Mise En Place 🍱), 10Patch-For-Review, 10Release, 10Train Deployments: 1.40.0-wmf.21 deployment blockers - https://phabricator.wikimedia.org/T325584 (10Zabe) [23:23:29] 10Release-Engineering-Team (GitLab IV: Mise En Place 🍱), 10Patch-For-Review, 10Release, 10Train Deployments: 1.40.0-wmf.21 deployment blockers - https://phabricator.wikimedia.org/T325584 (10dancy) All is well.