[00:00:06] "This job does not run automatically and must be started manually, but you do not have access to it." -- not fancy enough to fix things apparently (even with gitlab admin rights) [00:00:33] oh well, everyone has limits [00:06:59] urandom: more confusing things -- https://gitlab.wikimedia.org/repos/sre/data-gateway/-/runners/1484 -- that sure looks like the gitlab ui thinks you have access to a trusted runner [00:10:20] hrmm [00:10:34] curiouser and curiouser [02:26:32] (03open) 10matmarex: Draft: OAuth: Authorize in a popup without reloading the page [repos/ci-tools/patchdemo] - 10https://gitlab.wikimedia.org/repos/ci-tools/patchdemo/-/merge_requests/615 [06:44:15] any chance I could have someone look at https://phabricator.wikimedia.org/T348655 ? [07:39:36] 10GitLab (Infrastructure), 06collaboration-services, 13Patch-For-Review: Create a custom GitLab Prometheus exporter - https://phabricator.wikimedia.org/T354656#9798447 (10Jelto) [07:59:59] (03update) 10brouberol: Update projects.json [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/73 (https://phabricator.wikimedia.org/T364795) [08:00:33] (03update) 10brouberol: Update projects.json [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/73 (https://phabricator.wikimedia.org/T364795) [08:14:34] 10GitLab, 10Phabricator, 10Toolforge: Look for ways to consolidate "we trust this human" access lists - https://phabricator.wikimedia.org/T364516#9798495 (10hashar) I am removing #gerrit and #continuous-integration-config since this task is not immediate actionable for any of those projects and that requires... [08:45:51] 10Phabricator, 06collaboration-services: QUESTION: Transition to Phorge? - https://phabricator.wikimedia.org/T364908#9798604 (10Aklapper) Hi, please bring up questions on https://www.mediawiki.org/wiki/Talk:Phabricator/Help as linked from the frontpage - thanks! [09:28:29] (03CR) 10Hashar: [C:03+2] zuul: [mediawiki/extensions/MediaWikiChat] Add phan dependencies for MWC [integration/config] - 10https://gerrit.wikimedia.org/r/1031173 (owner: 10Jack Phoenix) [09:30:10] (03Merged) 10jenkins-bot: zuul: [mediawiki/extensions/MediaWikiChat] Add phan dependencies for MWC [integration/config] - 10https://gerrit.wikimedia.org/r/1031173 (owner: 10Jack Phoenix) [09:34:51] (03CR) 10Hashar: [C:03+2] Translate: Add phan dependency on Scribunto [integration/config] - 10https://gerrit.wikimedia.org/r/1031178 (https://phabricator.wikimedia.org/T359918) (owner: 10Abijeet Patro) [09:36:25] (03Merged) 10jenkins-bot: Translate: Add phan dependency on Scribunto [integration/config] - 10https://gerrit.wikimedia.org/r/1031178 (https://phabricator.wikimedia.org/T359918) (owner: 10Abijeet Patro) [09:40:06] (03CR) 10Hashar: [C:03+2] "Deployed!" [integration/config] - 10https://gerrit.wikimedia.org/r/1031178 (https://phabricator.wikimedia.org/T359918) (owner: 10Abijeet Patro) [11:04:44] 10Continuous-Integration-Config: Selenium fails with "waiting for container: unexpected EOF" - https://phabricator.wikimedia.org/T364979 (10Tgr) 03NEW [11:34:27] (03open) 10btullis: Add repos/data-engineering/kubernetes/csi to the trusted-runners [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/75 (https://phabricator.wikimedia.org/T327259 https://phabricator.wikimedia.org/T364472) [12:11:12] (03PS1) 10Zoranzoki21: Review access change [extensions/UnifiedTaskOverview] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/1031855 [12:11:21] (03CR) 10Zoranzoki21: [V:03+2 C:03+2] "Oops, wrong button" [extensions/UnifiedTaskOverview] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/1031855 (owner: 10Zoranzoki21) [12:27:07] Project mediawiki-core-doxygen build #1056: 04FAILURE in 9 min 4 sec: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen/1056/ [12:29:44] !log integration: sudo cumin --force '*' 'rm -f /etc/apt/sources.list.d/openstack-zed*' [12:37:15] Yippee, build fixed! [12:37:15] Project mediawiki-core-doxygen build #1057: 09FIXED in 10 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen/1057/ [12:47:50] (03approved) 10joal: Update plugins and dependencies to latest versions. [repos/ci-tools/wmf-jvm-parent-pom] - 10https://gitlab.wikimedia.org/repos/ci-tools/wmf-jvm-parent-pom/-/merge_requests/11 (owner: 10gehel) [13:12:33] (03open) 10aklapper: Remove downstream changes in AphrontFileResponse.php [repos/phabricator/phabricator] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/phabricator/-/merge_requests/51 (https://phabricator.wikimedia.org/T364720) [13:14:22] */clear [13:23:09] (03update) 10esanders: Major config refactor [repos/ci-tools/banana-checker] (configfile) - 10https://gitlab.wikimedia.org/repos/ci-tools/banana-checker/-/merge_requests/16 (https://phabricator.wikimedia.org/T364843) [13:23:35] (03open) 10esanders: Improve formatting of error messages [repos/ci-tools/banana-checker] (config) - 10https://gitlab.wikimedia.org/repos/ci-tools/banana-checker/-/merge_requests/17 [13:24:25] (03PS1) 10Arthur taylor: [WIP] Add parallel execution for PHPUnit extensions suite [integration/quibble] - 10https://gerrit.wikimedia.org/r/1031903 (https://phabricator.wikimedia.org/T361190) [13:30:27] (03merge) 10gehel: Update plugins and dependencies to latest versions. [repos/ci-tools/wmf-jvm-parent-pom] - 10https://gitlab.wikimedia.org/repos/ci-tools/wmf-jvm-parent-pom/-/merge_requests/11 [13:33:56] Not sure if anyone saw the back-scroll, but I'm hoping someone can point me at how to fix https://gitlab.wikimedia.org/repos/sre/data-gateway/-/jobs/261836 (a first run of build-and-publish-production-image that is stuck) [13:48:38] (03update) 10esanders: Improve formatting of error messages [repos/ci-tools/banana-checker] (config) - 10https://gitlab.wikimedia.org/repos/ci-tools/banana-checker/-/merge_requests/17 [14:20:44] (03update) 10esanders: Improve formatting of error messages [repos/ci-tools/banana-checker] (config) - 10https://gitlab.wikimedia.org/repos/ci-tools/banana-checker/-/merge_requests/17 [14:30:52] !log deployment-prep: armed keyholder on deployment-cumin (credentials are in the deployment-puppetserver-1 private git repo) [14:30:54] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:31:17] !log deployment-prep: sudo cumin --force '*' 'rm -f /etc/apt/sources.list.d/openstack-zed*' [14:31:18] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:50:16] (03merge) 10dancy: Add repos/data-engineering/kubernetes/csi to the trusted-runners [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/75 (https://phabricator.wikimedia.org/T327259 https://phabricator.wikimedia.org/T364472) (owner: 10btullis) [15:13:18] (03update) 10esanders: Improve formatting of error messages [repos/ci-tools/banana-checker] (config) - 10https://gitlab.wikimedia.org/repos/ci-tools/banana-checker/-/merge_requests/17 [15:14:36] (03update) 10esanders: Major config refactor [repos/ci-tools/banana-checker] (configfile) - 10https://gitlab.wikimedia.org/repos/ci-tools/banana-checker/-/merge_requests/16 (https://phabricator.wikimedia.org/T364843) [15:23:02] urandom: Taking a look [15:24:07] (03update) 10esanders: Major config refactor [repos/ci-tools/banana-checker] (configfile) - 10https://gitlab.wikimedia.org/repos/ci-tools/banana-checker/-/merge_requests/16 (https://phabricator.wikimedia.org/T364843) [15:24:46] 10GitLab (Pipeline Services Migration🐤), 10Data-Engineering (Q4 2024 April 1st - June 30th), 10Event-Platform: Migrate Data Engineering Pipelinelib repos to GitLab - https://phabricator.wikimedia.org/T344730#9800601 (10Ottomata) Nice! [15:24:52] (03update) 10esanders: Improve formatting of error messages [repos/ci-tools/banana-checker] (config) - 10https://gitlab.wikimedia.org/repos/ci-tools/banana-checker/-/merge_requests/17 [15:24:53] urandom: https://gitlab.wikimedia.org/repos/sre/data-gateway/-/jobs/261836 was initiated before https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/74 was merged. That's why it didn't work then, but it should work now. [15:25:34] urandom: oh wait.. that's not right. I didn't look carefully at the date... rechecking... [15:26:33] urandom: Give it another try anyway please. [15:28:42] dancy: ok, I have some changes coming, let me couple them to those and try again, rather than a noop. I'll try again in a bit [15:29:59] 10Continuous-Integration-Config, 06Growth-Team, 10MediaWiki-extensions-OATHAuth, 10StructuredDiscussions, and 3 others: PHP 8.2 CI for OATHAuth on 1.42 fails due to missing pimple/container for Flow... - https://phabricator.wikimedia.org/T364986#9800681 (10Lucas_Werkmeister_WMDE) PHP 8.2 seems to be workin... [15:36:08] (03update) 10esanders: Major config refactor [repos/ci-tools/banana-checker] (configfile) - 10https://gitlab.wikimedia.org/repos/ci-tools/banana-checker/-/merge_requests/16 (https://phabricator.wikimedia.org/T364843) [15:36:22] (03update) 10esanders: Improve formatting of error messages [repos/ci-tools/banana-checker] (config) - 10https://gitlab.wikimedia.org/repos/ci-tools/banana-checker/-/merge_requests/17 [15:40:31] James_F: Any thoughts about https://gerrit.wikimedia.org/r/c/integration/config/+/1028900 ? [15:41:53] dancy: It looks reasonable. Sorry, been busy with lots of meetings so not had time for CI stuff yet. Deploy away, or I can fiddle on maybe Friday? [15:42:57] OK. I'll probably deploy today. [15:43:01] +1 [15:47:40] Project mediawiki-core-doxygen build #1062: 04FAILURE in 2 min 10 sec: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen/1062/ [15:49:36] ^ 15:47:40 stderr: fatal: unable to access 'https://gerrit.wikimedia.org/r/p/mediawiki/core.git/': Failed to connect to gerrit.wikimedia.org port 443: Connection timed out [15:50:17] Project beta-code-update-eqiad build #495847: 04FAILURE in 7 min 16 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/495847/ [15:50:51] hashar: Shall we restart Gerrit? [15:51:03] why would you restart it? [15:51:29] that is a https query which is fronted by Apache2 [15:51:38] so I guess something is hammering the web server in front of gerrt [15:51:53] https://grafana.wikimedia.org/d/L0-l1o0Mz/apache?orgId=1&refresh=1m&var-host=gerrit1003&var-port=9117&from=now-1h&to=now&viewPanel=2 [15:51:55] yeah [15:52:11] there is discussion of this on -operations btw [15:52:24] the thread pool has a baseline of ~ 150 workers which I think are the default [15:52:36] and it had bunch of `read` threads [15:52:46] * hashar file a task [15:54:05] 10Gerrit, 06Release-Engineering-Team: Gerrit not reachable over HTTPS - https://phabricator.wikimedia.org/T365041 (10hashar) 03NEW [15:54:21] 10Gerrit, 06Release-Engineering-Team: Gerrit not reachable over HTTPS - https://phabricator.wikimedia.org/T365041#9800848 (10hashar) p:05Triage→03Unbreak! [15:55:10] Project beta-code-update-eqiad build #495848: 04STILL FAILING in 2 min 9 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/495848/ [15:56:04] hashar: did you do anything to get it back? [15:56:17] I was also about to try restarting it but did not..after I saw this here [15:56:26] it works again [15:57:34] 10Gerrit, 06Release-Engineering-Team: Gerrit not reachable over HTTPS - https://phabricator.wikimedia.org/T365041#9800861 (10hashar) I could not spot much on Gerrit but then it is fronted by Apache 2. The threads scoreboard looks bad at https://grafana.wikimedia.org/d/L0-l1o0Mz/apache : {F53346716 size=full} [15:57:36] as sukhe points out the "last puppet run" on that host is wrong in motd [15:57:47] it claims the last run was April 30.. which makes no sense [15:57:51] because puppet runs [15:58:40] I will follow up in -operations [15:58:44] and update the task with the details [15:59:33] ack [15:59:45] the usual problem is that we have too many channels [16:00:12] I started a thread in Slack [16:00:13] j/k [16:01:46] dancy: I just answered a thread about it on the engineering slack x) [16:02:20] haha [16:02:35] you beat me claime I was just coming over here to find the task :) [16:03:21] omg [16:05:22] Yippee, build fixed! [16:05:22] Project beta-code-update-eqiad build #495849: 09FIXED in 2 min 21 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/495849/ [16:06:33] 10Gerrit, 06Release-Engineering-Team: Gerrit not reachable over HTTPS - https://phabricator.wikimedia.org/T365041#9800910 (10hashar) 05Open→03Resolved a:03hashar I have looked at the Apache log, that was a short burst of too many requests . Not much to worry about beside that exhausted the Apache thr... [16:07:17] dancy: I guess I should have paired the investigation with you :D [16:07:35] Team meeting started! [16:08:08] but in short https://grafana.wikimedia.org/d/L0-l1o0Mz/apache [16:09:21] and Apache ECS log https://logstash.wikimedia.org/app/dashboards#/view/825c5c80-8aef-11eb-8ab2-63c7f3b019fc [16:09:26] that applies for both Gerrit and Phabricator [16:26:44] Yippee, build fixed! [16:26:44] Project mediawiki-core-doxygen build #1063: 09FIXED in 8 min 41 sec: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen/1063/ [16:30:07] 10Phabricator (phabricator-next), 10Wikimedia-Phabricator-Extensions, 07I18n: Phabricator translate export script should not overwrite manual edits to qqq.json - https://phabricator.wikimedia.org/T351363#9801015 (10Aklapper) [16:32:37] 10Phabricator (phabricator-next), 10Wikimedia-Phabricator-Extensions, 07I18n: Phabricator translate export script should not overwrite manual edits to qqq.json - https://phabricator.wikimedia.org/T351363#9801020 (10Pppery) 05Open→03Resolved (this doesn't need to be deployed to be considered resolved... [16:45:11] 10GitLab, 06Diffusion-Repository-Administrators, 10Projects-Cleanup, 10Wikimedia Design Style Guide, and 2 others: Archive Design Style Guide code bases / project / docs - https://phabricator.wikimedia.org/T360362#9801112 (10Aklapper) [16:51:56] 10Scap, 06serviceops-radar, 06SRE, 13Patch-For-Review: Confusing failed httpbb check for totoro.wikimedia.org during scap deployment - https://phabricator.wikimedia.org/T364880#9801162 (10hashar) I had a similar issue while deploying the train this morning. One of the httpbb test failed due to mwdebug2002... [17:27:42] 10Release-Engineering-Team (Priority Backlog 📥): Get GitLab to render `T{\d}+` in MR overviews, comments, etc. as links to Phabricator - https://phabricator.wikimedia.org/T337570#9801328 (10dduvall) 05Stalled→03Resolved [18:46:03] dancy: I tried again. It's stuck again. https://gitlab.wikimedia.org/repos/sre/data-gateway/-/jobs/262001 [18:46:12] Taking a look [19:01:28] for blubber, the "lives.in" value is the working directory, right? [19:01:38] a la https://gitlab.wikimedia.org/repos/releng/blubber/-/blob/main/examples/05-copying-from-other-variants.feature#L18 ? [19:05:54] inflatador: yea, that seems right. per "we define our working directory" found in some tutorial [19:07:56] urandom: The issue is that the tag needs to be protected too. The easiest way is to protect all tags by using the `*` wildcard in the repo configuration: https://gitlab.wikimedia.org/repos/sre/data-gateway/-/settings/repository [19:07:57] _After_ you do that, the next tag you create should get through the workflow. [19:08:53] dancy: ooooh [19:09:00] that makes sense [19:09:48] hmmm...I'm having some weirdness with virtualenv, but it could also be escaping [19:10:27] We _intend_ for this expression: `if: $CI_COMMIT_TAG && $CI_COMMIT_REF_PROTECTED` to mean only run this job for protected tags... but what's happening is that $CI_COMMIT_TAG is true (the name of the tag) and $CI_COMMIT_REF_PROTECTED is true because the commit is the head of a protected branch. [19:12:50] dancy: it worked! [19:13:12] yay!! Sorry it took so long for me to figure out. I went down a completely wrong rabbithole. [19:15:05] no worries, thanks for your help! [19:28:17] figured out the venv thing btw, was an escaping problem [19:42:05] * bd808 tries to remember the solution to urandom's trusted runner mystery for the next time this pops up [19:43:37] yeah, I guess this should be documented somewhere? [19:43:45] or maybe it is...? [19:46:54] (03open) 10dancy: Simplify logic in PythonConfig.InstructionsForPhase() [repos/releng/blubber] - 10https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/80 [19:46:56] (03update) 10dancy: Simplify logic in PythonConfig.InstructionsForPhase() [repos/releng/blubber] - 10https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/80 [19:47:38] https://www.mediawiki.org/wiki/GitLab/Workflows/Deploying_services_to_production#Trusted_runners and https://wikitech.wikimedia.org/wiki/GitLab/Gitlab_Runner/Trusted_Runners would be reasonable places to add troubleshooting tips I think [19:54:15] I'll add a note about this recent problem to https://wikitech.wikimedia.org/wiki/GitLab/Gitlab_Runner/Trusted_Runners [19:59:56] 10GitLab: GitLab Private Repository Request for: - https://phabricator.wikimedia.org/T365061 (10Scootcoffeeteahouse) 03NEW [20:03:37] 10GitLab: GitLab Private Repository Request for: - https://phabricator.wikimedia.org/T365061#9801809 (10Dzahn) 05Open→03Invalid [20:14:16] bd808: https://wikitech.wikimedia.org/wiki/GitLab/Gitlab_Runner/Trusted_Runners updated [20:16:35] awesome. thanks dancy [20:22:54] (03update) 10dancy: Simplify logic in PythonConfig.InstructionsForPhase() [repos/releng/blubber] - 10https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/80 [20:22:57] (03update) 10dancy: Simplify logic in PythonConfig.InstructionsForPhase() [repos/releng/blubber] - 10https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/80 [20:23:34] (03update) 10dancy: Simplify logic in PythonConfig.InstructionsForPhase() [repos/releng/blubber] - 10https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/80 [20:23:39] (03update) 10dancy: Simplify logic in PythonConfig.InstructionsForPhase() [repos/releng/blubber] - 10https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/80 [20:24:08] (03update) 10dancy: Simplify logic in PythonConfig.InstructionsForPhase() [repos/releng/blubber] - 10https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/80 [20:32:43] 10GitLab, 10Phabricator, 10Toolforge: Look for ways to consolidate "we trust this human" access lists - https://phabricator.wikimedia.org/T364516#9801906 (10bd808) >>! In T364516#9798495, @hashar wrote: > I am removing #gerrit and #continuous-integration-config since this task is not immediate actionable for... [20:36:43] (03update) 10dancy: Simplify logic in PythonConfig.InstructionsForPhase() [repos/releng/blubber] - 10https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/80 [22:02:08] 10Gerrit: Can't use search query that matches gerrit'ssearch operators - https://phabricator.wikimedia.org/T365076 (10Daimona) 03NEW [22:02:22] 10Gerrit: Can't use search query that matches gerrit's search operators - https://phabricator.wikimedia.org/T365076#9802231 (10Daimona) [22:04:51] 10GitLab (Integrations), 10preview-environment: Spike: Research hosts for preview environment - https://phabricator.wikimedia.org/T283894#9802241 (10thcipriani) 05Open→03Declined Not actively working on this, not slated to be worked on. If this is something we plan on working on in future, we can creat...