[08:21:42] (03CR) 10Kosta Harlan: "recheck" [integration/quibble] - 10https://gerrit.wikimedia.org/r/866596 (owner: 10Kosta Harlan) [08:34:25] kostajh: ;) [08:35:05] so does it reports anything? [08:59:19] hashar: no :\ not sure what is going on. I suspect it would work fine with e.g. directy testing a core patch or GrowthExperiments patch, because that is what I verified and tested with locally [08:59:41] maybe the env variable is not set ? [08:59:46] but the quibble-fullrun setup is somewhat different: we're depending on a core patch with a quibble.yaml... it should still work, I will debug some [08:59:57] the env variable is set [09:00:26] in the logs for the earlywarningbot app, I now see https://gist.github.com/kostajh/19095a37e77d8bcff670d80c005de6a4 [09:00:43] whereas before the API key was set, I would get an error that the API key didn't match or wasn't provided [09:01:16] so at least messages are emitted aren't they? [09:02:53] also I was wondering about the message `* npm test in /workspace/src" to https://earlywarningbot.toolforge.org` [09:03:00] I just added some debug logs [09:03:16] and I had missed it a log message starting with `quibble.commands:Sending error data for command ` [09:03:30] which then has the command line str enclosed, which in this case has multiline [09:03:35] yeah, my eventual goal would be to provide specific information about what exacty failed in e.g. "npm test" [09:03:53] (03CR) 10Kosta Harlan: "recheck" [integration/quibble] - 10https://gerrit.wikimedia.org/r/866596 (owner: 10Kosta Harlan) [09:04:23] if the linters/test suite reported the error as json/xml that would make it slightly easier to retrieve probably [09:04:50] and theorically we could have the Gerrit javascript plugin to retrieves those from the CI jobs and then display the errors directly in the Gerrit ui [09:04:56] but that is another topic realy [09:05:03] right [09:05:20] you might also be interested in https://wikitech.wikimedia.org/wiki/Tool:Fix_Suggester_Bot for that :) [09:05:58] it works on GrowthExperiments, but has some off-by-one issues that only happen sometimes :\ so I have not promoted it further [09:06:27] but if you have e.g. a phpcs flagged issue in your code, the bot will provide a gerrit fix suggestion for it [09:14:54] 10Quibble, 10Design-Systems-Team, 10MediaWiki-ResourceLoader, 10Performance-Team, 10ci-test-error (WMF-deployed Build Failure): CI failure - TypeError: Cannot read property 'registerCompiler' of undefined - https://phabricator.wikimedia.org/T330314 (10kostajh) [09:17:27] kostajh: you could add a screenshot of that Fix Suggester bot on the wiki page ;D [09:17:43] what I thought is that all those linters do have support to emit json reports [09:17:50] (03PS22) 10Kosta Harlan: CI: Add reporting URL for ci-fullrun [integration/quibble] - 10https://gerrit.wikimedia.org/r/866596 [09:17:55] so we can use some javascript in the Gerrit frontend to retrieve them and expose them as finding [09:18:01] maybe I should giev it a try [09:18:40] the devil is having the linters to write the results to the log dir ( $LOG_DIR ) which would need the linters configs to be adjusted in all repos [09:18:47] anyway, that is a side track really [09:18:53] I think the way to do it is via the bot approach, because you need an "actor" to make the Robot Comments [09:19:57] one way you could do it with Quibble: when a lint job fails, re-run with JSON/XML structured output and POST to the fix suggester bot, and let that make the comment [09:22:54] 10Quibble, 10Design-Systems-Team, 10MediaWiki-ResourceLoader, 10Performance-Team: CI failure - TypeError: Cannot read property 'registerCompiler' of undefined - https://phabricator.wikimedia.org/T330314 (10Krinkle) [09:26:57] 10Quibble, 10Design-Systems-Team, 10MediaWiki-ResourceLoader, 10Performance-Team: CI failure - TypeError: Cannot read property 'registerCompiler' of undefined - https://phabricator.wikimedia.org/T330314 (10kostajh) For the job mentioned in the task description, the Apache error log is empty. I do see this... [09:27:08] ack [09:30:44] 10Quibble, 10Design-Systems-Team, 10MediaWiki-ResourceLoader, 10Performance-Team: CI failure - TypeError: Cannot read property 'registerCompiler' of undefined - https://phabricator.wikimedia.org/T330314 (10Krinkle) Ack, that could be it. Perhaps too much concurrency for the database to handle. Although the... [09:38:16] (03Abandoned) 10Kosta Harlan: CI: Add reporting URL for ci-fullrun [integration/quibble] - 10https://gerrit.wikimedia.org/r/866596 (owner: 10Kosta Harlan) [09:38:19] (03Restored) 10Kosta Harlan: CI: Add reporting URL for ci-fullrun [integration/quibble] - 10https://gerrit.wikimedia.org/r/866596 (owner: 10Kosta Harlan) [09:44:34] 10Quibble, 10Design-Systems-Team, 10MediaWiki-ResourceLoader, 10Performance-Team: CI failure - TypeError: Cannot read property 'registerCompiler' of undefined - https://phabricator.wikimedia.org/T330314 (10hashar) We might be able to have #quibble to spin up MySQL with error logging and attach the resultin... [09:47:31] (03CR) 10Kosta Harlan: "recheck" [integration/quibble] - 10https://gerrit.wikimedia.org/r/866596 (owner: 10Kosta Harlan) [10:04:58] hashar: yeah, the "early warning" comment is not working because of the Depends-On situation. It is trying to use a request URL for "https://gerrit.wikimedia.org/r/a/changes/mediawiki%2Fcore~866596/revisions/22/review", the change ID (866596) belongs to quibble, not to core, and the revision number belongs to quibble, not to core. So it should actually use a project name of [10:04:59] integration%2Fquibble in this case, but it isn't doing so, cause Quibble tells it that core is under test [10:05:21] (03PS23) 10Kosta Harlan: CI: Add reporting URL for ci-fullrun [integration/quibble] - 10https://gerrit.wikimedia.org/r/866596 [10:05:39] (03PS24) 10Kosta Harlan: CI: Add reporting URL for ci-fullrun [integration/quibble] - 10https://gerrit.wikimedia.org/r/866596 [10:10:07] (03Abandoned) 10Kosta Harlan: CI: Add reporting URL for ci-fullrun [integration/quibble] - 10https://gerrit.wikimedia.org/r/866596 (owner: 10Kosta Harlan) [10:11:08] so, tldr I I don't think we can verify this with Quibble fullrun jobs :\ [10:22:33] (or we need to teach quibble how to handle the depends-on issues) [11:25:14] oh, duh, it's because we specifically set ZUUL_PROJECT to "mediawiki/core" for ci-fullrun.sh [11:32:17] (03CR) 10Kosta Harlan: "Because we have hardcoded "ZUUL_PROJECT" as mediawiki/core, the early warning bot can't comment correctly on this patch. I've made I043a2e" [integration/quibble] - 10https://gerrit.wikimedia.org/r/866596 (owner: 10Kosta Harlan) [13:17:19] kostajh: yeah ;) [13:17:44] so there is no way to find out it got triggered from integration/quibble [13:22:14] right. I think https://gerrit.wikimedia.org/r/c/integration/config/+/891514 is the best way to verify this... test in production :) [13:23:20] or you can retrieve the repository based on the `ZUUL_CHANGE` , assuming that is not overriden by the full run script [13:23:31] or yeah test in prod \o/ [13:24:10] deploying it [13:29:57] :D [13:30:21] that has refreshed 207 jobs [13:30:37] which are not all the quibble jobs, we would have to pass the option to all the jjb job templates for that [13:30:51] or maybe add support for reading from a /etc/quibble/config.yaml [13:31:12] ok, I will run a recheck on https://gerrit.wikimedia.org/r/c/mediawiki/core/+/867535 [13:31:59] yeah, I realized there were more jobs to update, but just wanted to pick one to start with for an example to verify with [13:35:28] erm https://integration.wikimedia.org/ci/job/wmf-quibble-core-vendor-mysql-php74-docker/10446/console [13:35:53] it looks like those jobs are using 1.4.7 and not 1.5.0, sorry I missed that [13:38:44] hashar: revert created https://gerrit.wikimedia.org/r/c/integration/config/+/891303 [14:18:18] sorry I was dealing with some other things [14:18:27] hence the lag [14:23:56] no worries, again, sorry I missed that to begin with [14:33:43] the "recheck" on https://gerrit.wikimedia.org/r/c/mediawiki/core/+/867535 doesn't work because the composertest job doesn't have the reporting url set 🤦🏻‍♂️ [14:34:07] so I made https://gerrit.wikimedia.org/r/c/integration/config/+/891559 to enable in a broader set of jobs where I think the feature would be useful [14:38:52] ahh [14:38:57] yeah we would need a global config [14:39:09] I am pretty sure I had a patch for quibble for that [14:40:49] ah yeah https://gerrit.wikimedia.org/r/c/integration/quibble/+/735414 (abandoned) [14:40:55] cause I could not figure out what I wanted to do [14:41:37] I would love a python library that fetches config magically from the user home directory / etc dir (based on XDG env variable) [14:41:41] use ConfigParser [14:41:52] read from the environment variable [14:41:59] and finally build the argparse for us [14:42:15] maybe I should write it `import magic` [14:42:44] anyway, jobs are refreshing [14:42:46] heh [14:50:27] done [14:50:46] thanks. so... getting closer [14:51:06] "Applying label "Verified": -1 is restricted". I am pretty sure that was not the case a few months ago, maybe a Gerrit update changed this? In any case I will test without applying a vote [14:56:48] hey, finally... https://gerrit.wikimedia.org/r/c/mediawiki/core/+/867535/7#message-233a3973c5057d873f422dcd98868831a03a02ad [14:58:40] that permission can be granted [14:59:41] your bot votes Verified -1 / +1 right? [15:00:00] looks like we can use the group LabelBots https://gerrit.wikimedia.org/r/admin/groups/49f1b64d8e3b8552b0487ff4933fdfad426b6388 [15:00:15] which grants right to verify -1/+1 on all projects [15:00:31] it has a few bots https://gerrit.wikimedia.org/r/admin/groups/49f1b64d8e3b8552b0487ff4933fdfad426b6388,members :] [15:07:09] kostajh: I have added kharlan+earlywarningbot@wikimedia.org to the group so the bot should be able to vote on all repos [15:07:15] but NOT Verified +2 [15:08:21] note that in gate and submit, a job might fail and report [15:08:44] but the change can later be rescheduled and moved ahead in the queue and then have all jobs passing [15:08:53] that is the case if a change ahead in the queue caused the issue [15:08:58] we will find out ;) [15:11:28] also a verified-1 will prevent the change from being submitted, so if someone recheck we would need a way to clear out the vote if things got resolved [15:59:13] hmm, good point [16:01:18] you're sure that -1 verified would block a merge? I've +2'ed changes with a -1 from jenkins-bot that I knew were due to a flaky test, and those got merged [16:01:50] (btw I don't like the name "early warning bot", suggestions welcome.) [16:18:51] kostajh: I think that is because Zuul will clear the Verified-1 vote it has previously set [16:19:36] to rephrase, when one does a CR+2, the event enters `gate-and-submit` which is configured to remove the Verified vote [16:20:22] https://gerrit.wikimedia.org/g/integration/config/+/refs/heads/master/zuul/layout.yaml#765 [16:20:24] start: [16:20:25] gerrit: [16:20:31] verified: 0 [16:21:50] then if a new patch is send (or the change is rebased), the Verified votes is always cleared [16:23:00] unlike Code-Review label which is carried to the new patchset on rebase or when solely the commit message got adjusted ( https://gerrit.wikimedia.org/r/c/All-Projects/+/124891 ) [16:23:07] sorry that is a lot of details ;-] [16:23:12] we will see [16:33:16] right [16:33:34] I guess the problem is that if there's a flaky test, the Verified label won't get reset with a recheck or re-apply of +2 :\ [16:36:58] yeah [16:37:20] or you get a series of change A < B, A causing B to fail which triggers the early warning bot [16:37:28] Zuul then drop A cause it failed, reschedules B [16:37:38] then all jobs pass, but the early warning bot is still there ;) [16:37:52] anyway we will see whether it is a problem in practice [16:39:54] I might look at adding the js code to have that bot show up in the Gerrit Checks tab ( my poorly organized code is at https://gerrit.wikimedia.org/g/operations/software/gerrit/+/refs/heads/deploy/wmf/stable-3.5/plugins/wm-checks-api.js ) [16:41:07] anyway, it is a good feature [16:42:37] kostajh: can you make the bot to send the message with a different tag than autogenerated:ci ? Aka: `autogenerated:earlywarning` [16:43:22] it is not so important, will just make the processing of those messages slightly nicer in the code [16:43:48] yes, it's easy to do [16:43:59] there are a bunch of conditionals at https://gerrit.wikimedia.org/g/operations/software/gerrit/+/refs/heads/deploy/wmf/stable-3.5/plugins/wm-checks-api.js#634 [16:44:15] though I could add oone based on the real author name [16:44:20] hashar: https://gitlab.wikimedia.org/kharlan/earlywarningbot/-/blob/main/src/main.rs#L104 [16:44:33] oh it is in Rust haha [16:44:36] I thought autogenerated:ci was desired because that allows you to filter out bot comments easily [16:44:37] <3 [16:45:04] yeah good point, then Gerrit filters out any message that has a tag starting by `autogenerated:` [16:45:12] ah, ok [16:45:14] so you can use whatever suffix to help differentiate [16:45:33] i'm signing off now but can push the code change later, but merge requests are welcome :) [16:45:35] maybe my code should not rely on the tag but solely on the real author instead [16:46:05] for Zuul (jenkins-bot) each Pipeline report with a different tag forged after the Zuul Pipeline (`autogenerated:test`, `autogenerated:gate-and-submit` etc) [16:46:11] will do [16:46:15] have a good evening! [16:52:46] **** *** GitLab [16:53:16] I have pushed to `gerrit-tag` branch on your repo but I can't do a merge request [16:53:17] https://gitlab.wikimedia.org/kharlan/earlywarningbot/-/commits/gerrit-tag [16:53:29] despite: [16:53:31] remote: To create a merge request for gerrit-tag, visit: [16:53:31] remote: https://gitlab.wikimedia.org/kharlan/earlywarningbot/-/merge_requests/new?merge_request%5Bsource_branch%5D=gerrit-tag [16:56:44] so yeah I had to fork [16:56:52] kostajh: good morning! https://gitlab.wikimedia.org/kharlan/earlywarningbot/-/merge_requests/1 [20:37:37] hashar: thanks for the MR, merged! [20:38:56] kostajh: you are welcome :] [20:39:15] and deployed [20:39:49] can I add you to the toolforge project, so it's not just me on there? [20:50:58] for sure [20:51:23] but I will need a tour eventually cause I barely use toolforge (short of restarting a few bots from time to time by copy pasting from the doc on wikitech) [20:51:30] I should learn about it for sure [20:58:54] I am off again [21:36:27] (03PS1) 10Kosta Harlan: reporting: Remove slash from build URL [integration/quibble] - 10https://gerrit.wikimedia.org/r/891653 (https://phabricator.wikimedia.org/T323750)