[07:08:47] greetings [07:56:15] morning [08:08:53] andrewbogot.t: created T439512 to follow up on the capacity issues we saw last week [08:08:53] T439512: [o11y,k8s,infra] Current stats used for capacity are not reliable enough - https://phabricator.wikimedia.org/T439512 [13:32:28] dhinus: how do I test the alerts-api on lima-kilo without creating a new tool (or making one error out) in prod? [13:33:50] dcaro: in a meeting, will reply later :) [13:34:08] ack [13:34:34] thx dcaro [13:55:56] some quick reviews https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/379 and https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/9 [14:09:47] dhinus: if it's not already, please make the new toolforge alerting thing use a non-default $anything (see https://wikitech.wikimedia.org/wiki/Help:Toolforge/Email#Mail_to_a_Tool) so that alert destinations can be customised independently of other mails to a tool [14:11:06] dcaro: I was planning to use tools.toolname@toolforge.org which is also listed in that page, but that makes sense, I'll use a different alias [14:11:27] dcaro: sorry that reply was for taavi :) [14:14:15] where do emails from jobs come from? [14:15:05] dcaro: the lima-kilo testing options are a bit limited at the moment, I was thinking of adding a "fake prometheus" http server where we can generate fake alerts for testing... [14:15:48] currently in lima-kilo you can test against the tools prometheus, which has several alerting tools, but those tool names will not exist in lima-kilo [14:16:10] it has hardcoded the tool test-maintainer-alerts as the only one opted in, that's what I was missing [14:17:24] yes at the moment you can not test a full end-to-end flow in lima-kilo that actually tries to send or log something [14:19:19] all the alerts will be ignored as "not opted in", which is not ideal, let me try to find a way to do it [14:19:47] re: emails from jobs ("jobs-emailer"), I checked and they are sent to tools.toolname@toolforge.org, from noreply@toolforge.org [14:20:08] I created a failing job in that tool to make it trigger [14:20:17] https://www.irccloud.com/pastebin/akKj34fe/ [14:20:18] yes that might work [14:20:27] nice [14:31:52] dcaro: I created T439554 today to track removing the hardcoded `test-maintainer-alerts` and reading the actual opted-in tools [14:31:53] T439554: Read the opted-in tools from components-api - https://phabricator.wikimedia.org/T439554 [14:32:12] but my idea was to deploy to tools with the hardcoded name, test it in prod, then remove the hardcoding after [14:32:12] 👍 [14:32:31] both work for me, just needed a way to test it :) [14:33:00] yeah I didn't give much thought to testing in lima-kilo, which is actually a very good idea, especially with your trick of re-using the test-maintainer-alerts name [16:14:18] andrewbogott: do you have a way of easily testing https://gerrit.wikimedia.org/r/c/openstack/horizon/wmf-proxy-dashboard/+/1308167 locally? [16:15:10] I do not, but you can roll it out in codfw1dev first [16:15:16] taavi: I updated https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/8 to use toolname.alerts@toolforge.org instead of tools.toolname@, thanks for suggesting it! I also updated https://wikitech.wikimedia.org/wiki/Help:Toolforge/Alerts [16:15:36] dhinus: neat, thanks! [16:15:46] taavi: well, you can build and run horizon locally but you have to tweak some firewall rules to make it work from your laptop [16:15:54] I think there are instructions... [16:16:46] yeah, embarassing instructions at the bottom of https://gitlab.wikimedia.org/repos/cloud/cloud-vps/horizon/deploy/-/blob/main/README.md?ref_type=heads [16:18:17] hmh [16:24:54] sorry taavi were you asking how to test it or asking me to test it? I can probably do that in the next couple of days if you want. But deploying to codfw1dev is maybe the most for-real way to test, which you can do by merging but then only changing the docker version for cloudweb2002 [16:26:48] I can try to find some time at some point, it's not very urgent. but if you're already poking at Horizon, feel free to do that as well! [16:29:05] I just rolled out a patch, we'll see if I have to take another stab [16:44:26] I added those instructions xd, feel free to change [16:48:41] dcaro: the instructions themselves are embarassing -- just the lack of a better facility than "stop puppet, hand-edit files on three prod servers" [16:48:46] for actual testing :) [16:48:51] *aren't* [16:49:00] I mean -- the instructions are clear and work! [16:49:09] No complaints about the docs, just about the reality [16:50:30] ack :) [17:40:43] * dhinus off