[11:47:37] I'm going to disable puppet von WMCS GitLab Runners for ~2h to test https://gerrit.wikimedia.org/r/q/I5e60f33ea97e60b71d1b4ae385a08a35c1dc5e3b [18:40:47] if there was monitoring for gitlab.wikimedia and something happened.. and bots started to talk about it on this channel..here in -gitlab. then would you think "oh, cool, that's good to know and it belongs to the topic of this channel and where interested people are" or would you think more like "eh, sorry, this is a channel for talking, the convention is that bots and alerting is in [18:40:53] -operations. please don't add spammy noisy bots over here randomly" [18:45:19] hmmmm [18:45:45] i think if it wasn't _too_ spammy it might be good in here? [18:45:56] just because things have a way of getting lost in the noise of -operations [18:46:40] (i am very open to other opinions about this) [18:48:29] it's kind of an open survey for now. appreciate the input and will just wait a bit for more :) [18:49:07] I just commented on a related Gerrit change how alerting for gitlab or vrts could be on their "topic" channels #wikimedia-gitlab , #wikimedia-vrt OR on -operations OR both [18:49:42] this is because on a team-based channel, like -serviceops , people said to please not have the bot output because they are conversatin channels [18:49:57] i can see it being relevant to operations, but of course also it's even more botnoise in there [18:50:01] there are always different points of view about this topic where to put bots :) [18:50:05] always has been [18:51:00] yeah, it's trick. i wish there were an easier "toggle the display of bot traffic" in this channel mechanism for irc [18:51:03] 'tricky [18:54:30] I was just trying to translate a German proverb. "Was dem einen seine Eule, ist dem anderen seine Nachtigall." Literal translation is "what is an owl for one is a nightingale for another" and I guess the closes ist "beauty is in the eye of the beholder". so I wanted to replace the birds with "valuable monitoring information" and "horrible bot spam" :) [18:55:56] hahah [18:56:22] as someone who used to have a great horned owl roosting outside his bedroom window, this proverb resonates. [18:56:31] I'm not on any tech teams, but imo bots are best kept to feed/spam channels and put people on separate channels [18:58:37] thanks for the input! I am honestly more in the other camp and think the feed channels mean nobody looks at the alerts anymore [18:59:27] but also it's like a cycle we go through, add bots, bots are considered noisy, move bots, think it's useful to bring them back to the people, repeat [19:01:11] it's getting more complicated than ever because now we have all the options, -team based, -topic based, centralized monitoring, decentralized per team channel [19:02:45] for example if we say gitlab alerts should be just where the topic is gitlab.. then a feed channel would be very limited, it would be gitlab-feed and exist for no other purpose than those alerts [19:03:24] while..if we decide to have ALL monitoring in one place,.and everyone watches -operations.. then that's also ok..you can't talk there but you can talk on the topic channels [19:04:10] but then that channel is moving so quickly that it's super easy to miss one.. unlesss everyone makes highlights for keywords... and so it goes [19:07:26] yeah, in practice i spot-check -operations a couple times a day, but i'm not *generally* keeping up with notifications there unless i'm running the train that week or similar. [19:08:50] yea, exactly. very similar for me [19:11:16] the answer can always be "both". in alertmanager we can define our own "receivers" that can be connected to monitoring checks. so we can easily have any combination of "email to whereever, IRC on any or multiple channels, automatic phab ticket with any tag, actual SMS to people" and even do different ones for severity, warning vs crit [19:11:34] this is all the new prometheus based monitoring instead of icinga [19:12:09] so that's also why we have all the options and it's easier to customize that stuff [19:13:11] so it could even be "forget about IRC, just create tickets and mail this list" [19:14:56] all of this is true for gerrit/phab/vtrs/doc/whatever..not just gitlab. sorry for wall of text. [19:56:07] mutante: the obvious fix is to send the alerts to a slack channel (/me ducks) [19:57:41] * Sario readys his trout, takes aim at bd808 [19:58:32] mutante: you've said it well before [19:58:34] https://bash.toolforge.org/quip/AWOyQVWee0jClLAO5zo1 [19:58:35] Sario: it was certainly a troutable statement [20:24:21] i've been halfway through figuring out how to have a hotkey to toggle bot visibility in weechat for like 3 years now. [20:35:21] I use an OSX-only client (Textual). I have a theme that allows me to make messages from bots be displayed in a smaller font with a lighter text color. This makes them pretty easy to ignore when I want to focus on talk with humans.