[08:58:36] volans: I might need some extra coffee but any idea why cumin 'R:acme_chief::cert and A:puppet7' doesn't work as expected? [09:00:41] vgutierrez: yes you need the global grammar, P{R:...} and A:... [09:00:53] aliases are part of the global grammar, not the puppet one [09:02:23] dcaro: we got 3 instances.. cloudlb200[1-3]-dev rejecting SSH from cumin hosts at the moment, is that expected? [09:02:44] vgutierrez: if it's from cumin1002 yes nftables issue [09:02:45] use cumin2002 [09:03:06] see T356174 [09:03:07] T356174: Connection errors to some hosts from cumin1002 - https://phabricator.wikimedia.org/T356174 [09:03:08] thx again [09:03:45] tl;dr migrating to nftables requires a reboot for a proper clean setup [09:05:07] good to know :) [09:11:14] I'm going through the list in https://phabricator.wikimedia.org/T356174 to backfill reboots, will reach out for cloudlb soon [09:21:56] volans: I guess you're already aware but FYI https://www.irccloud.com/pastebin/AwjKSXiS/ [09:22:39] wht would be the bug? it's trying to get the uptime and the host is going down [09:23:07] sleep a bit before starting polling? but each host would be different and could take much longer than others [09:24:09] yep.. nothing to be worried about I guess [09:25:36] a wild arturo appeared :) [09:25:53] 😊 [09:25:56] arturo: nice to see you around ;P [09:26:06] vgutierrez: thanks, happy to be back [09:27:31] welcome back! [09:29:05] o/ [09:46:38] welcome back indeed arturo ! [09:47:10] :-) thanks [09:58:50] hey arturo, great to see you ! [10:05:04] likewise :-P [10:24:43] Hello arturo 👋 [10:24:58] hey! o/ [10:28:40] arturo: welcome ! how was onboarding ? [10:49:10] I'm having a fast onboarding here this time around, I guess [11:07:34] hi arturo :) [11:07:45] o/ [12:07:54] WB arturo o/ [12:08:03] lmata: o/ [12:40:59] brouberol: so I think it's deployed https://codesearch.wmcloud.org/analytics/?q=foo&files=&excludeFiles=&repos=#repos/data-engineering/eventutilities-python [12:44:22] Fixed the frontend too [13:04:10] Thank you! [13:12:47] welcome back arturo ! [13:27:25] arnaudb: o/ thanks [15:53:53] Hi! Can someone invite me to wikimedia-security? I think I disconnected some time ago and did not notice until now [16:01:35] dcaro: you mean mediawiki_security? [16:01:46] yes, sorry [16:02:06] ok, I'll do that now [16:02:17] Oh, I can join :) [16:02:21] yeah [16:02:24] you are in the list [16:02:25] thanks xd [17:35:40] Does anyone have an example of alerting based on a logstash message? [21:32:40] inflatador: we usually generate metrics from logs then alert on those metrics. An example can be found in alerts/team-sre/mediawiki.yaml `log_mediawiki_servergroup_level_channel_doc_count`. [21:44:15] cwhite Thanks, will take a look [21:44:52] Happy to answer questions if you have :) [22:01:38] cwhite: hello, if you re still around, I have noticed you have merged my patch for Apache Error log in ECS format [22:01:48] but I havent verified whether that is actually working ;) [22:03:32] I guess the next step is to apply the rule to one of our apache [22:04:33] ( https://phabricator.wikimedia.org/T332672 ) [22:05:25] That sounds like the next logical step to me. Try the new directive on a host and see what happens :)