[00:40:42] razzi: since you don't have a Wikimedia cloak, can you make an edit on-wiki saying "I am razzi on Libera Chat" or something? [06:49:08] good morning! [06:49:25] the ripe atlas esams looks down from icinga [06:49:32] can't find a task or a discussion about it on IRC [06:49:41] so I guess it is not maintenance [06:53:10] elukey: ah yeah I wanted to open a dcops task about it, the console is dead as well, but the switch port is up and seeing traffic [06:54:10] XioNoX: perfect thanks :) [08:52:46] elukey: https://phabricator.wikimedia.org/T303242 [09:56:22] godog: maybe you know, is there a way to combine searches on alertmanager ui? As in "team=wmcs || summary=~cloud" kind of thing [10:02:57] dcaro: mmhh not afaict from a quick search [10:06:18] ack, it seems kind of a basic feature, but yes, I don't find anything about it either [10:07:54] upstream is nice and helpful, I recommend opening an issue/feature request at https://github.com/prymitive/karma [10:08:00] maybe there's alreay a way and we're missing it [10:08:08] 👍 will do! [14:30:34] really basic question: when typing tags into phabricator, how do you get it to complete a current suggestion without changing focus to the next field? (tab does this) [14:31:46] kormat: Enter? [14:32:07] 🤦‍♀️ [14:32:21] btullis: thanks! [14:34:30] kormat: A pleasure. It's nice to be able to get one right once in a while :-) [16:31:25] cwhite: good morning! thanks for the quick review for the test fix on modules/smart ( https://gerrit.wikimedia.org/r/c/operations/puppet/+/769060/ and follow up) [16:31:48] that got raised by andrew earlier cause another patch ended up failing [16:32:02] however I don't have +2 / puppet-merge so you will have to do those :D [16:32:28] hashar: Hi! Thanks for patching those :) [16:32:48] I will do the merge then, no problem. [16:55:00] btullis: puppetmaster1001 confd template alert is complaining about datahubsearch, you need to pool at least one backend server [17:00:56] vgutierrez: Oh, sorry. Looking now. [17:01:52] and clean the error files on /var/run/confd-template [17:02:17] Will do. [17:02:26] thx <3 [17:16:25] is there are convenient way from puppet to place files in PYTHONPATH? Particularly i need to do something that reuses 80% of a python script we already deploy from puppet (for icinga), so i'd like to separate that 80% out into a file i can import from [17:25:11] ebernhardson: I'm sure that many will find this ugly, but https://github.com/wikimedia/puppet/blob/0d05a635f50beb36b4dcb60bff3f1bbb063dcc8d/modules/openstack/manifests/clientpackages/wallaby/bullseye.pp#L27-L33 [17:28:11] bd808: lol, i had pondered that but thought we might have a better way. That works too, and having it already somewhere else seems to make it more justified [17:30:14] we've all done that at some point, but IMHO it should be a temporary solution towards something else. If the software is growing it probably deserves its own repo, if it's doing orchestration stuff, consider converting it into a cookbook, if it's something generic enough talk to me and jo.hn to see if it might fit into wmflib [17:31:09] volans: hmm, maybe wmflib. In particular this opens tickets in phabricator and ack's icinga/nagios alerts [17:32:09] ebernhardson: https://doc.wikimedia.org/wmflib/master/api/wmflib.phabricator.html#wmflib.phabricator.Phabricator.task_comment [17:32:11] (as an icinga event_handler) [17:32:27] ah, like the raid handler [17:32:27] ? [17:32:40] volans: yes, i'm puting 80% of raid_handler into a reusable file so i can, well ,reuse it :) [17:32:41] is something that can be migrated to alertmanager or is planned to be migrated to alertmanager? [17:33:14] because AM has phabricator task as a possible alert mechanism [17:33:19] I guess it could go in prometheus, but seemed odd. The dates i'm checking change 2 to 4 times a year. It would just be storing a constant value in prometheus [17:33:36] this essentially looks to see if certain things are too old and creates a phab ticket to make it new [17:33:38] * volans curious, what data? [17:33:42] index creation dates [17:34:26] I think it might be totally ok to have in prometheus, but check with o11y, it might be the simplest solution [17:35:01] would perhaps be easier, what i have so far is a bunch of super-specific pieces that have to tie together [17:37:17] what memories you've brought back to me... that raid handler is one of the first things I did when I started in my current position [17:40:39] ebernhardson: if needed I'm available to discuss more details with you about that quite old code and see what we can do without spending too much time on it [17:41:33] volans: i'm currently pondering :) It's not too hard to put it into prometheus, we already have a custom collector that can be adjusted fairly easily. Might be best to dump this raid_handler work i have so far (only spent last afternoon on it) [17:42:03] i suppose mostly i skipped prometheus for theoretical reasons that aren't particularly important since i'm checking a total of 4 indexes across dc's [17:42:33] I'm sure those 4 ints will make prometheus fill the disks! :-P [17:43:18] indeed :) [18:56:15] Are the outages in any way related to dirty pipe? [18:57:07] Naleksuh: all 3 major cloud providers have issues [18:57:17] Anyone else? [18:57:27] I know some wikimedia routes are backed by cloudflare, not sure if related [18:57:50] Please don't use the C word [18:58:12] the incident is not related to dirty pipe, no [18:59:04] SRE is investigating the issue [18:59:27] It's up for me, but might just be a coincidence [18:59:47] not related to Cloudflare neither fyi [19:02:33] Ty [19:47:11] hey all, for https://phabricator.wikimedia.org/T293862 we (Search team) would like to install the .deb at https://github.com/Netflix-Skunkworks/jvmquake/releases on some prod hosts, what is the recommended way to do this? [19:49:30] More specifically, are there any objections to manually uploading it to reprepro instead of trying to rebuild it? [19:49:57] There does not seem to be a signature of those packages available in the github release [20:14:59] inflatador, gehel: seems like a question for m.oritz when he's around [20:15:55] kormat understood, he was helping us earlier but we ran into some issues building it manually. Will reach out to him tomorrow