[09:21:10] topranks: re: rpki I think I did it, changing the folder only doesn't result in a "changeable" edit afaics, I wiggled the tags and saved the dashboard and that did the trick [09:22:16] godog: ah ok, little bug then in the UI? [09:22:32] no probs sounds like an easy workaround, thanks! [09:22:42] yeah looks like a grafana bug to me [13:46:10] godog: I'm getting an error from keyholder on alet1002, would you mind if I restart it? (and re-arm ofc) [13:48:00] volans: sure go ahead [13:49:58] done, but getting the same issue [13:50:02] sign_and_send_pubkey: signing failed for ED25519 "metamonitor" from agent: agent refused operation [13:50:28] does it ring a bell? I used an ssh command from my history so I'm fairly sure it worked in the past [13:52:27] volans: it does, looks like the proxy needs to be restarted too? e.g. https://phabricator.wikimedia.org/T307351 [13:53:10] I've alrady done it twice [13:53:32] I'm usually the one telling people that trick :D [13:53:45] lol ok then I don't know [14:32:40] FIRING: [2x] LogstashIndexingFailures: Logstash Elasticsearch indexing errors - https://wikitech.wikimedia.org/wiki/Logstash#Indexing_errors - https://alerts.wikimedia.org/?q=alertname%3DLogstashIndexingFailures [14:37:40] RESOLVED: [2x] LogstashIndexingFailures: Logstash Elasticsearch indexing errors - https://wikitech.wikimedia.org/wiki/Logstash#Indexing_errors - https://alerts.wikimedia.org/?q=alertname%3DLogstashIndexingFailures [15:27:15] cwhite just a heads-up, we will soon merge https://gerrit.wikimedia.org/r/c/operations/puppet/+/1091325 ...based on PCC, it does not look like it will affect your logstash hosts. Hit me up if you have questions [17:44:12] inflatador: c.white is AFK today but thanks for the heads up [18:14:41] lmata ACK, thanks for pinging. Everything looks good AFAICT [18:15:21] Sweet, thanks ! [18:25:18] lmata np! sorta-related question: do you (or rest of team) know where the code that builds y'all's opensearch package might live? I vaguely remember Cole telling me he has a python script [18:27:24] lmata NM, found the instructions https://wikitech.wikimedia.org/wiki/OpenSearch [18:27:51] Ack!