[09:19:47] not sure if a grafana8 related problem, but FYI if I click the share button to share a dashboard/graph and then I hit the big blue COPY button, the text is not copied into my clipboard, despite the green popup that says "Content copied to clipboard" [17:30:21] herron: 👋 [17:32:10] rzl: hey, just a sec [17:32:57] Hello. I have a strange version issue with`prometheus-elasticsearch-exporter` - On the new datahubsearch100[1-3] nodes it refuses to start with these values. [17:32:59] https://github.com/wikimedia/puppet/blob/production/modules/prometheus/templates/initscripts/prometheus-elasticsearch-exporter.systemd.erb#L8 [17:34:04] It wants long format (e.g. `--es.uri` instead of `-es.uri`) [17:34:58] I've checked on a logstash node which also has the same debian package of the exporter installed (I think) and it's happy with the short options. [17:35:05] https://www.irccloud.com/pastebin/L8ErVkmM/ [17:37:15] btullis: just a bystander but logstash1024 is on buster and datahubsearch1001 is on bullseye, so I wouldn't be surprised if there's a version gap there [17:37:16] Oh, different debian packages for buster and bullseye. [17:37:57] OK, I know where to look now. Thanks. [17:40:35] btullis: ah makes sense, happy to look at a patch to adjust that if you go that route [17:41:38] rzl: ok, ready, I'll depool thanos-fe1001 sound good? [17:42:18] works for me 👍 we probably don't even need to depool, the envoy hot restarter should do this completely non-disruptively [17:42:26] but certainly no objection if you'd rather [17:42:42] ok, yeah I'll do it just on the first host and then trust it [17:42:47] sounds good [17:44:13] rzl: depooled [17:44:40] rzl@cumin2002:~$ sudo debdeploy deploy -u 2022-03-09-envoyproxy.yaml -Q thanos-fe1001.eqiad.wmnet [17:45:06] rzl@thanos-fe1001:~$ curl -s localhost:9631/server_info | jq .version [17:45:06] "da2724c7457c3aefe2150377b95e915849d554a6/1.18.3/Clean/RELEASE/BoringSSL" [17:45:08] 👍 [17:47:22] herron: Thanks. Will ping you when I get a patch ready. [17:47:48] hmm, checking with the example from https://phabricator.wikimedia.org/T300119#7670842 I still see the same 404 on the upgraded host [17:48:35] rzl: yeah seeing the same, seems the :443 in the host header still gives that 404 [17:48:57] okay, good to know -- I'll update the task when we're done here [17:49:13] we might as well continue though, we want to get this upgrade done anyway :) as long as it isn't causing any regressions [17:49:32] sounds good yeah, alright I'll repool [17:51:16] rzl: ready when you are for the remaining hosts [17:51:40] cool -- want me to do one DC before the other, or just let er rip? [17:51:48] up to you [17:52:03] might as well finish eqiad and listen for screaming :) [17:52:08] k [17:52:24] rzl@cumin2002:~$ sudo debdeploy deploy -u 2022-03-09-envoyproxy.yaml -Q thanos-fe1\* [17:52:33] and done [17:56:06] rzl ack, lgtm [17:56:54] finishing up then! [17:57:04] rzl@cumin2002:~$ sudo debdeploy deploy -u 2022-03-09-envoyproxy.yaml -Q thanos-fe\* [17:57:33] that should be it, thanks for the help! glad it didn't turn out to be much :) [17:58:41] rzl: np! thx for the cautious upgrade [18:09:34] aha, the behavior didn't change because it's behind a config flag, now that we're upgraded we can flip that on [18:10:14] I'll either send a patch or just wait for g.odog, but good to know why nothing happened by default :) [18:11:50] rzl: nice was just reading the docs on that option, sounds good [18:12:05] happy to review a patch too, and if I have time might write one [18:12:12] rad [19:20:01] cwhite: thanks for T303550 (potentially triggered by my comment from this morning here :) ) [19:42:30] volans: thank you for reporting it!