[10:16:56] <jynus>	 not sure who to ask regarding increased latency on mw k8s misc eqiad- if I should just ignore it or ping someone?
[10:17:08] <jynus>	 it seems to be happening since 7:15 today
[10:17:44] <jynus>	 https://grafana.wikimedia.org/goto/BHuYQi4Sz?orgId=1
[10:27:59] <_joe_>	 jynus: mw-misc?
[10:28:04] <_joe_>	 that's noc.wikimedia.org
[10:28:13] <_joe_>	 and the latency is below 2 ms
[10:28:46] <_joe_>	 so I don't think it's going to be actionable or a problem
[10:29:04] <jynus>	 ok, thanks
[10:29:11] <_joe_>	 we can look at the access logs to see if there's increased traffic
[10:29:31] <jynus>	 I mentioned it because it was spaming ops-
[10:32:01] <_joe_>	 the busy workers alert? yeah it doesn't even make sense to have it tbh
[15:12:08] <jynus>	 ^ see previous conversation with context for urandom, jhathaway
[15:12:24] <jhathaway>	 jynus: thanks
[15:14:44] <urandom>	 jynus: thanks
[16:09:13] <vgutierrez>	 somebody with reprepro super powers? we are seeing some unexpected behavior with https://gerrit.wikimedia.org/r/c/operations/puppet/+/957766
[16:10:18] <volans>	 vgutierrez: morit.z is back on wed. IIRC
[16:10:48] <vgutierrez>	 yeah... just in case we got a backup for him ;P
[16:11:00] <volans>	 I'm having a look if I can spot anything obvious
[16:11:09] <volans>	 whatissue are you getting beside "it doesn't work"? :D
[16:11:28] <vgutierrez>	 volans: it's importing bullseye package rather than bookworm one
[16:11:52] <vgutierrez>	 https://www.irccloud.com/pastebin/3x5RjwCX/
[16:12:09] <taavi>	 the names of the updates blocks don't seem to match the ones used in Update: in distributions-wikimedia
[16:12:14] <volans>	 vgutierrez: I see thirdparty/haproxy26 instead of thirdparty/haproxy26-bookworm does that matter?
[16:12:29] <volans>	 or should it be smart enough to do that automatically?
[16:12:51] <vgutierrez>	 taavi: hmmm
[16:14:03] <sukhe>	 those don't match for bullseye though?
[16:14:25] <volans>	 but we do have Name: thirdparty/haproxy26 pre-existing
[16:14:29] <vgutierrez>	 yeah
[16:14:49] <volans>	 so my first approach would be to use the matching names correctly
[16:15:30] <vgutierrez>	 ack, let me fix that
[16:15:36] <vgutierrez>	 taavi: beer++
[16:15:46] <vgutierrez>	 (or whatever is your drink of choice of course)
[16:15:47] <volans>	 I said the same... no beers for me? :-P
[16:16:09] <vgutierrez>	 volans: I'm afraid I would overflow your counter and you'll end up with no beers
[16:16:18] <volans>	 rotfl
[16:20:33] <vgutierrez>	 dropping haproxy24 for bullseye cause it doesn't exist at all :)
[16:20:43] <vgutierrez>	 https://gerrit.wikimedia.org/r/c/operations/puppet/+/969814
[16:22:00] <volans>	 go ahead and break it
[16:28:20] <vgutierrez>	 vgutierrez@apt1001:~$ sudo -i reprepro -C thirdparty/haproxy26 ls haproxy
[16:28:20] <vgutierrez>	 haproxy | 2.6.13-1~bpo10+1 | buster-wikimedia | amd64
[16:28:21] <vgutierrez>	 vgutierrez@apt1001:~$ 
[16:28:30] <vgutierrez>	 definitely broke it :)
[16:28:53] <volans>	 maybe it needs to delete them manually?
[16:28:55] <volans>	 the ones you removed
[16:29:17] <vgutierrez>	 triggered a reprepro clearvanished
[16:29:54] <vgutierrez>	 so now vgutierrez@apt1001:~$ sudo -i reprepro -V --noskipold --component thirdparty/haproxy26 checkupdate bookworm-wikimedia looks saner
[16:30:07] <vgutierrez>	 as in it's actually checking aptmethod got 'http://haproxy.debian.net/dists/bookworm-backports-2.6/InRelease'
[16:35:58] <vgutierrez>	 volans: tempted to revert seeing the impact :/
[16:36:07] <volans>	 no luck so far?
[16:36:18] <volans>	 usually when I google for reprepro I get wikitech results
[16:36:33] <vgutierrez>	 it's worse.. we lost the packages for haproxy on bullseye
[16:37:00] <volans>	 ouch...
[16:37:13] <volans>	 let me check quickly the docs
[16:37:17] <taavi>	 which version?
[16:37:19] <_joe_>	 jhathaway, urandom all quiet on the eastern front btw
[16:37:28] <vgutierrez>	 taavi: 2.6
[16:37:35] <jhathaway>	 _joe_: great thanks
[16:37:39] <volans>	 vgutierrez: you run already ' reprepro --delete clearvanished ' right?
[16:37:43] <urandom>	 _joe_: thanks; quiet is good
[16:37:48] <vgutierrez>	 volans: yeah
[16:38:00] <vgutierrez>	 nothing that can be fixed with a revert && fetching the packages again though
[16:38:30] <volans>	 you mean before both patches?
[16:38:45] <taavi>	 now the component in bullseye is named 'thirdparty/haproxy26-bullseye', but I think it should be just 'thirdparty/haproxy26'
[16:38:47] <marostegui>	 phabricator is down
[16:39:09] <vgutierrez>	 volans: just the one I merged a few minutes ago
[16:39:23] <volans>	 ack go ahead
[16:39:31] <volans>	 sorry I'll have to step afk very soon
[16:39:44] <vgutierrez>	 taavi: that would collision with the buster one?
[16:40:08] <taavi>	 component names are independent of the update block names
[16:40:16] <taavi>	 give me a minute, I'll make a patch
[16:40:59] <vgutierrez>	 so.. do you mean modules/aptrepo/files/distributions-wikimedia should have thirdparty/haproxy26 rather than thirdparty/haproxy26-bullseye?
[16:42:41] <taavi>	 in the Components: section, yes
[16:43:45] <taavi>	 I think you should do something like https://gerrit.wikimedia.org/r/c/operations/puppet/+/969819/, so always use thirdparty/haproxyXY in the component names and thirdparty/haproxyXY-DISTRO for the update block names to make it easier to work with
[16:46:14] <vgutierrez>	 gotcha... Components should have the component name and the updates section should contain the update name defined on the updates file
[16:47:36] <taavi>	 yep. components are the thing apt clients see, and the update section names, although similar in our config, are unrelated and just used in the config to decide which updates to do for which distro
[16:49:27] <vgutierrez>	 https://gerrit.wikimedia.org/r/c/operations/puppet/+/969821 that should fix it without extra changes
[16:49:55] <taavi>	 that would indeed fix the immediate issue
[16:53:55] <vgutierrez>	 https://www.irccloud.com/pastebin/0mL2ZY11/
[16:54:09] <vgutierrez>	 taavi: I owe you 2x $DRINK now :) thx
[16:57:51] <taavi>	 :)