[07:58:14] <vgutierrez>	 hnowlan: smelling fresh coffee here.. ready whenever you are
[09:07:17] <hnowlan>	 vgutierrez: great - I'll be ready in like 10 minutes 
[09:07:22] <vgutierrez>	 nice
[09:27:24] <hnowlan>	 vgutierrez: think I'm good to go if you are 
[09:29:09] <vgutierrez>	 hnowlan: sure
[09:29:23] <vgutierrez>	 curl -v -H 'Host: en.wikipedia.org' -H 'X-Forwarding-Proto: https' http://127.0.0.1:3128/api/rest_v1/feed/onthisday/all/02/06 -o /dev/null
[09:29:23] <vgutierrez>	 looks like a nice testcase, right?
[09:30:04] <hnowlan>	 yep, looks good!
[09:30:21] <vgutierrez>	 cool, the usual cp2037 dance?
[09:30:33] <hnowlan>	 yeah cool :) 
[09:30:40] <vgutierrez>	 CR +1ed
[09:31:07] <hnowlan>	 thanks! disabling puppet
[09:31:41] <hnowlan>	 depooled cp2037
[09:35:13] <vgutierrez>	 hnowlan: hmm new endpoint doesn't set cache-control at all?
[09:35:58] <vgutierrez>	 old one sends: < cache-control: s-maxage=300, max-age=60 
[09:37:39] <hnowlan>	 hmph, that was missed.
[09:40:40] <hnowlan>	 we'll need to roll back until that's in place I guess
[09:44:27] <vgutierrez>	 ack
[09:45:12] <hnowlan>	 revert here https://gerrit.wikimedia.org/r/c/operations/puppet/+/946656 
[09:45:35] <hnowlan>	 but while we're in this state - if it's not too cheeky, once we have the revert in place I have another route I was going to ask about :) 
[09:46:12] <hnowlan>	 https://gerrit.wikimedia.org/r/c/operations/puppet/+/946928 this is a new service endpoint so it's not connected to anything, but we'd like to expose it via the gateway 
[09:46:22] <hnowlan>	 if you'd rather not, then no problem 
[09:49:37] <vgutierrez>	 "not connected to anything"?
[09:49:47] <vgutierrez>	 meaning it isn't replacing anything?
[09:50:07] <hnowlan>	 yeah, sorry - not consumed by anything external 
[09:51:17] <vgutierrez>	 hnowlan: do you have a sample request handy?
[09:55:45] <hnowlan>	 vgutierrez: curl -v -H 'Host: wikimedia.org' -H 'X-Forwarding-Proto: https' http://127.0.0.1:3128/api/rest_v1/metrics/knowledge-gap/per-category/en.wikipedia/gender/female/20210101/20231201
[09:56:13] <vgutierrez>	 so /en.wikipedia.org/v1/metrics/knowledge-gap/per-category/en.wikipedia/gender/female/20210101/20231201 on the origin?
[09:56:29] <vgutierrez>	 hmm wikimedia.org sorry
[09:58:59] <vgutierrez>	 curl -H 'Host: wikimedia.org' -H 'X-Forwarded-Proto: https' https://rest-gateway.discovery.wmnet:4113/wikimedia.org/v1/metrics/knowledge-gap/per-category/en.wikipedia/gender/female/20210101/20231201 -v
[09:59:06] <vgutierrez>	 that's currently triggering a 404
[10:02:44] <vgutierrez>	 sigh...
[10:03:09] <hnowlan>	 incredibly poor timing for my bouncer to drop :\
[10:03:12] <hnowlan>	 apologies vgutierrez 
[10:03:21] <vgutierrez>	 no problem
[10:04:24] <hnowlan_>	 dunno where I dropped off - the service can be queried on the gateway via `curl https://rest-gateway.discovery.wmnet:4113/analytics.wikimedia.org/v1/metrics/knowledge-gap/per-category/en.wikipedia/gender/female/20210101/20231201`
[10:04:51] <vgutierrez>	 so host is analytics.wikimedia.org rather than wikimedia.org
[10:05:50] <hnowlan_>	 the gateway will rewrite the hostname for all domains when the request arrives at the service so it doesn't really matter
[10:06:22] <vgutierrez>	 curl https://rest-gateway.discovery.wmnet:4113/analytics.wikimedia.org/v1/metrics/knowledge-gap/per-category/en.wikipedia/gender/female/20210101/20231201 currently triggers a 404 here
[10:06:36] <vgutierrez>	 here == cp2037
[10:08:15] <hnowlan_>	 aha. it appears they didn't deploy the service to codfw? heh 
[10:08:30] <hnowlan_>	 okay, let's revert the initial and just come back to this another time
[10:08:42] <vgutierrez>	 👍
[10:10:24] <vgutierrez>	 hnowlan_: eqiad doesn't look great either
[10:11:06] <vgutierrez>	 https://www.irccloud.com/pastebin/n1I987Oh/
[10:13:15] <vgutierrez>	 hnowlan_: quick check on a 200 response shows cache-control but no etag or last-modified header, could we get any of those?
[10:13:50] <hnowlan_>	 yep, I'll try to get those added 
[10:14:09] <hnowlan_>	 it looks like the endpoint works sporadically, some kind of deployment issue maybe. I'll follow up with the devs
[10:14:12] <vgutierrez>	 cheers
[10:14:39] <hnowlan_>	 revert merged, puppet re-enabled
[10:14:48] <hnowlan_>	 repooling cp2037
[10:14:55] <vgutierrez>	 thx
[10:16:00] <hnowlan>	 thanks for the help! 
[13:34:49] <wikibugs>	 10netops, 10Infrastructure-Foundations, 10SRE, 10SRE-tools, 10Patch-For-Review: Improve Homer output when Juniper device rejects config - https://phabricator.wikimedia.org/T328747 (10ayounsi) a:03ayounsi
[13:42:21] <wikibugs>	 10netops, 10Infrastructure-Foundations, 10SRE, 10netbox: Netbox Juniper report - https://phabricator.wikimedia.org/T306238 (10ayounsi) I sent a new email to Juniper yesterday to ask again about the best next steps here.
[13:44:45] <wikibugs>	 10netops, 10Infrastructure-Foundations, 10SRE, 10Traffic-Icebox: Anycast: consistent routers->servers routing - https://phabricator.wikimedia.org/T253666 (10ayounsi) 05Resolved→03Declined
[13:44:53] <wikibugs>	 10netops, 10Infrastructure-Foundations, 10SRE, 10Traffic-Icebox, 10Patch-For-Review: Anycast AuthDNS - https://phabricator.wikimedia.org/T98006 (10ayounsi)
[13:45:01] <wikibugs>	 10netops, 10Infrastructure-Foundations, 10SRE, 10Traffic-Icebox, 10Patch-For-Review: Anycast AuthDNS - https://phabricator.wikimedia.org/T98006 (10ayounsi)
[13:45:15] <wikibugs>	 10netops, 10Infrastructure-Foundations, 10SRE, 10Traffic-Icebox: Anycast: consistent routers->servers routing - https://phabricator.wikimedia.org/T253666 (10ayounsi) 05Stalled→03Resolved a:03ayounsi Boldly closing this as Katran will solve some if not all those limitations.
[13:48:53] <wikibugs>	 10netops, 10Infrastructure-Foundations, 10SRE, 10Traffic-Icebox, 10User-jbond: Anycast: consistent ICMP packet too big routing - https://phabricator.wikimedia.org/T253732 (10ayounsi) @Vgutierrez do you know how the future L4LB will handle ICMP PTB packets? Can it route it to the proper source host?
[14:17:20] <wikibugs>	 10netops, 10Infrastructure-Foundations, 10SRE, 10Traffic-Icebox, 10User-jbond: Anycast: consistent ICMP packet too big routing - https://phabricator.wikimedia.org/T253732 (10Vgutierrez) >>! In T253732#9080504, @ayounsi wrote: > @Vgutierrez do you know how the future L4LB will handle ICMP PTB packets? Can...
[14:17:59] <vgutierrez>	 XioNoX: ^^ let me know if you need more details or that's enough
[14:35:50] <XioNoX>	 vgutierrez: noted thanks! I guess we don't know what their timeline is?
[14:36:02] <vgutierrez>	 timeline for...?
[14:36:38] <vgutierrez>	 code is there and ICMP PTB packets will reach the proper real server
[14:37:15] <vgutierrez>	 what seems to be a WIP is the lower limit in MTU
[14:37:24] <XioNoX>	 ah right
[14:37:36] <XioNoX>	 I thought they were not going to the proper one yet
[14:37:39] <XioNoX>	 cool
[14:37:54] <vgutierrez>	 as long as the real server is still there :)
[14:39:41] <XioNoX>	 of course
[14:42:34] <wikibugs>	 10netops, 10Infrastructure-Foundations, 10SRE, 10Traffic-Icebox, 10User-jbond: Anycast: consistent ICMP packet too big routing - https://phabricator.wikimedia.org/T253732 (10ayounsi) 05Open→03Declined Thanks, then like {T253666} I'm going to boldly close this task.
[14:42:50] <wikibugs>	 10netops, 10Infrastructure-Foundations, 10SRE, 10Traffic-Icebox, 10Patch-For-Review: Anycast AuthDNS - https://phabricator.wikimedia.org/T98006 (10ayounsi)
[14:44:04] <wikibugs>	 10netops, 10Infrastructure-Foundations, 10SRE: Detect IP address collisions - https://phabricator.wikimedia.org/T189522 (10ayounsi) 05Open→03Resolved a:03ayounsi We have a working solution for the mgmt network (until it's time to split mgmt into smaller subnets). And for production, automation and per...
[15:03:40] <hnowlan>	 vgutierrez: would you have time for another attempt at wikifeeds? cache-control headers have been added everywhere they were missing 
[15:04:40] <sukhe>	 hnowlan: vg is out for the weekend now
[15:05:43] <hnowlan>	 ah, okay 
[15:07:51] <hnowlan>	 would anyone else like to help with an ATS config change? We'd like to route traffic to wikifeeds via the rest-gateway (already tried via  https://gerrit.wikimedia.org/r/945558 - the routing worked but the service was missing cache-control for some endpoints, which has since been fixed) ) 
[15:08:14] <sukhe>	 hnowlan: since it has his +1, I am happy 
[15:08:15] <sukhe>	 to
[15:09:25] <sukhe>	 what kind of help are you looking for here? simply merging this and being around or something else?
[15:10:49] <hnowlan>	 sukhe: yep, pretty much! Just a +1 on the re-review and being around in case anything goes wrong :)
[15:10:57] <hnowlan>	 (it won't, I swear!)
[15:11:10] <sukhe>	 hnowlan: you have his +1, merge away :)
[15:11:16] <sukhe>	 and happy to help if things go south
[15:21:55] <hnowlan>	 sukhe: I actually had to revert earlier so could you review this please? <3 https://gerrit.wikimedia.org/r/c/operations/puppet/+/947372 
[15:22:06] <sukhe>	 looking
[15:22:27] <sukhe>	 cool, matches the revert exactly 
[15:22:34] <sukhe>	 as in the previously reviewed change
[15:26:48] <hnowlan>	 thanks! 
[15:26:59] <sukhe>	 hnowlan: merge away!
[15:27:39] <hnowlan>	 gonna disable puppet on A:cp beforehand just to be careful, will depool cp2037 and try it out there 
[15:27:55] <sukhe>	 thanks
[15:35:16] <hnowlan>	 alright, looks okay to me. See the cache-control headers as expected. I think I'm comfortable reenabling puppet on the rest of the hosts 
[15:37:44] <sukhe>	 hnowlan: looks ok!
[15:39:32] <hnowlan>	 sweet, let's go 
[15:40:13] <hnowlan>	 puppet enabled, cp2037 repooled 
[15:41:15] <sukhe>	 thanks!
[15:42:12] <hnowlan>	 thank you! 
[16:11:04] <wikibugs>	 10netops, 10Infrastructure-Foundations, 10SRE: cr2-esams:FPC0 Parity error - https://phabricator.wikimedia.org/T318783 (10ayounsi) Opened high priority case 2023-0809-747283 asking for a RMA.
[17:14:52] <hnowlan>	 okay, sadly we'll need to roll back again - it seems restbase was shielding us from some misconfigured clients that are making wikifeeds 5xx https://gerrit.wikimedia.org/r/c/operations/puppet/+/946665 
[17:29:25] <sukhe>	 ah ok, thanks hnowlan!
[19:33:29] <wikibugs>	 10netops, 10Infrastructure-Foundations, 10SRE: cr2-esams:FPC0 Parity error - https://phabricator.wikimedia.org/T318783 (10cmooney) RMA in progress, Juniper happy with address for replacement and staff at destination are aware of delivery.  I will decom the existing faulty card on Sunday when on site and prep...
[20:13:55] <wikibugs>	 10Traffic, 10SRE: Offer AuthDNS service over IPv6 - https://phabricator.wikimedia.org/T81605 (10ssingh)
[20:15:54] <wikibugs>	 10Traffic, 10SRE: Offer AuthDNS service over IPv6 - https://phabricator.wikimedia.org/T81605 (10ssingh) In discussion with @cmooney, we will be revisiting this task again when Traffic does some other authdns-related work, so removing it from the Traffic-Icebox.