[09:36:15] <elukey>	 Hi folks!
[09:36:27] <elukey>	 I have opened https://gerrit.wikimedia.org/r/c/operations/puppet/+/933866 to add the config for ores-legacy.wikimedia.org
[09:36:35] <elukey>	 never done it before, lemme know if it works
[09:36:50] <elukey>	 the plan is to create the DNS change after this one is reviewed/merged etc..
[09:42:49] <vgutierrez>	 elukey: why would you create the DNS after and not before?
[09:43:09] <vgutierrez>	 and by DNS are you referring to the wikimedia.org one or the discovery.wmnet one?
[09:44:22] <elukey>	 vgutierrez: o/ the .wikimedia.org one, I thought it was going to be needed only after ats/varnish knew about it
[09:44:33] <vgutierrez>	 yep, that makes sense
[09:45:03] <elukey>	 https://gerrit.wikimedia.org/r/c/operations/dns/+/933869 :)
[09:46:41] <elukey>	 ahhh snap I didn't add the SAN!
[09:46:47] <elukey>	 stupid me
[09:46:59] <elukey>	 I'll also add the ores.wikimedia.org one so that in the future the CNAME works
[09:47:02] <vgutierrez>	 consider adding ores.wm.o as well if you plan to unite them as some point
[09:47:02] <vgutierrez>	 :)
[09:47:20] <elukey>	 yep yep you are totally right, didn't think about it 
[10:03:46] <elukey>	 fixed :)
[10:10:25] <vgutierrez>	 https://www.irccloud.com/pastebin/ofMcozk9/
[10:10:35] <vgutierrez>	 you might need to fix the istio-envoy config
[10:10:45] <vgutierrez>	 with Host: ores-legacy.wm.o is returning a 404
[10:10:49] <vgutierrez>	 a 200 without the header
[10:12:26] <elukey>	 nice, this is new for me, I think something is missing indeed
[10:12:38] <elukey>	 (first service like this on lift wing)
[10:12:53] <vgutierrez>	 look at you with all your fancy services ;P
[10:13:18] * vgutierrez needs a "back in my day" meme
[10:18:38] <elukey>	 :D
[10:18:57] <elukey>	 I'd argue that those services are everything but fancy
[10:19:23] <vgutierrez>	 hnowlan: same as with the other one.. let's disable puppet on A:cp-text and test on a single node :)
[10:20:35] <hnowlan>	 vgutierrez: great! I'll use cp2037 again 
[10:23:40] <hnowlan>	 puppet is disabled, just merged 
[10:23:54] <vgutierrez>	 ack
[10:24:08] <hnowlan>	 enabling puppet on cp2037
[10:27:08] <vgutierrez>	 hnowlan: ok... your script is working and request ends targeting rest-gateway.discovery.wmnet 
[10:27:17] <vgutierrez>	 I'm getting a 404 though
[10:27:35] <hnowlan>	 Yeah, most likely a config error on rest-gateway
[10:30:50] <vgutierrez>	 hnowlan: are you able to confirm that rb-mw-mangling is still working as expected?
[10:31:17] <vgutierrez>	 aka it's hitting the right URL on the backend service
[10:35:04] <hnowlan>	 vgutierrez: trying to confirm that now - requests from ats are arriving as "/en.wikipedia.org/v1/page/pdf/Mountain" but the gateway is configured for /v1/pdf. Probably an error on the gateway side. "
[10:35:17] <hnowlan>	 are we okay to hold in our current state for a few minutes while I debug? 
[10:38:25] <vgutierrez>	 nope
[10:38:28] <vgutierrez>	 let's rollback
[10:38:38] <vgutierrez>	 and please set a beta environment for this :)
[10:38:48] <vgutierrez>	 Date:2023-06-28 Time:10:37:26 ConnAttempts:3 ConnReuse:0 TTFetchHeaders:-1 ClientTTFB:6 CacheReadTime:0 CacheWriteTime:0 TotalSMTime:6 TotalPluginTime:0 ActivePluginTime:0 TotalTime:6 OriginServer:appservers-ro.discovery.wmnet OriginServerTime:-1 CacheResultCode:ERR_CONNECT_FAIL CacheWriteResult:- ReqMethod:GET RespStatus:502 OriginStatus:000 ReqURL:http://127.0.0.1:3128/api/rest_v1/page/summary/Ade_Laoye?vgutierrez=1 
[10:38:48] <vgutierrez>	 ReqHeader:User-Agent:curl/7.74.0 ReqHeader:Host:127.0.0.1:3128 ReqHeader:X-Client-IP:- ReqHeader:Cookie:- BerespHeader:Set-Cookie:- BerespHeader:Cache-Control:- BerespHeader:Connection:- RespHeader:X-Cache-Int:cp2037 miss RespHeader:Backend-Timing:-
[10:38:56] <fabfur>	 btw I'm running the haproxy upgrade cookbook on codfw upload cluster 
[10:39:03] <vgutierrez>	 fabfur: :_)
[10:39:14] <vgutierrez>	 fabfur: you wanna stop that
[10:39:47] <fabfur>	 at the moment I had just an error on cp2027 for puppet is disabled
[10:40:10] <fabfur>	 that is on the text cluster
[10:40:19] <hnowlan>	 okay, revert here: https://gerrit.wikimedia.org/r/c/operations/puppet/+/933633 
[10:40:19] <vgutierrez>	 hnowlan: not a big deal sorry
[10:40:28] <fabfur>	 so the text cluster will be upgraded another time
[10:40:30] <vgutierrez>	 hnowlan: the 502 was a problem between the keyboard and the chir
[10:40:35] <vgutierrez>	 *chair
[10:40:35] <hnowlan>	 vgutierrez: phew! 
[10:41:01] <vgutierrez>	 (malformed host header)
[10:41:03] <hnowlan>	 waiting on verification from restbase devs that proton url patterns are correct
[10:41:12] <vgutierrez>	 they should be ok
[10:41:22] <vgutierrez>	 cause rb-mw-mangling is working as expected for other restbase services
[10:41:43] <vgutierrez>	 (validating with curl -H 'Host: en.wikipedia.org' -H 'X-Forwarded-Proto: https' http://127.0.0.1:3128/api/rest_v1/page/summary/Ade_Laoye?vgutierrez=1 -v -o /dev/null)
[10:44:31] <vgutierrez>	 fabfur: where did it fail on text? which host?
[10:44:37] <vgutierrez>	 and what's the current state of that host?
[10:44:46] <fabfur>	 cp2027
[10:44:49] <elukey>	 (fixed ores-legacy's envoy config)
[10:45:15] <fabfur>	 from the puppet log :
[10:45:17] <fabfur>	 https://www.irccloud.com/pastebin/SxrhjmJd/
[10:45:37] <vgutierrez>	 fabfur: so the node is currently depooled and with HAProxy misconfigured
[10:45:50] <vgutierrez>	 so we're unable to repool it till hnowlan finishes :)
[10:46:06] <fabfur>	 no prob, I'll work on the upload cluster in the meantime
[10:53:20] <vgutierrez>	 elukey: +1ed, please hold till hnowlan finishes :)
[10:57:36] <hnowlan>	 be done in a minute - problems with the mapping from rb-url to service url on the gateway side
[10:57:44] <hnowlan>	 doesn't seem like there's any issues with the ATS/lua side of things
[11:00:36] <vgutierrez>	 still getting 404s after your last deploy :)
[11:07:03] <hnowlan>	 making one last attempt and then we'll roll back. 
[11:07:47] <vgutierrez>	 hnowlan: ack
[11:08:00] <vgutierrez>	 curious about why curl-ing directly works but via ATS doesn't
[11:08:18] <vgutierrez>	 of course you'll have some extra headers like XFF and X-Client-IP
[11:11:31] <hnowlan>	 which curl are you seeing that works directly? 
[11:12:00] <hnowlan>	 The problem I'm seeing is that restbase adds "page" to the URL, which proton doesn't like. Even when adding a rewrite for that though I'm not seeing a correct response
[11:12:43] <vgutierrez>	 oh right
[11:12:55] <vgutierrez>	 the /page/ part isn't there in your examples that we tested the other day
[11:13:26] <hnowlan>	 yeah, restbase is adding that :( surprise
[11:15:59] <hnowlan>	 okay, I think I need go back to the team to figure this out. I'm pleased the lua stuff works though
[11:16:18] <vgutierrez>	 good
[11:16:23] <vgutierrez>	 let's revert it now then
[11:16:25] <hnowlan>	 revert here https://gerrit.wikimedia.org/r/933633 
[11:18:58] <hnowlan>	 Puppet reenabled 
[11:19:14] <hnowlan>	 thanks for the time again! 
[11:19:15] <vgutierrez>	 fabfur, elukey ^^
[11:21:09] <vgutierrez>	 hnowlan: rememeber that the traffic team is proudly fueled by ☕ and 🍺
[11:21:24] <vgutierrez>	 *remember 
[11:22:08] * hnowlan makes a note 
[11:23:14] <fabfur>	 But not mixed together
[11:24:04] <hnowlan>	 not even coffee stouts? 
[11:56:44] <vgutierrez>	 imperial coffee stout here please :)
[11:57:13] <vgutierrez>	 fabfur: please take care of cp2027, it's still depooled
[12:40:36] <fabfur>	 vgutierrez: I'll run the cookbook against cp2027 
[12:47:27] <fabfur>	 ok cp2027 repooled 
[12:47:39] <fabfur>	 I'll restart the cookbook against text@codfw
[12:47:58] <fabfur>	 hnowlan: is that ok for you?
[12:48:14] <fabfur>	 (the cookbook is for haproxy upgrade)
[12:53:28] <vgutierrez>	 yeah.. no blockers righjt now
[12:53:44] <fabfur>	 ack, proceeding
[12:58:49] <fabfur>	 vg: next dc after codfw? 
[13:11:04] <vgutierrez>	 fabfur: esams & eqiad tomorrow if nothing burns first
[13:11:08] <vgutierrez>	 and drmrs of course
[13:12:14] <fabfur>	 ok
[13:23:24] <wikibugs>	 10Traffic, 10SRE, 10envoy, 10serviceops: Refactor envoy.filters.http.router and envoy.filters.listener.tls_inspector - https://phabricator.wikimedia.org/T337405 (10JMeybohm) 05Open→03Resolved
[13:23:33] <wikibugs>	 10Traffic, 10SRE, 10envoy, 10serviceops, 10Patch-For-Review: Upgrade Envoy to supported version - https://phabricator.wikimedia.org/T300324 (10JMeybohm)
[13:33:04] <elukey>	 fabfur: o/ are you doing any work on cp nodes? Otherwise I'll merge my change :)
[13:33:43] <vgutierrez>	 text@codfw was getting an HAProxy upgrade
[13:33:49] <fabfur>	 I finished w/ cp nodes on codfw, thanks!
[13:41:46] <elukey>	 nice! ok to go then?
[13:42:26] <vgutierrez>	 yep
[13:42:34] <fabfur>	 for me is a go
[13:43:04] <elukey>	 <3
[13:52:27] <elukey>	 tried on a couple of nodes and it seems to work fine, I'll let puppet to do its work during the next half an hour
[13:52:34] <elukey>	 then is https://gerrit.wikimedia.org/r/c/operations/dns/+/933869 ok to merge?
[13:53:00] <sukhe>	 elukey: ok by us :)
[13:53:21] <elukey>	 sukhe: thank youuu
[14:04:06] <hnowlan>	 fabfur: no problem for me 
[15:43:59] <hnowlan>	 vgutierrez: would we have time to try again? I believe we've fixed the URL pathing issues in the gateway. I know it's late in the day for you though so happy to try tomorrow. 
[15:44:18] <vgutierrez>	 hnowlan: let's do that tomorrow morning please
[15:44:25] <hnowlan>	 sounds good, thanks 
[15:44:25] <vgutierrez>	 easier to handle potential issues that way
[15:57:47] <wikibugs>	 10netops, 10Infrastructure-Foundations, 10SRE: Connection errors from users on Vodafone DE (AS3209) [28.06.2023] - https://phabricator.wikimedia.org/T340670 (10cmooney) p:05Triage→03Low