[08:39:44] <wikibugs>	 10Traffic, 10Data-Engineering, 10Data-Platform-SRE, 10SRE: Move varnishkafka to PKI - https://phabricator.wikimedia.org/T337825 (10elukey) 05Open→03Resolved a:03elukey
[08:59:57] <hnowlan>	 hey vgutierrez - would you be available to give me a hand with rolling https://gerrit.wikimedia.org/r/c/operations/puppet/+/929674 out this morning? 
[09:01:24] <vgutierrez>	 hnowlan: morning, I think so
[09:01:31] <hnowlan>	 just wondering as regards the depooling process if there's any specific hosts that would be better to pick, and also which service they're using - I tried `confctl select service=text-https get` and I go no results 
[09:01:56] <vgutierrez>	 service=ats-be :)
[09:02:16] <hnowlan>	 ahh heh
[09:03:21] <vgutierrez>	 any of 'service=ats-be,cluster=cache_text,dc=codfw' should do the trick
[09:04:51] <hnowlan>	 looks like cp1082.eqiad.wmnet isn't pooled atm, could I just use that or is it special? 
[09:05:28] <hnowlan>	 ah that's cache_upload, nm
[09:09:39] <hnowlan>	 randomly picked cp2037.codfw.wmnet - just waiting for one of the dev team to be on hand and I'll let you know
[09:10:12] <vgutierrez>	 hnowlan: ack
[09:10:35] <vgutierrez>	 hnowlan: BTW, do you have a sample request  that's valid on the new endpoint?
[09:16:37] <hnowlan>	 vgutierrez: yep: https://rest-gateway.discovery.wmnet:4113/en.wikipedia.org/v1/pdf/Tornado/a4/desktop 
[09:17:31] <hnowlan>	 at the edge that would be https://en.wikipedia.org/api/rest_v1/page/pdf/Tornado 
[09:17:34] <hnowlan>	 I think we should be good to go whenever suits
[09:18:04] <vgutierrez>	 fabfur: how's your cookbook going?
[09:18:36] <vgutierrez>	 hnowlan: give us a few minutes while we finish an  upgrade in eqsin
[09:18:40] <fabfur>	 no errors at the moment
[09:19:39] <fabfur>	 about ~3 hosts done for text and 3 for upload
[09:19:40] <hnowlan>	 vgutierrez: ack, no worries 
[09:19:46] <vgutierrez>	 hnowlan: hmm that triggers a 404 here
[09:19:55] <vgutierrez>	 vgutierrez@cp3060:~$ curl -H 'Host: en.wikipedia.org' https://rest-gateway.discovery.wmnet:4113/en.wikipedia.org/v1/pdf/Tornado/a4/desktop 
[09:19:55] <vgutierrez>	 {"httpCode":404,"httpReason":"Not Found"}
[09:23:13] <hnowlan>	 ahh ofc the host header will still be set when the request gets rewritten 🤦
[09:23:24] <hnowlan>	 envoy is very strict about that, easy to fix though 
[09:23:32] <vgutierrez>	 hnowlan: you need to reissue the TLS certs as wel
[09:23:34] <vgutierrez>	 *well
[09:24:17] <vgutierrez>	 hnowlan: the TLS cert needs to include the public hostnames as well
[09:24:26] <hnowlan>	 ahhh
[09:25:17] <vgutierrez>	 https://www.irccloud.com/pastebin/mtRo6DWo/
[09:25:27] <vgutierrez>	 you can see there restbase cert VS rest-gateway one
[09:25:38] <hnowlan>	 good point, thanks for that
[09:26:00] <vgutierrez>	 hnowlan: basically any SAN listed in our unified cert should be there as well
[09:26:38] <vgutierrez>	 no problem and sorry I've missed this while checking your CR
[09:32:27] <hnowlan>	 vgutierrez: think this should cover it https://phabricator.wikimedia.org/P49482 I took the alt_names from the restbase.discovery.wmnet config
[09:33:00] <vgutierrez>	 nice :)
[09:48:31] <vgutierrez>	 we already finished with the eqsin upgrade, let me know when you're ready :)
[09:54:29] <hnowlan>	 cool, just rolling out the certs now 
[10:09:51] <hnowlan>	 vgutierrez: okay, rolled out - the request you were making earlier looks good 
[10:13:14] <vgutierrez>	 yep
[10:13:23] <vgutierrez>	 but https://rest-gateway.discovery.wmnet:4113/en.wikipedia.org/v1/pdf/Tornado/a4/desktop doesn't match what you got in the CR
[10:13:49] <vgutierrez>	  https://rest-gateway.discovery.wmnet:4113/api/rest_v1/$2/pdf/$3
[10:14:14] <hnowlan>	 will the glob in the CR not catch everything after /pdf/? 
[10:14:56] <vgutierrez>	 hnowlan: what about /v1/ in your sample request VS /api/rest_v1/ :?
[10:18:07] <hnowlan>	 vgutierrez: that's matching the pattern of the restbase URLs - the rb-mw-mangling.lua rewrites the path right? 
[10:20:20] <vgutierrez>	 hnowlan: hmm
[10:20:51] <vgutierrez>	 yep, you're right
[10:21:17] <hnowlan>	 ideally in future we'll support that natively in the rest-gateway and remove our reliance upon that 
[10:33:47] <vgutierrez>	 hnowlan: so.. let's disable puppet on A:cp-text and depool cp2037?
[10:34:16] <hnowlan>	 sounds good!
[10:34:43] <hnowlan>	 `sudo confctl select service=ats-be,cluster=cache_text,dc=codfw,name=cp2037.codfw.wmnet set/pooled=no` looks ok?
[10:35:28] <vgutierrez>	 you don't need cluster or dc keys
[10:35:35] <vgutierrez>	 name= is already more specific than that
[10:35:46] <hnowlan>	 ack
[10:36:55] <vgutierrez>	 hmmm
[10:37:00] <hnowlan>	 should I just use name and not service? I see it's also pooled for cdn 
[10:37:18] <vgutierrez>	 hnowlan: I've mentioned cluster and dc, not service :)
[10:37:26] <vgutierrez>	 service+name makes sense
[10:37:37] <vgutierrez>	 BTW... I see that rest-gatewayt doesn't send an ETag header
[10:37:45] <vgutierrez>	 is that a feature or a bug? :)
[10:38:21] <vgutierrez>	 even if it sends access-control-expose-headers: etag
[10:40:10] <hnowlan>	 vgutierrez: it's both, heh. restbase sends that header and it was requested that we match restbase behaviour as closely as is possible 
[10:40:22] <hnowlan>	 but restbase also doesn't set ETags 
[10:40:49] <vgutierrez>	 ook
[10:41:11] <hnowlan>	 seems like a good candidate to remove in future though
[10:42:05] <hnowlan>	 I've depooled cp2037, going to disable puppet 
[10:46:13] <vgutierrez>	 looking good :
[10:46:16] <vgutierrez>	 :)
[10:46:43] <hnowlan>	 gonna merge and enable puppet on cp2037 - do I need to do a reload or will puppet handle that? 
[10:47:07] <vgutierrez>	 puppet handles everything
[10:49:03] <vgutierrez>	 hnowlan: weird.. I saw the puppet run being triggered before the +2 message on -operations
[10:49:19] <vgutierrez>	 and indeed.. it isn't applying your commit :)
[10:49:28] <vgutierrez>	 Jun 27 10:49:00 cp2037 puppet-agent[2209342]: Applying configuration version '(ae599e626a) Alexandros Kosiaris - url_downloader: Remove the esams entries marked TODO'
[10:49:42] <hnowlan>	 vgutierrez: yep, that was me, just a habit. 
[10:50:12] <hnowlan>	 going for real now
[10:51:16] <vgutierrez>	 nice
[10:51:20] <vgutierrez>	 ATS reloaded as expected
[10:51:44] <vgutierrez>	 you can target ATS on 127.0.0.1:3128
[10:52:02] <vgutierrez>	 just set Host and X-Forwarded-Proto: https headers
[10:53:35] <hnowlan>	 looks okay.. I think! testing a bit more 
[10:54:19] <vgutierrez>	 hmmm
[10:54:22] <vgutierrez>	 CacheResultCode:ERR_CLIENT_READ_ERROR CacheWriteResult:- ReqMethod:GET RespStatus:200 OriginStatus:000
[10:54:33] <vgutierrez>	 can you share the curl output?
[10:54:51] <hnowlan>	 on cp2037: curl -H "Host: en.wikipedia.org" -H "X-Forwarded-Proto: https" 127.0.0.1:3128/api/rest_v1/page/pdf/Tornado 
[10:55:00] <hnowlan>	 just returns binary output as expected
[10:56:00] <vgutierrez>	 hnowlan: server: restbase2016 :)
[10:56:19] <vgutierrez>	 you're getting a cache hit on ATS for that URL
[10:56:26] <hnowlan>	 ahh.
[10:57:18] <vgutierrez>	 if you append something like ?hnowlan=1 to the URL you'll see that it's still hitting the old service
[10:58:22] <hnowlan>	 ahh, sigh
[11:03:30] <hnowlan>	 so the match isn't working at all I guess
[11:03:55] * vgutierrez double checking some stuff
[11:07:42] <vgutierrez>	 hnowlan: yep, that seems to be the case
[11:08:45] <hnowlan>	 I'll revert for now
[11:09:31] <hnowlan>	 https://gerrit.wikimedia.org/r/c/operations/puppet/+/933408 
[11:12:58] <vgutierrez>	 hnowlan: hmmm yeah, it looks like regex_map only supports regex on the Host header
[11:13:11] <vgutierrez>	 per https://docs.trafficserver.apache.org/en/9.1.x/admin-guide/files/remap.config.en.html?highlight=regex_map#regular-expression-regex-remap-support
[11:13:27] <vgutierrez>	 Only the host field can contain a regex; the scheme, port, and other fields cannot. For path manipulation via regexes, use the Regex Remap Plugin.
[11:14:11] <vgutierrez>	 and that's https://docs.trafficserver.apache.org/en/9.1.x/admin-guide/plugins/regex_remap.en.html#regex-remap-plugin
[11:14:43] <vgutierrez>	 hnowlan: you could also perform the redirection in lua
[11:15:14] <vgutierrez>	 probably safer cause you can provide an unittest for that use case
[11:16:02] <hnowlan>	 ahhhh, cool 
[11:16:08] <hnowlan>	 yeah that'd be ideal 
[11:18:30] <hnowlan>	 I'll give that a go so 
[11:18:35] <hnowlan>	 reverted on cp2037 
[11:18:42] <vgutierrez>	 hnowlan: you could use https://gerrit.wikimedia.org/r/c/operations/puppet/+/900704 as an example
[11:19:02] <vgutierrez>	 ping me if you need help to come with a PoC
[11:19:36] <hnowlan>	 vgutierrez: oooh yeah, that would make it easier to migrate other services gradually too once there's a working version in place 
[11:20:21] <hnowlan>	 I've repooled cp2037 and I'll reenable puppet now. Thanks for all the help! 
[11:21:31] <vgutierrez>	 they do the matching based on the Host header
[11:21:34] <vgutierrez>	 you need to use https://docs.trafficserver.apache.org/en/9.1.x/admin-guide/plugins/lua.en.html#ts-client-request-get-uri
[14:11:48] <wikibugs>	 10Traffic, 10MW-on-K8s, 10SRE, 10serviceops, and 3 others: Migrate group0 to Kubernetes - https://phabricator.wikimedia.org/T337490 (10Clement_Goubert)
[14:14:24] <wikibugs>	 10Traffic, 10MW-on-K8s, 10SRE, 10serviceops, and 2 others: Migrate group1 to Kubernetes - https://phabricator.wikimedia.org/T340549 (10Clement_Goubert)
[14:14:50] <wikibugs>	 10Traffic, 10MW-on-K8s, 10SRE, 10serviceops, and 2 others: Migrate group1 to Kubernetes - https://phabricator.wikimedia.org/T340549 (10Clement_Goubert) p:05Triage→03Medium
[15:08:44] <wikibugs>	 10Traffic, 10Patch-For-Review: Write a cookbook to handle upgrades of ATS - https://phabricator.wikimedia.org/T335531 (10BCornwall) 05Open→03Resolved
[19:13:20] <wikibugs>	 10netops, 10Data-Engineering, 10Infrastructure-Foundations, 10Product-Analytics, and 2 others: Maybe restrict domains accessible by webproxy - https://phabricator.wikimedia.org/T300977 (10leila)
[19:13:29] <wikibugs>	 10netops, 10Data-Engineering, 10Infrastructure-Foundations, 10Product-Analytics, and 2 others: Maybe restrict domains accessible by webproxy - https://phabricator.wikimedia.org/T300977 (10leila) I'm going to remove this task from the Backlog lane of the #Research board given that there is no task for Resea...