[01:09:32] <wikibugs>	 06Traffic, 10MobileFrontend, 10MediaWiki-Platform-Team (Radar), 13Patch-For-Review: MobileFrontend should declare "X-Subdomain" variance via "Vary" response header - https://phabricator.wikimedia.org/T390929#10933402 (10Krinkle) >>! In T390929#10930426, @gerritbot wrote: > Change #1161168 had a related pat...
[01:14:17] <wikibugs>	 06Traffic, 10MobileFrontend, 10MediaWiki-Platform-Team (Radar), 13Patch-For-Review: MobileFrontend should declare "X-Subdomain" variance via "Vary" response header - https://phabricator.wikimedia.org/T390929#10933403 (10Krinkle)
[02:04:55] <Krinkle>	 sukhe: ping re https://gerrit.wikimedia.org/r/c/operations/puppet/+/1160381. can we do it tomorrow? (I'm in -8/PST timezone at the moment)
[02:05:42] <Krinkle>	 PDT* (I keep forgetting that the "S" stands for Standard and not Summer)
[02:54:16] <wikibugs>	 06Traffic, 10MobileFrontend, 10MediaWiki-Platform-Team (Radar), 13Patch-For-Review: MobileFrontend should declare "X-Subdomain" variance via "Vary" response header - https://phabricator.wikimedia.org/T390929#10933436 (10Krinkle) In my audit of [Codesearch for shouldDisplayMobileView OR usingMobileDomain](h...
[13:14:45] <sukhe>	 Krinkle: we usually don't do VCL deploys on Friday but since this is just appending something to X-Analytics and not a change in other logic, it should be OK. but I will let you be the judge of whether you want to do it today or Monday :)
[13:15:05] <sukhe>	 in any case, please ping me when online and if you want to do it today, happy to roll it out (same for Monday)
[13:30:17] <inflatador>	 vgutierrez no rush or anything, but we think we fixed the elastic code so it sets up LVS correctly. I moved T387569 into "Needs Review" so you can verify whenever
[13:30:17] <stashbot>	 T387569: Update Elastic puppet code to filter LVS config based on cluster membership - https://phabricator.wikimedia.org/T387569
[13:30:34] <vgutierrez>	 inflatador: thx, will check soon(TM)
[13:39:19] <inflatador>	 ACK, np
[14:36:10] <wikibugs>	 06Traffic, 06Infrastructure-Foundations, 06SRE: Avoid using codfw expansion cage for non-IPIP LVS-fronted services - https://phabricator.wikimedia.org/T394286#10934684 (10cmooney) 05Open→03Resolved a:03cmooney Gonna close this one, I trust you guys have the heads up on what can't go there just yet.
[15:01:58] <vgutierrez>	 inflatador: so... if I understood correctly the tasks that I flagged as blocked now should be unblocked and we could proceed with the IPIP migration, right?
[15:18:02] <inflatador>	 vgutierrez yeah, if the current changes look OK, I can get started on the IPIP patches and CC on them if that works
[15:18:35] <vgutierrez>	 inflatador: I'll rebase https://gerrit.wikimedia.org/r/c/operations/puppet/+/1123652
[15:22:03] <inflatador>	 vgutierrez we'll need to target a different role since we just migrated off elasticsearch. I can take over the patch if you like
[15:29:45] <wikibugs>	 06Traffic, 10Liberica: unable to fetch forwarding plane metrics when using katran - https://phabricator.wikimedia.org/T396227#10934880 (10Vgutierrez) 05Open→03Resolved
[18:51:23] <Krinkle>	 sukhe: pong. Today would be great yeah (was hoping Wed or Thu, but I pinged you too late in the day both times). Monday we can start comparing the data for next steps in AQS.
[19:08:08] <sukhe>	 ok cool
[19:08:11] <sukhe>	 let's do it now then
[19:09:31] <sukhe>	 do you have a way of testing it if I merge it on one host in ulsfo? (since I presume you are there)
[19:09:41] <sukhe>	 you can let me know which host you are hitting and I can merge there
[19:09:49] <sukhe>	 otherwise I will just do a canary host and then full rollout
[19:09:50] <sukhe>	 Krinkle: ^
[19:14:10] <Krinkle>	 sukhe: yes, I can test it.
[19:14:41] <Krinkle>	 I have sudo on varnishes so you pick and I'll test there. Assuming curl is installed.
[19:15:17] <sukhe>	 yes curl should be there everywhere
[19:15:31] <sukhe>	 I am testing on cp4037 to see if varnish is happy and will roll it out everywhere else after
[19:18:36] <sukhe>	 looks good, rolling it out everywhere
[19:23:21] <Krinkle>	 Hm.. not seeing the result. It may be getting stripped before I see can it, but maybe it's a no-op.
[19:23:23] <Krinkle>	 krinkle@cp4037:~$ curl -vI --connect-to ::$HOSTNAME 'https://en.wikipedia.org/wiki/Special:BlankPage' -H 'X-Wikimedia-Debug: 1' -H 'User-Agent: Krinkle Mobile';
[19:23:41] <Krinkle>	 HTTP/2 302 
[19:23:41] <Krinkle>	 x-analytics: https=1;debug=1;client_port=7205;nocookies=1
[19:23:41] <Krinkle>	 location: https://en.m.wikipedia.org/wiki/Special:BlankPage
[19:23:56] <Krinkle>	 (debug to preserve x-analytics)
[19:25:34] <Krinkle>	 I'll have a closer look with varnishlog in a bit to see if it does get set for normal requests
[19:25:41] <Krinkle>	 I'll write a vcl test as well for puppet
[19:26:17] <Krinkle>	 I mean the order of operations must be right, surely vcl_deliver can't run before vcl_recv
[19:27:46] <Krinkle>	 Well, no because it worked in beta.
[19:29:19] <Krinkle>	 OK, I got it. I was testing the redirect rather than the m-dot request itself. 
[19:29:23] <Krinkle>	 All good.
[19:29:46] <Krinkle>	 Thanks sukhe!
[19:29:58] <sukhe>	 yeah, running the above with .m does indeed give is `ismobile=1'
[19:30:12] <sukhe>	 no worries, rolling it out elsewhere slowly in batches of 11 for a total of 112 hosts. I will ping you when done.
[19:41:58] <sukhe>	 Krinkle: all done.
[19:46:55] <Krinkle>	 Ack
[20:51:57] <wikibugs>	 06Traffic, 06Data-Engineering, 10MediaWiki-Platform-Team (Radar): Add ismobile attribute to X-Analytics header - https://phabricator.wikimedia.org/T390924#10935663 (10Krinkle) 05Open→03Resolved This is now live.  ` $ curl -I 'https://en.m.wikipedia.org/wiki/Main_Page' -H 'X-Wikimedia-Debug: 1' … x-an...