[08:58:37] 10Traffic: Webrequests live data shows traffic without TLS on varnish for upload.w.o - https://phabricator.wikimedia.org/T340097 (10Volans) [09:14:38] 10Traffic: Webrequests live data shows traffic without TLS on varnish for upload.w.o - https://phabricator.wikimedia.org/T340097 (10Vgutierrez) 05Open→03In progress p:05Triage→03Medium [09:17:43] 10Traffic: Let HAProxy handle port 80 - https://phabricator.wikimedia.org/T323557 (10Vgutierrez) [09:17:57] 10Traffic, 10SRE: port 80 paging on scheduled single host maintenance in text@esams - https://phabricator.wikimedia.org/T339898 (10Vgutierrez) 05Open→03Resolved Mitigated by tightening port 80 timeouts (https://gerrit.wikimedia.org/r/c/operations/puppet/+/932173/1/hieradata/common/profile/cache/haproxy.yam... [09:18:14] 10Traffic: Webrequests live data shows traffic without TLS on varnish for upload.w.o - https://phabricator.wikimedia.org/T340097 (10Vgutierrez) [09:18:17] 10Traffic: Let HAProxy handle port 80 - https://phabricator.wikimedia.org/T323557 (10Vgutierrez) [09:24:29] 10Traffic: Webrequests live data shows traffic without TLS on varnish for upload.w.o - https://phabricator.wikimedia.org/T340097 (10Vgutierrez) [09:24:31] 10Traffic: Let HAProxy handle port 80 - https://phabricator.wikimedia.org/T323557 (10Vgutierrez) [09:38:49] 10Traffic, 10SRE: Webrequests live data shows traffic without TLS on varnish for upload.w.o - https://phabricator.wikimedia.org/T340097 (10Vgutierrez) This seems like a varnish (VCL) bug. Varnish is getting requests with X-Connection-Properties header set but it's failing to issue the expected X-Analytics-TLS... [09:48:10] 10Traffic, 10SRE: Webrequests live data shows traffic without TLS on varnish for upload.w.o - https://phabricator.wikimedia.org/T340097 (10Vgutierrez) Full log of a request showing the misbehaviour: `counterexample * << Request >> 713485145... [09:55:48] 10Traffic, 10SRE: Webrequests live data shows traffic without TLS on varnish for upload.w.o - https://phabricator.wikimedia.org/T340097 (10Vgutierrez) as shown on the full request example, this is happening on request restarts: `Begin req 713485144 restart` and our VCL logic excludes from TLS data be... [10:01:36] 10Traffic, 10SRE: Webrequests live data shows traffic without TLS on varnish for upload.w.o - https://phabricator.wikimedia.org/T340097 (10Vgutierrez) Regarding healthcheck.wikimedia.org those are actually plain text requests being issued by the UDS healthcheck: ` * << Request >> 621520726 - Begin... [11:05:59] hello there, FYI I've added to the DNS wikitech page (recursors section) a paragraph to use the sre.dns.wipe-cache cookbook in addition to the manual steps [11:06:04] feel free to amend at will: https://wikitech.wikimedia.org/w/index.php?title=DNS&diff=2087499&oldid=2079372 [12:31:40] Hi folks, another day another rollout of varnishkafka - https://gerrit.wikimedia.org/r/c/operations/puppet/+/932217 [12:31:46] lemme know if it is ok (eqiad) [12:44:48] 10netops, 10Infrastructure-Foundations: Configure bgp-error-tolerance on Juniper routers - https://phabricator.wikimedia.org/T340111 (10ayounsi) [13:01:59] 10Traffic, 10Mobile-Content-Service, 10Product-Infrastructure-Team-Backlog-Deprecated, 10RESTbase Sunsetting, and 2 others: Setup allowed list for MCS decom - https://phabricator.wikimedia.org/T340036 (10MSantos) >>! In T340036#8952843, @akosiaris wrote: >>>! In T340036#8952836, @MSantos wrote: >> Sounds g... [13:51:37] eqiad varnishkafkas restarted, all good [14:02:50] 10Traffic, 10SRE: Webrequests live data shows traffic without TLS on varnish for upload.w.o - https://phabricator.wikimedia.org/T340097 (10Vgutierrez) @Volans even if this is the expected behavior right now we need to clarify the dashboards a little bit. The first scenario (upload.wm.o and req.restarts >= 1) s... [14:03:08] elukey: lovely [14:30:35] 10Traffic, 10Content-Transform-Team-WIP, 10Mobile-Content-Service, 10RESTbase Sunsetting, and 2 others: Setup allowed list for MCS decom - https://phabricator.wikimedia.org/T340036 (10MSantos) [14:31:26] 10Traffic, 10SRE: Webrequests live data shows traffic without TLS on varnish for upload.w.o - https://phabricator.wikimedia.org/T340097 (10Volans) @Vgutierrez I don't see that field in druid so I think we have to check if that's available when benthos parses the stream and set a field for it. This for the live... [15:23:39] 10Traffic, 10Content-Transform-Team-WIP, 10Mobile-Content-Service, 10RESTbase Sunsetting, and 2 others: Setup allowed list for MCS decom - https://phabricator.wikimedia.org/T340036 (10akosiaris) Rules created, but NOT enabled, The corresponding VCL is ` // FILTER T340036 // Give wikiwand and kiwix an ext... [15:56:57] 10Traffic, 10Content-Transform-Team-WIP, 10Mobile-Content-Service, 10RESTbase Sunsetting, and 2 others: Setup allowed list for MCS decom - https://phabricator.wikimedia.org/T340036 (10MSantos) @akosiaris the deadline we defined for the deprecation is July 1st 2023, we can flip the switch then. Does the ru... [17:30:09] 10Traffic, 10Patch-For-Review: Write a cookbook to handle restarts of Wikimedia DNS - https://phabricator.wikimedia.org/T335533 (10BCornwall) 05In progress→03Resolved [19:19:30] sukhe: got a librenms alert about doh6001: Not accepting/receiving prefixes from anycast BGP peer [19:22:51] XioNoX: we did a restart today so that's the correlation but still doesn't explain why :( [19:23:13] glad that the alert works, just saw it [19:23:16] yeah [19:23:17] https://grafana.wikimedia.org/d/dxbfeGDZk/anycast?orgId=1&var-protocol=BGP&var-site=drmrs&var-cluster=doh&var-ip_version=All [19:23:27] says it exports the prefixes [19:23:36] thanks, will check when I am back at the computer [19:23:42] so I guess it's another case of rebooting bird will solve it, but it's not great [19:25:21] we have a patch for restarting bird automatically [19:25:23] but yeah [19:25:27] (systemd binding) [19:25:32] thanks for the ping [22:29:29] 10Traffic, 10Content-Transform-Team-WIP, 10Mobile-Content-Service, 10RESTbase Sunsetting, and 2 others: Setup allowed list for MCS decom - https://phabricator.wikimedia.org/T340036 (10akosiaris) >>! In T340036#8956346, @MSantos wrote: > @akosiaris the deadline we defined for the deprecation is July 1st 202...