[08:14:50] 10Traffic: Investigate IPVS IPIP encapsulation support - https://phabricator.wikimedia.org/T348837 (10Vgutierrez) [08:15:27] 10Traffic: Investigate IPVS IPIP encapsulation support - https://phabricator.wikimedia.org/T348837 (10Vgutierrez) [08:15:30] 10Traffic: Replace current L4LB with with Katran-based alternative - https://phabricator.wikimedia.org/T332027 (10Vgutierrez) [08:15:35] 10Traffic: Investigate IPVS IPIP encapsulation support - https://phabricator.wikimedia.org/T348837 (10Vgutierrez) p:05Triage→03Medium [08:19:23] 10Traffic: Investigate IPVS IPIP encapsulation support - https://phabricator.wikimedia.org/T348837 (10ayounsi) [08:50:32] 10HTTPS, 10MediaWiki-Action-API, 10MediaWiki-extensions-CentralAuth: Central login notice appears on unencrypted API format=*fm pages, where reloading does not affect login status - https://phabricator.wikimedia.org/T94125 (10matmarex) 05Open→03Declined This only affects CentralAuth in a non-WMF configur... [09:34:56] 10Traffic, 10SRE: Investigate IPVS IPIP encapsulation support - https://phabricator.wikimedia.org/T348837 (10Southparkfan) Alternative to consider: injecting REDIRECTs for traffic meant for a VIP. See the second section at http://www.linuxvirtualserver.org/docs/arp.html. I haven't tested it and it requires som... [10:16:33] 10netops, 10Cloud-VPS, 10Infrastructure-Foundations, 10SRE: Change cloud-instance-transport vlan subnets from /30 to /29 - https://phabricator.wikimedia.org/T348140 (10cmooney) Codfw equivalent subnet that needs changing also: ` cmooney@cloudcontrol2005-dev:~$ sudo wmcs-openstack subnet show 2596edb4-5a40-... [10:35:32] 10HTTPS, 10SRE, 10Traffic-Icebox, 10Wikimedia-Performance-recommendation: Enable HTTP/3 (QUIC) support on Wikimedia servers - https://phabricator.wikimedia.org/T238034 (10Diskdance) [11:58:02] 10Traffic, 10SRE: Add custom HAProxy backend only for healthchecks - https://phabricator.wikimedia.org/T348851 (10Fabfur) [12:35:45] 10Traffic, 10SRE: Add custom HAProxy backend only for healthchecks - https://phabricator.wikimedia.org/T348851 (10Vgutierrez) [12:40:05] 10Traffic, 10SRE: Investigate IPVS IPIP encapsulation support - https://phabricator.wikimedia.org/T348837 (10Vgutierrez) [12:44:10] 10netops, 10Cloud-VPS, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: Change cloud-instance-transport vlan subnets from /30 to /29 - https://phabricator.wikimedia.org/T348140 (10dcaro) p:05Low→03High This is causing some issues, should be fixed sooner than later, bumping priority [12:50:28] 10netops, 10Cloud-VPS, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: Change cloud-instance-transport vlan subnets from /30 to /29 - https://phabricator.wikimedia.org/T348140 (10dcaro) [12:54:34] 10Traffic, 10SRE: Add custom HAProxy backend only for healthchecks - https://phabricator.wikimedia.org/T348851 (10Fabfur) [13:39:01] 10Traffic, 10SRE: Investigate IPVS IPIP encapsulation support - https://phabricator.wikimedia.org/T348837 (10Vgutierrez) >>! In T348837#9249192, @Southparkfan wrote: > Alternative to consider: injecting REDIRECTs for traffic meant for a VIP. See the second section at http://www.linuxvirtualserver.org/docs/arp.... [15:06:03] 10Traffic, 10SRE: Investigate IPVS IPIP encapsulation support - https://phabricator.wikimedia.org/T348837 (10jhathaway) @Vgutierrez thanks for opening this ticket and investigating ipip support in ipvs. Another alternative would be [[ https://datatracker.ietf.org/doc/html/draft-ietf-intarea-gue-06 | GUE ]] enc... [15:19:31] vgutierrez: a few coffees later, I now see why the netfilter solution solves a problem that was already solved. Sorry ;-) [15:19:50] Southparkfan: no problem, thanks for your feedback BTW :) [15:19:54] Got confused. The MTU problem still exists, though [15:19:58] indeed [15:20:25] I tried IPVS w/ IPIP a year ago or something. Couldn't get it working [15:20:50] But I guess nowadays, there are other, better L4LBs out there I should try [15:20:57] the Vagrantfile attached to the phab task that I wrote this morning works [for me, here] [15:21:37] we are considering some XDP alternatives to IPVS [15:21:59] Neat :) [15:22:00] those require moving from L2 to L3 and use some kind of encapsulation to reach the real servers [15:22:34] so switching first from L2 to L3 in IPVS will help reducing the risks and the amount of things that we do differently between the old and the new world [15:24:59] Better to avoid the big bang indeed [15:26:48] 10Traffic, 10DC-Ops, 10SRE, 10ops-eqiad: Q1:rack/setup/install cp11[00-15] - https://phabricator.wikimedia.org/T342159 (10VRiley-WMF) [15:26:59] If katran doesn't require writing eBPF programs myself, I can try it... [15:29:02] katran provides the XDP/eBPF programs and a C++ control plane on top of folly [15:29:28] we are considering the XDP part of katran + https://github.com/cilium/ebpf [15:29:43] so instead of using the control plane provided by katran use our own written in go [15:30:24] super cool [15:33:08] 10Traffic, 10DC-Ops, 10SRE, 10ops-eqiad: Q1:rack/setup/install cp11[00-15] - https://phabricator.wikimedia.org/T342159 (10VRiley-WMF) [16:08:56] 10Traffic: Investigate why Traffic SLO Grafana dashboard has negative values on combined SLI - https://phabricator.wikimedia.org/T341606 (10herron) >>! In T341606#9248134, @BCornwall wrote: > @herron Thanks for all of your help. We've implemented varnish_sli_bad. I followed the formulae presented at the top of... [16:26:36] 10Traffic, 10DC-Ops, 10SRE, 10ops-eqiad: Q1:rack/setup/install cp11[00-15] - https://phabricator.wikimedia.org/T342159 (10VRiley-WMF) [17:00:40] 10Traffic, 10DC-Ops, 10SRE, 10ops-eqiad, 10Patch-For-Review: Q1:rack/setup/install cp11[00-15] - https://phabricator.wikimedia.org/T342159 (10Jclark-ctr) a:05ssingh→03Jclark-ctr [17:10:28] 10Traffic, 10DC-Ops, 10SRE, 10ops-eqiad, 10Patch-For-Review: Q1:rack/setup/install cp11[00-15] - https://phabricator.wikimedia.org/T342159 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1001 for host cp1115.eqiad.wmnet with OS bullseye [18:03:18] 10Traffic, 10DC-Ops, 10SRE, 10ops-eqiad: Q1:rack/setup/install cp11[00-15] - https://phabricator.wikimedia.org/T342159 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1001 for host cp1115.eqiad.wmnet with OS bullseye completed: - cp1115 (**PASS**) - Removed from Puppet... [19:21:11] 10Traffic, 10DC-Ops, 10SRE, 10ops-eqiad: Q1:rack/setup/install cp11[00-15] - https://phabricator.wikimedia.org/T342159 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1001 for host cp1114.eqiad.wmnet with OS bullseye [19:24:09] 10Traffic, 10DC-Ops, 10SRE, 10ops-eqiad: Q1:rack/setup/install cp11[00-15] - https://phabricator.wikimedia.org/T342159 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1001 for host cp1112.eqiad.wmnet with OS bullseye [19:29:00] 10Traffic, 10DC-Ops, 10SRE, 10ops-eqiad: Q1:rack/setup/install cp11[00-15] - https://phabricator.wikimedia.org/T342159 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1001 for host cp1109.eqiad.wmnet with OS bullseye [19:30:28] 10Traffic, 10DC-Ops, 10SRE, 10ops-eqiad: Q1:rack/setup/install cp11[00-15] - https://phabricator.wikimedia.org/T342159 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1001 for host cp1108.eqiad.wmnet with OS bullseye [19:48:46] 10Traffic, 10DC-Ops, 10SRE, 10ops-eqiad: Q1:rack/setup/install cp11[00-15] - https://phabricator.wikimedia.org/T342159 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1001 for host cp1113.eqiad.wmnet with OS bullseye [19:52:12] 10Traffic, 10DC-Ops, 10SRE, 10ops-eqiad: Q1:rack/setup/install cp11[00-15] - https://phabricator.wikimedia.org/T342159 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1001 for host cp1107.eqiad.wmnet with OS bullseye [19:55:23] 10Traffic, 10DC-Ops, 10SRE, 10ops-eqiad: Q1:rack/setup/install cp11[00-15] - https://phabricator.wikimedia.org/T342159 (10Jclark-ctr) [19:57:10] 10Traffic, 10DC-Ops, 10SRE, 10ops-eqiad: Q1:rack/setup/install cp11[00-15] - https://phabricator.wikimedia.org/T342159 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1001 for host cp1111.eqiad.wmnet with OS bullseye [20:04:52] 10Traffic, 10DC-Ops, 10SRE, 10ops-eqiad: Q1:rack/setup/install cp11[00-15] - https://phabricator.wikimedia.org/T342159 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1001 for host cp1109.eqiad.wmnet with OS bullseye completed: - cp1109 (**PASS**) - Removed from Puppet... [20:05:33] 10Traffic, 10DC-Ops, 10SRE, 10ops-eqiad: Q1:rack/setup/install cp11[00-15] - https://phabricator.wikimedia.org/T342159 (10Jclark-ctr) [20:11:36] 10Traffic, 10DC-Ops, 10SRE, 10ops-eqiad: Q1:rack/setup/install cp11[00-15] - https://phabricator.wikimedia.org/T342159 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1001 for host cp1112.eqiad.wmnet with OS bullseye completed: - cp1112 (**WARN**) - Removed from Puppet... [20:11:41] 10Traffic, 10DC-Ops, 10SRE, 10ops-eqiad: Q1:rack/setup/install cp11[00-15] - https://phabricator.wikimedia.org/T342159 (10Jclark-ctr) [20:12:28] 10Traffic, 10DC-Ops, 10SRE, 10ops-eqiad: Q1:rack/setup/install cp11[00-15] - https://phabricator.wikimedia.org/T342159 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1001 for host cp1106.eqiad.wmnet with OS bullseye [20:12:34] 10Traffic, 10DC-Ops, 10SRE, 10ops-eqiad: Q1:rack/setup/install cp11[00-15] - https://phabricator.wikimedia.org/T342159 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1001 for host cp1107.eqiad.wmnet with OS bullseye [20:15:49] 10Traffic, 10DC-Ops, 10SRE, 10ops-eqiad: Q1:rack/setup/install cp11[00-15] - https://phabricator.wikimedia.org/T342159 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1001 for host cp1107.eqiad.wmnet with OS bullseye executed with errors: - cp1107 (**FAIL**) - Removed f... [20:23:12] 10Traffic, 10DC-Ops, 10SRE, 10ops-eqiad: Q1:rack/setup/install cp11[00-15] - https://phabricator.wikimedia.org/T342159 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1001 for host cp1108.eqiad.wmnet with OS bullseye executed with errors: - cp1108 (**FAIL**) - Removed f... [20:28:30] 10Traffic, 10DC-Ops, 10SRE, 10ops-eqiad: Q1:rack/setup/install cp11[00-15] - https://phabricator.wikimedia.org/T342159 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1001 for host cp1113.eqiad.wmnet with OS bullseye completed: - cp1113 (**PASS**) - Removed from Puppet... [20:28:50] 10Traffic, 10DC-Ops, 10SRE, 10ops-eqiad: Q1:rack/setup/install cp11[00-15] - https://phabricator.wikimedia.org/T342159 (10Jclark-ctr) [20:41:26] 10Traffic, 10DC-Ops, 10SRE, 10ops-eqiad: Q1:rack/setup/install cp11[00-15] - https://phabricator.wikimedia.org/T342159 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1001 for host cp1114.eqiad.wmnet with OS bullseye executed with errors: - cp1114 (**FAIL**) - Removed f... [20:49:41] 10Traffic, 10DC-Ops, 10SRE, 10ops-eqiad: Q1:rack/setup/install cp11[00-15] - https://phabricator.wikimedia.org/T342159 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1001 for host cp1111.eqiad.wmnet with OS bullseye executed with errors: - cp1111 (**FAIL**) - Removed f... [21:32:42] 10Traffic, 10DC-Ops, 10SRE, 10ops-eqiad: Q1:rack/setup/install cp11[00-15] - https://phabricator.wikimedia.org/T342159 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1001 for host cp1106.eqiad.wmnet with OS bullseye executed with errors: - cp1106 (**FAIL**) - Removed f... [21:32:49] 10Traffic, 10DC-Ops, 10SRE, 10ops-eqiad: Q1:rack/setup/install cp11[00-15] - https://phabricator.wikimedia.org/T342159 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1001 for host cp1107.eqiad.wmnet with OS bullseye executed with errors: - cp1107 (**FAIL**) - Downtimed...