[05:08:47] Looks like a mgmt switch went down? [05:09:03] Seems to be in row D in eqiad [05:12:51] Created https://phabricator.wikimedia.org/T394258 [06:03:23] marostegui: https://phabricator.wikimedia.org/T394109 [06:03:51] Ah! [06:03:58] I will close it then :) [06:04:11] Ah you did already, thank you! [07:38:52] marostegui, volans, I'm depooling esams for router upgrade [07:39:00] XioNoX: thanks for the heads up [07:40:24] sure thing [07:56:37] i'm guessing that router upgrade is why i can't connect to bast3007? [07:57:55] nevermind, now it works again [08:00:17] <_joe_> volans, marostegui I'm upgrading requestctl; I will need to run a schema upgrade so for a couple minutes you might get weird messages from the cli and the web ui [08:00:30] _joe_: thanks [08:00:37] ack [08:04:51] <_joe_> schema upgrade done; if you need to use requestctl, please use the cli on puppetserver1001 until further notice [08:08:01] <_joe_> {{done}} the UI is usable again. You'll see that all of our rules seem to be created at the start of the unix epoch, that will change soon-ish, but for now bear with me :) [08:08:22] lol, ok [08:08:58] <_joe_> volans: I want to put a sensible value there from the git history of the git log of changes to requestctl :) [08:09:20] <_joe_> but that requires git horrors and bash one-liners that would make cdanis proud [08:09:39] ahahah [08:13:05] <_joe_> oh also, the "priority" field I just added to actions [08:13:26] <_joe_> for now it has no influence on the generated output :D [08:14:00] rebooting cr2-esams (no impact expected except some alerting noise) [08:14:02] <_joe_> that's on purpose, I'm still unsure how we want to do it [08:14:39] <_joe_> but I wanted to do all schema changes in one go [08:24:02] mgmt is up, waiting for prod interfaces [08:24:58] and up [08:35:00] all done with cr2, going to start with cr1 [08:35:25] ack [08:35:40] cr1 have 2 routing engines so it will take a bit longer [09:05:17] cr1:re1 is back, doing the switchover now [09:06:14] k [09:06:47] 👍 [09:08:51] waiting for the ports to come back up [09:13:55] up, now doing the same for re0 [09:29:59] once again, waiting for the ports to come back up, should be faster now as the PFE is already running the same version as re0 [09:39:27] all done with cr1, doing the cleanup and putting back into service [09:49:15] volans, marostegui, fabfur all done! repooling esams [09:49:23] XioNoX: ack! [09:49:26] cc topranks ^ [09:49:28] 👍 thanks [09:50:15] great, thanlks XioNoX [09:50:34] that was smooth, makes me want to do the same without even depooling the site :) [09:50:39] nice work XioNoX ! [09:52:04] haha yep, I was only thinking what a coward you were :P [09:52:22] hahaha [11:13:48] <_joe_> heads up: I'm updating the metadata of requestctl objects. [11:20:58] <_joe_> {{done}} [11:39:59] FYI, I've failed over the kadmin server to krb1002.eqiad.wmnet (the server where to run kadmin commands, e.g. to generate new keytabs) [12:51:41] topranks: I was about to answer at https://phabricator.wikimedia.org/T389375#10821374 (yes, that's my recollection too, at least after the edit), however as I was rethinking it a bit to refresh my memory, one question came to mind. The return traffic isn't going to be egressing via the ipip0 interface, right? The wikikube hosts will be returning [12:51:41] straight to the client, right? [12:51:52] vgutierrez too: ^ [12:52:16] given they have the VIP configured on their lo interfaces and all [12:58:12] arnoldokoth: correct, we're still doing direct return [12:59:04] er akosiaris ^ [12:59:45] yes the return still goes direct [13:00:47] cool. That confirms my world model. And now the $1M question. Why does the return path, that has no encapsulation whatsoever, require an MTU that is less than 1500 ? [13:01:21] akosiaris: it doesn't [13:01:51] I sense a but ... [13:02:14] the client sends its own MSS, and the server sends the return packet to fits it's own MTU and the client MSS (so the lowest of both) [13:02:19] the key thing is the client should not send a packet bigger than 1480 [13:02:22] no but :) [13:02:41] limiting the MTU on the server side causes it to send lower MSS, and thus ensures the client sticks to that [13:04:21] to my mind it's an elegant way to do it, and the 20 byte smaller packets not much of a hit to efficiency [13:04:31] +1 [13:05:16] but, each direction of a data flow can use a different MSS AFAIK. [13:05:27] the virtual tube needs a consistent MTU like it's one tube, even though it's two tubes (Ted Stevens version) [13:06:25] akosiaris: indeed. so if you wish you could leave MTUs at 1500, and just mangle the MSS in one direction [13:06:39] this would keep server -> client packets at 1500 [13:09:45] OK, I might be misunderstanding something here, but we are going to lower the MTU on the server side, to hint the client side to use a lower MTU ? Why would the client honor that hint? What am I missing? [13:10:23] it's the TCP handshake [13:10:57] the client says my MSS is X, the server says my MSS is Y (both based on their own MTU) [13:12:08] It's in the TCP spec. The server says "don't send more than X to me I won't get it". The client needs to respect that if it wants its data to be received. [13:12:32] so if the server MTU is 1400, it will tell the client its MSS is 1380, and the client won't send larger TCP packets [13:14:49] vgutierrez: while we're on the topic, based on your last comment [13:15:23] is it the case that if the LVS forwards an IPv6 packet over IPIP it uses IPv6 to reach the realserver? [13:15:38] and conversely it uses IPv4 encapsulation if the packet being wrapped up is IPv4? [13:16:33] another thing to keep in mind, is that we already clamp the MSS in the sites we have Cloudflare ready to go: https://github.com/wikimedia/operations-homer-public/blob/8f9023921e71c64c30cde3210463afc03d38c475/config/sites.yaml#L44 [13:17:28] ah, the TCP MSS option in the SYN and SYN/ACK packets. Had to refresh my memory [13:17:29] yep, though that only affects external/internet flows [13:18:06] note that this only decreases the MSS, never increase it, but reallistically most our users see a MSS modified in flight by our routers, and not the one sent by the servers [13:18:36] topranks: yeah, just to point out that it works and it's respected/common [13:38:35] topranks: yes, that's right [13:38:51] vgutierrez: cool, thanks for confirming [13:39:38] The POD MTU should be 1460 in that case. Probably easier to go with that now even if they aren't using v6. [13:42:31] yeah.. all low traffic servers are ipv4 only afaik [13:42:38] *services [14:11:48] <_joe_> akosiaris: btw, k8s-to-k8s requests don't go via LVS iirc, unless they're going cross-dc [14:13:12] <_joe_> because the LVS IP is present on the k8s worker node, so the request goes pod -> nodeport on the node -> kube-proxy [14:17:38] yeah, a packet never leaves the k8s node with the LVS IP as the destination IP if the traffic is for a service in the cluster. It will get DNATed immediately upon egress from the pod to the node and go straight to the chosen destination pod. [14:33:15] I heard ChatGPT would replace us all soon. Just tell it to configure all this stuff to work correctly, it'll be fine :P [15:11:28] * inflatador suddenly remembers the Nokia presentation where they had ChatGPT integrated into their devices