[07:08:38] 10netops, 10Data-Persistence, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate servers in codfw rack B5 from asw-b5-codfw to lsw1-b5-codfw - https://phabricator.wikimedia.org/T355549 (10Marostegui) [07:13:56] 10netops, 10Data-Persistence, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate servers in codfw rack B5 from asw-b5-codfw to lsw1-b5-codfw - https://phabricator.wikimedia.org/T355549 (10Marostegui) Database hosts are depooled - @cmooney confirm if you will downtime them or if I should do it myself [08:34:34] 10Traffic, 10Security-Team, 10WMF-General-or-Unknown, 10ContentSecurityPolicy, 10Patch-Needs-Improvement: Add restrictive CSP to upload.wikimedia.org - https://phabricator.wikimedia.org/T117618 (10TheDJ) >>! In T117618#8637047, @Bawolff wrote: > Got deployed to `/wikipedia/(el|fr|ru|it|de|uk|ja|id|he|fi|... [08:59:16] 10Traffic, 10Security-Team, 10WMF-General-or-Unknown, 10ContentSecurityPolicy, 10Patch-Needs-Improvement: Add restrictive CSP to upload.wikimedia.org - https://phabricator.wikimedia.org/T117618 (10Bawolff) > So why did this stall ? For reference, this was around the time when i left my job at Wikimedia.... [10:05:25] 10Traffic, 10Security-Team, 10WMF-General-or-Unknown, 10ContentSecurityPolicy, 10Patch-Needs-Improvement: Add restrictive CSP to upload.wikimedia.org - https://phabricator.wikimedia.org/T117618 (10TheDJ) https://logstash.wikimedia.org/goto/02dea35a7640ce8ad1babf9674174ed4 Dashboard ContentSecurityPolicy,... [10:21:26] 10netops, 10Data-Persistence, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate servers in codfw rack B5 from asw-b5-codfw to lsw1-b5-codfw - https://phabricator.wikimedia.org/T355549 (10cmooney) >>! In T355549#9487462, @Marostegui wrote: > Database hosts are depooled - @cmooney confirm if you wi... [10:27:39] 10netops, 10Data-Persistence, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate servers in codfw rack B5 from asw-b5-codfw to lsw1-b5-codfw - https://phabricator.wikimedia.org/T355549 (10Marostegui) Great thank you! [11:17:13] fabfur: vgutierrez : I believe that the team is ready to deploy this VCL change for chrome preview detection again: https://gerrit.wikimedia.org/r/c/operations/puppet/+/992782 [11:17:52] btullis: checking right now, thanks! [11:19:22] Great. I'm just seeking final confirmation from the Data Engineering side too, but I expect that imminently. [11:35:23] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate servers in codfw rack B4 from asw-b4-codfw to lsw1-b4-codfw - https://phabricator.wikimedia.org/T355860 (10cmooney) p:05Triage→03Medium [11:35:41] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate servers in codfw rack B4 from asw-b4-codfw to lsw1-b4-codfw - https://phabricator.wikimedia.org/T355860 (10cmooney) [11:35:49] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate hosts from codfw row A/B ASW to new LSW devices - https://phabricator.wikimedia.org/T355544 (10cmooney) [11:36:54] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate servers in codfw rack A2 from asw-a2-codfw to lsw1-a2-codfw - https://phabricator.wikimedia.org/T355861 (10cmooney) p:05Triage→03Medium [11:37:53] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate servers in codfw rack A3 from asw-a3-codfw to lsw1-a3-codfw - https://phabricator.wikimedia.org/T355862 (10cmooney) p:05Triage→03Medium [11:38:03] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate servers in codfw rack A3 from asw-a3-codfw to lsw1-a3-codfw - https://phabricator.wikimedia.org/T355862 (10cmooney) [11:38:19] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate servers in codfw rack A2 from asw-a2-codfw to lsw1-a2-codfw - https://phabricator.wikimedia.org/T355861 (10cmooney) [11:39:06] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate servers in codfw rack A4 from asw-a4-codfw to lsw1-a4-codfw - https://phabricator.wikimedia.org/T355863 (10cmooney) p:05Triage→03Medium [11:39:17] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate hosts from codfw row A/B ASW to new LSW devices - https://phabricator.wikimedia.org/T355544 (10cmooney) [11:39:26] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate servers in codfw rack A4 from asw-a4-codfw to lsw1-a4-codfw - https://phabricator.wikimedia.org/T355863 (10cmooney) [11:40:30] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate servers in codfw rack A2 from asw-a2-codfw to lsw1-a2-codfw - https://phabricator.wikimedia.org/T355861 (10cmooney) [11:40:35] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate hosts from codfw row A/B ASW to new LSW devices - https://phabricator.wikimedia.org/T355544 (10cmooney) [11:41:05] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate hosts from codfw row A/B ASW to new LSW devices - https://phabricator.wikimedia.org/T355544 (10cmooney) [11:41:13] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate servers in codfw rack A3 from asw-a3-codfw to lsw1-a3-codfw - https://phabricator.wikimedia.org/T355862 (10cmooney) [11:42:32] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate servers in codfw rack A5 from asw-a5-codfw to lsw1-a5-codfw - https://phabricator.wikimedia.org/T355864 (10cmooney) p:05Triage→03Medium [11:42:40] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate servers in codfw rack A5 from asw-a5-codfw to lsw1-a5-codfw - https://phabricator.wikimedia.org/T355864 (10cmooney) [11:42:49] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate hosts from codfw row A/B ASW to new LSW devices - https://phabricator.wikimedia.org/T355544 (10cmooney) [11:43:30] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate servers in codfw rack A6 from asw-a6-codfw to lsw1-a6-codfw - https://phabricator.wikimedia.org/T355866 (10cmooney) p:05Triage→03Medium [11:43:37] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate servers in codfw rack A6 from asw-a6-codfw to lsw1-a6-codfw - https://phabricator.wikimedia.org/T355866 (10cmooney) [11:43:45] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate hosts from codfw row A/B ASW to new LSW devices - https://phabricator.wikimedia.org/T355544 (10cmooney) [11:45:25] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate servers in codfw rack A7 from asw-a7-codfw to lsw1-a7-codfw - https://phabricator.wikimedia.org/T355867 (10cmooney) p:05Triage→03Medium [11:45:33] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate servers in codfw rack A7 from asw-a7-codfw to lsw1-a7-codfw - https://phabricator.wikimedia.org/T355867 (10cmooney) [11:45:41] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate hosts from codfw row A/B ASW to new LSW devices - https://phabricator.wikimedia.org/T355544 (10cmooney) [11:47:00] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate servers in codfw rack B2 from asw-b2-codfw to lsw1-b2-codfw - https://phabricator.wikimedia.org/T355868 (10cmooney) p:05Triage→03Medium [11:47:09] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate hosts from codfw row A/B ASW to new LSW devices - https://phabricator.wikimedia.org/T355544 (10cmooney) [11:47:17] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate servers in codfw rack B2 from asw-b2-codfw to lsw1-b2-codfw - https://phabricator.wikimedia.org/T355868 (10cmooney) [11:52:14] 10netops, 10Infrastructure-Foundations, 10SRE: Create netbox script to support moving a cable from one network port to another - https://phabricator.wikimedia.org/T355869 (10cmooney) p:05Triage→03Low [11:53:15] 10netops, 10Infrastructure-Foundations, 10SRE: Migrate servers in codfw rack B3 from asw-b3-codfw to lsw1-b3-codfw - https://phabricator.wikimedia.org/T355870 (10cmooney) p:05Triage→03Medium [11:53:22] 10netops, 10Infrastructure-Foundations, 10SRE: Migrate servers in codfw rack B3 from asw-b3-codfw to lsw1-b3-codfw - https://phabricator.wikimedia.org/T355870 (10cmooney) [11:53:30] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate hosts from codfw row A/B ASW to new LSW devices - https://phabricator.wikimedia.org/T355544 (10cmooney) [11:54:33] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate servers in codfw rack B6 from asw-b6-codfw to lsw1-b6-codfw - https://phabricator.wikimedia.org/T355871 (10cmooney) p:05Triage→03Medium [11:54:42] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate servers in codfw rack B6 from asw-b6-codfw to lsw1-b6-codfw - https://phabricator.wikimedia.org/T355871 (10cmooney) [11:54:50] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate hosts from codfw row A/B ASW to new LSW devices - https://phabricator.wikimedia.org/T355544 (10cmooney) [11:55:31] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate servers in codfw rack B7 from asw-b7-codfw to lsw1-b7-codfw - https://phabricator.wikimedia.org/T355872 (10cmooney) p:05Triage→03Medium [11:55:42] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate hosts from codfw row A/B ASW to new LSW devices - https://phabricator.wikimedia.org/T355544 (10cmooney) [11:55:50] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate servers in codfw rack B7 from asw-b7-codfw to lsw1-b7-codfw - https://phabricator.wikimedia.org/T355872 (10cmooney) [11:56:33] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate servers in codfw rack B8 from asw-b8-codfw to lsw1-b8-codfw - https://phabricator.wikimedia.org/T355873 (10cmooney) p:05Triage→03Medium [11:56:44] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate hosts from codfw row A/B ASW to new LSW devices - https://phabricator.wikimedia.org/T355544 (10cmooney) [11:56:52] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate servers in codfw rack B8 from asw-b8-codfw to lsw1-b8-codfw - https://phabricator.wikimedia.org/T355873 (10cmooney) [11:57:37] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate hosts from codfw row A/B ASW to new LSW devices - https://phabricator.wikimedia.org/T355544 (10cmooney) [12:01:02] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate servers in codfw rack A8 from asw-a8-codfw to lsw1-a8-codfw - https://phabricator.wikimedia.org/T355874 (10cmooney) p:05Triage→03Medium [12:01:11] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate servers in codfw rack A8 from asw-a8-codfw to lsw1-a8-codfw - https://phabricator.wikimedia.org/T355874 (10cmooney) [12:01:19] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate hosts from codfw row A/B ASW to new LSW devices - https://phabricator.wikimedia.org/T355544 (10cmooney) [12:23:05] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate hosts from codfw row A/B ASW to new LSW devices - https://phabricator.wikimedia.org/T355544 (10cmooney) [12:26:26] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate hosts from codfw row A/B ASW to new LSW devices - https://phabricator.wikimedia.org/T355544 (10cmooney) [12:26:39] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate servers in codfw rack A8 from asw-a8-codfw to lsw1-a8-codfw - https://phabricator.wikimedia.org/T355874 (10cmooney) [12:41:08] fabfur: Deploying that now. Standard puppet 30 minute roll-out. Thanks again. [12:49:35] thx btullis and fabfur. i set a reminder to check on some basic queries after morning routine and meetings [12:49:49] * dr0ptp4kt wanders away [15:16:47] XioNoX, topranks question for you.. should tcp-mss-clamper clamp RST packets? [15:18:22] vgutierrez: my gut feeling is no, like we're not gonna transfer any data afterwards right? [15:18:52] topranks: what happens if a new client gets the MSS value from a RST packet instead of a SYNACK? [15:18:58] although a quick search is showing me examples of people doing just that [15:19:30] but if we send a RST the connection is finished [15:20:05] a new tcp connection has a new MSS right? does the OS cache mss values for IPs or anything like that? [15:21:10] that was the underlying question for you :) [15:21:32] and I'll come back to you on it - I think not, but I will look into it more [15:21:36] interesting question :) [15:36:45] thx [15:46:42] 10netops, 10Data-Persistence, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate servers in codfw rack B5 from asw-b5-codfw to lsw1-b5-codfw - https://phabricator.wikimedia.org/T355549 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=34ae871a-7149-43dd-8180-02ddd5b8c983) set by... [15:47:18] yes mss can be cached I think [15:47:26] my understanding is that the MSS option can only be set if the SYN flag is set, and SYN with RST is not valid [15:47:32] so there is some value, in the sense that a clamped RST could inform a future connection attempt [15:47:45] (well, unless it can't be done!) [15:50:59] 10netops, 10Infrastructure-Foundations, 10SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - https://phabricator.wikimedia.org/T348977 (10cmooney) Just an update here, the restriction still exists however I think I know how I went wrong. In order for the irb interface to be "up" the associated vlan ne... [15:52:31] the original TCP RFC seems to claim it's only sent with SYN as noted above [15:54:42] the history is a little complicated though [15:56:03] so RFC 793 (old old tcp) says about MSS: [15:56:04] "This field must only be sent in the initial connection request (i.e., in segments with the SYN control bit set). [15:56:07] " [15:56:47] but then RFC 1122 (ancient as well, " Requirements for Internet Hosts -- Communication Layers") says about TCP MSS: [15:57:39] 10netops, 10Data-Persistence, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate servers in codfw rack B5 from asw-b5-codfw to lsw1-b5-codfw - https://phabricator.wikimedia.org/T355549 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=e2f0518c-1df7-4528-89a1-5f2b248a7520) set by... [15:57:42] " TCP MUST implement both sending and receiving the Maximum [15:57:42] Segment Size option [TCP:4]. [15:57:42] TCP SHOULD send an MSS (Maximum Segment Size) option in [15:57:42] every SYN segment when its receive MSS differs from the [15:57:44] default 536, and MAY send it always." [15:58:01] I'm not really sure if "may send it always" means in non-SYN packets there or not, seems a little ambiguous [15:58:24] but maybe it just means "may send it even if you don't think mss differs from the default 536" [16:00:38] RFC 9293 updates 793, but says basically the same thing: [16:00:55] "TCP implementations SHOULD send an MSS Option in every SYN segment when its receive MSS differs from the default 536 for IPv4 or 1220 for IPv6 (SHLD-5), and MAY send it always (MAY-3)." [16:02:25] maybe the easiest way to get a pragmatic answer would be to check if the linux kernel sends mss with rst [16:16:20] FWIW, at least a basic RST from the kernel for a non-listening port, does not carry an MSS option [16:35:57] 10netops, 10Data-Persistence, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate servers in codfw rack B5 from asw-b5-codfw to lsw1-b5-codfw - https://phabricator.wikimedia.org/T355549 (10cmooney) Migration done! Serious props to @papaul and @Jhancock.wm for the smooth and super-fast execution!... [16:57:39] 10netops, 10Data-Persistence, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate servers in codfw rack B5 from asw-b5-codfw to lsw1-b5-codfw - https://phabricator.wikimedia.org/T355549 (10klausman) Nice work. On our machine (ml-serve2002), it was but four seconds: `[Thu Jan 25 16:09:14 2024] tg3... [17:05:40] 10Traffic, 10DNS, 10Foundational Technology Requests, 10SRE, 10Patch-For-Review: Ensure that store.wikimedia.org complies with Google's new email sender guidelines - https://phabricator.wikimedia.org/T355835 (10ssingh) @bcampbell: The changes have been merged, please try the authenticate domain part now.... [17:25:51] 10Traffic, 10DNS, 10Foundational Technology Requests, 10SRE, 10Patch-For-Review: Ensure that store.wikimedia.org complies with Google's new email sender guidelines - https://phabricator.wikimedia.org/T355835 (10bcampbell) @ssingh Thank you, I just initiated the process, which Shopify says may take 24 hou... [17:27:47] 10Traffic, 10DNS, 10Foundational Technology Requests, 10SRE: Ensure that wikimediafoundation.myshopify.com complies with Google's new email sender guidelines - https://phabricator.wikimedia.org/T355833 (10jhathaway) @bcampbell I assume the intent is to allow shopify to dkim sign their mail with keys we adv... [17:38:28] 10Traffic, 10DNS, 10Foundational Technology Requests, 10SRE: Ensure that wikimediafoundation.myshopify.com complies with Google's new email sender guidelines - https://phabricator.wikimedia.org/T355833 (10ssingh) >>! In T355833#9489071, @jhathaway wrote: > @bcampbell I assume the intent is to allow shopify... [17:59:23] 10Traffic, 10Data-Engineering, 10Movement-Insights, 10Patch-For-Review: Identify and label prefetch proxy data in our traffic - https://phabricator.wikimedia.org/T346463 (10dr0ptp4kt) It's back up and running. The following query is producing results. Note that the `is_goog_isp` field is mainly for helping... [18:05:18] 10netops, 10Data-Persistence, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate servers in codfw rack B5 from asw-b5-codfw to lsw1-b5-codfw - https://phabricator.wikimedia.org/T355549 (10Marostegui) @Jhancock.wm @papaul <3 [18:07:29] 10Traffic, 10DNS, 10Foundational Technology Requests, 10SRE, 10Patch-For-Review: Ensure that store.wikimedia.org complies with Google's new email sender guidelines - https://phabricator.wikimedia.org/T355835 (10ssingh) ` $ dig n1j._domainkey.wikimedia.org +short dkim1.327bdf87d37c.p413.email.myshopify.co... [18:21:14] 10Traffic: Restarting fifo-log-demux should not restart nginx - https://phabricator.wikimedia.org/T355905 (10BCornwall) [18:21:42] 10Traffic: Restarting fifo-log-demux should not restart nginx - https://phabricator.wikimedia.org/T355905 (10BCornwall) 05Open→03In progress p:05Triage→03Low [18:27:59] 10Traffic, 10GitLab (Project Migration): Migrate DNS repository from Gerrit to Gitlab - https://phabricator.wikimedia.org/T355906 (10BCornwall) [18:28:57] 10Traffic, 10GitLab (Project Migration): Migrate DNS repository from Gerrit to Gitlab - https://phabricator.wikimedia.org/T355906 (10ssingh) [18:31:17] 10Traffic, 10GitLab (Project Migration): Migrate DNS repository from Gerrit to Gitlab - https://phabricator.wikimedia.org/T355906 (10BCornwall) [19:30:32] 10Traffic, 10GitLab (Project Migration): Migrate DNS repository from Gerrit to Gitlab - https://phabricator.wikimedia.org/T355906 (10ssingh) I think we should do this once the GitLab workflow has been established and tested, given the critical nature of this repository. That said, the transition should look li... [19:50:53] 10Traffic, 10GitLab (Project Migration): Migrate DNS repository from Gerrit to Gitlab - https://phabricator.wikimedia.org/T355906 (10ssingh) I wanted to add: I haven't looked at and don't understand GitLab permissions but we should make sure that the folks in fr-tech (Fundraising) can also commit to this repos...