[12:10:04] 10Traffic, 10Data-Engineering, 10Movement-Insights: Identify and label prefetch proxy data in our traffic - https://phabricator.wikimedia.org/T346463 (10Milimetric) [[ https://github.com/wikimedia/operations-puppet/blob/8a82461c968c7ba44e786ccdbc05f240369a9d57/modules/varnish/templates/analytics.inc.vcl.erb#... [14:10:12] hi traffic friends, I see that LVSHighRx paged last night, looks like to an actual event but not one that was user-impacting? [14:11:17] I think we should probably make that alert non-paging [14:11:49] it can be a useful hint for what to look at during an actual outage, but it's not a symptom of an actual problem [14:14:40] cdanis: thanks! will submit a patch shortly [14:41:26] cdanis sukhe CC me on that patch if you have a chance, we tripped that alert in https://phabricator.wikimedia.org/T351475 as well [14:41:45] inflatador: it is already merged [14:42:27] inflatador: see https://gerrit.wikimedia.org/r/plugins/gitiles/operations/alerts/+/ef8c70b69f3160df4245191b49a237a40e3b2329 [14:45:46] 10Traffic, 10SRE-swift-storage: OpenSSL 3.x performance issues - https://phabricator.wikimedia.org/T352744 (10bking) FYI, Elasticsearch doesn't use envoy either. The Flink pipelines @Joe mentioned are all in k8s, and wdqs (consumer of flink) uses envoy, so I think we're OK there? [14:46:16] sukhe cdanis thanks, I will close the ticket with a link to the patch unless y'all think there is anything else we need to address [14:51:18] inflatador: sounds good! [15:27:24] 10Traffic, 10DC-Ops: cp4037 reimage for cookbook getting stuck at PXE boot - https://phabricator.wikimedia.org/T352876 (10ssingh) [15:27:52] 10Traffic, 10DC-Ops: cp4037 reimage for cookbook getting stuck at PXE boot - https://phabricator.wikimedia.org/T352876 (10ssingh) [15:51:43] 10Traffic, 10DC-Ops: cp4037 reimage for cookbook getting stuck at PXE boot - https://phabricator.wikimedia.org/T352876 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host cp4037.ulsfo.wmnet with OS bullseye [16:28:48] 10Traffic, 10MW-on-K8s, 10SRE, 10serviceops, 10Release-Engineering-Team (Seen): Serve production traffic via Kubernetes - https://phabricator.wikimedia.org/T290536 (10hnowlan) [16:29:22] 10Traffic, 10MW-on-K8s, 10SRE, 10serviceops, and 2 others: Move MediaWiki jobs to mw-on-k8s - https://phabricator.wikimedia.org/T349796 (10hnowlan) 05Open→03In progress a:03hnowlan [16:37:50] 10netops, 10Infrastructure-Foundations, 10SRE, 10serviceops: Test IP-renumbering on a kubestage2002.codfw.wmnet - https://phabricator.wikimedia.org/T352883 (10Clement_Goubert) [16:38:13] 10netops, 10Infrastructure-Foundations, 10SRE, 10serviceops: Test IP-renumbering on kubestage2002.codfw.wmnet - https://phabricator.wikimedia.org/T352883 (10Clement_Goubert) [16:41:20] 10netops, 10Infrastructure-Foundations, 10SRE, 10serviceops: Test IP-renumbering on kubestage2002.codfw.wmnet - https://phabricator.wikimedia.org/T352883 (10Clement_Goubert) p:05Triage→03Medium a:03Clement_Goubert [17:23:21] 10Traffic, 10Data-Engineering, 10Movement-Insights, 10Patch-For-Review: Identify and label prefetch proxy data in our traffic - https://phabricator.wikimedia.org/T346463 (10elukey) I created a tmux session on cp4038 with the following: ` sudo varnishlog -n frontend -q 'ReqHeader:Sec-Purpose eq "prefetch;... [17:23:46] 10Traffic, 10GitLab (Project Migration), 10Patch-For-Review: Migrate Traffic repositories from Gerrit to Gitlab - https://phabricator.wikimedia.org/T347623 (10BCornwall) [17:23:59] hello folks [17:24:15] as FYI I have a tmux on cp4038 with varnishlog (https://phabricator.wikimedia.org/T346463#9387601) [17:24:29] lemme know if it is a problem [17:25:17] ok already found what I was looking for, closed, nevermind :D [17:26:26] 10Traffic, 10Data-Engineering, 10Movement-Insights, 10Patch-For-Review: Identify and label prefetch proxy data in our traffic - https://phabricator.wikimedia.org/T346463 (10elukey) Confirmed, I found a request with: ` - ReqHeader sec-purpose: prefetch;anonymous-client-ip ` So it is a good confirma... [17:36:40] 10Traffic, 10DC-Ops: cp4037 reimage for cookbook getting stuck at PXE boot - https://phabricator.wikimedia.org/T352876 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host cp4037.ulsfo.wmnet with OS bullseye [17:36:53] 10Traffic, 10Data-Engineering, 10Movement-Insights, 10Patch-For-Review: Identify and label prefetch proxy data in our traffic - https://phabricator.wikimedia.org/T346463 (10elukey) If the final decision is to proceed with a new field in Webrequest, https://gerrit.wikimedia.org/r/c/operations/puppet/+/98091... [18:02:54] 10Traffic, 10DC-Ops: cp4037 reimage for cookbook getting stuck at PXE boot - https://phabricator.wikimedia.org/T352876 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host cp4037.ulsfo.wmnet with OS bullseye executed with errors: - cp4037 (**FAIL**) - Removed from... [19:03:14] 10Traffic, 10Data-Engineering, 10Movement-Insights, 10Patch-For-Review: Identify and label prefetch proxy data in our traffic - https://phabricator.wikimedia.org/T346463 (10dr0ptp4kt) I like where @elukey is going with this. I saw this ticket and thought I'd share some perspective. It may be this has all... [19:46:54] 10Traffic, 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Move lvs2012 primary uplink and connect to new row A/B vlans - https://phabricator.wikimedia.org/T352909 (10cmooney) [19:50:53] 10Traffic, 10netops, 10Infrastructure-Foundations, 10SRE, and 2 others: Move lvs2012 primary uplink and connect to new row A/B vlans - https://phabricator.wikimedia.org/T352909 (10cmooney) [19:54:38] 10Traffic, 10netops, 10Infrastructure-Foundations, 10SRE, and 2 others: Move lvs2012 primary uplink and connect to new row A/B vlans - https://phabricator.wikimedia.org/T352909 (10cmooney) [20:19:20] 10Traffic, 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Move lvs2011 primary uplink and connect to new row A/B vlans - https://phabricator.wikimedia.org/T352912 (10cmooney) [20:30:59] 10Traffic, 10netops, 10Infrastructure-Foundations, 10SRE, and 2 others: Move lvs2011 primary uplink and connect to new row A/B vlans - https://phabricator.wikimedia.org/T352912 (10cmooney) [20:31:44] 10Traffic, 10netops, 10Infrastructure-Foundations, 10SRE, and 2 others: Move lvs2012 primary uplink and connect to new row A/B vlans - https://phabricator.wikimedia.org/T352909 (10cmooney) [21:45:36] 10Traffic, 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Move lvs2012 from private1-b-codfw (row) to private1-b2-codfw (rack) vlan - https://phabricator.wikimedia.org/T352918 (10cmooney) [22:05:42] 10Traffic, 10netops, 10Infrastructure-Foundations, 10SRE, and 2 others: Move lvs2012 from private1-b-codfw (row) to private1-b2-codfw (rack) vlan - https://phabricator.wikimedia.org/T352918 (10cmooney) [22:10:12] 10Traffic, 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Move lvs2011 from private1-a-codfw (row) to private1-a2-codfw (rack) vlan - https://phabricator.wikimedia.org/T352920 (10cmooney) [22:19:01] 10Traffic, 10netops, 10Infrastructure-Foundations, 10SRE, and 2 others: Move lvs2011 from private1-a-codfw (row) to private1-a2-codfw (rack) vlan - https://phabricator.wikimedia.org/T352920 (10cmooney) [23:21:32] 10Traffic, 10Cloud-VPS, 10DNS, 10SRE: DNS name resolution failure with www.spacecom.mil from Cloud VPS - https://phabricator.wikimedia.org/T346471 (10Dzahn) Can confirm this is still the case. From a random different cloud VPS instance: ` dig www.spacecom.mil @172.20.255.1` fails. (and 172.20.255.1 is... [23:24:02] 10Traffic, 10Cloud-VPS, 10DNS, 10SRE: DNS name resolution failure with www.spacecom.mil from Cloud VPS - https://phabricator.wikimedia.org/T346471 (10Dzahn) It's not ALL of .mil either. For example "dig cybercoe.army.mil @172.20.255.1" works and also points to an Akamai edge. [23:32:57] 10Traffic, 10Cloud-VPS, 10DNS, 10SRE: DNS name resolution failure with www.spacecom.mil from Cloud VPS - https://phabricator.wikimedia.org/T346471 (10Don-vip) Yes, my tool scans for free media at following .mil domains without problem: - www.afspc.af.mil - www.buckley.spaceforce.mil - www.jtf-spaced...