[05:37:23] hello! see the email "Timeouts and long loading times at www.wikipedia.org" to noc@, I can't see anything abnormal at the network level. Could someone from traffic have a look at the traffic layer? [07:06:13] XioNoX: do we know what they are actually measuring? just total time for a GET / against https://www.wikipedia.org? [07:19:25] vgutierrez: nop, I don't have more informations [07:36:43] text@esams looks healthy, a GET / against https://www.wikipedia.org should result on a hit-front response and p99 for those is looking good over the last 90 days: https://grafana.wikimedia.org/goto/0ANpPCw4z?orgId=1 [07:37:15] I'll answer to the email asking for more information [07:38:00] thanks [09:16:02] XioNoX: from their answer it looks like they are having some issues establishing a TCP connection over IPv6 [09:18:13] vgutierrez: I didn't get the reply, was it unicast or multicast? [09:18:50] just forwarded their answer to noc@ [09:19:32] last execution shows 2s to establish a TCP connection [09:22:54] thanks! [09:23:08] I guess it's packet capture time! [09:30:34] something odd is going on.. I'm relatively far away from esams compared to them [09:31:12] I added your troubleshooting at the bottom of https://wikitech.wikimedia.org/wiki/Reporting_a_connectivity_issue too [09:34:37] and replied to their email [11:05:04] hey vgutierrez - would it be okay if I deployed https://gerrit.wikimedia.org/r/c/operations/puppet/+/920667 this morning? [12:56:06] hnowlan: I think sop [12:56:10] *so [13:22:58] XioNoX: hmm some kind of routing loop? [13:42:36] clearly something funky going on [14:04:09] 10Traffic, 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10ops-knams: Q4/Q1:knams racking elevations & planning - https://phabricator.wikimedia.org/T331886 (10cmooney) >>! In T331886#8688489, @RobH wrote: >>>! In T331886#8688402, @ayounsi wrote: >> Ideally we should also have 1 patch panel per rack.... [14:55:36] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10SRE, and 2 others: Q1:(Need By: TBD) rack/setup/install cloudswift100[12] - https://phabricator.wikimedia.org/T289882 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host cloudswift1001.eqiad.wmnet with OS... [15:14:03] 10Traffic, 10Anti-Harassment, 10Data-Engineering: Include User-Agent Client Hints in WebRequest logs - https://phabricator.wikimedia.org/T337947 (10kostajh) [15:19:55] 10Wikimedia-Apache-configuration, 10SRE, 10Security-Team, 10WMF-General-or-Unknown, and 2 others: Add security.txt to Wikimedia sites? (2023 edition) - https://phabricator.wikimedia.org/T337949 (10Reedy) [15:57:22] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10SRE, and 2 others: Q1:(Need By: TBD) rack/setup/install cloudswift100[12] - https://phabricator.wikimedia.org/T289882 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host cloudswift1001.eqiad.wmnet with OS bul... [15:59:50] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10SRE, and 3 others: Q1:(Need By: TBD) rack/setup/install cloudswift100[12] - https://phabricator.wikimedia.org/T289882 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host cloudswift1001.eqiad.wmnet with OS... [16:00:28] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10SRE, and 3 others: Q1:(Need By: TBD) rack/setup/install cloudswift100[12] - https://phabricator.wikimedia.org/T289882 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host cloudswift1001.eqiad.wmnet with OS bul... [16:01:24] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10SRE, and 3 others: Q1:(Need By: TBD) rack/setup/install cloudswift100[12] - https://phabricator.wikimedia.org/T289882 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host cloudswift1001.eqiad.wmnet with OS... [16:17:00] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10SRE, 10ops-eqiad: Q2:(Need By: TBD) Rows E/F network racking task - https://phabricator.wikimedia.org/T292095 (10cmooney) @Jclark-ctr let me know when it might suit to try and get more of these moves done. Thanks. [16:24:19] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10SRE, and 2 others: Q1:(Need By: TBD) rack/setup/install cloudswift100[12] - https://phabricator.wikimedia.org/T289882 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host cloudswift1001.eqiad.wmnet with OS bul... [16:28:24] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10SRE, and 2 others: Q1:(Need By: TBD) rack/setup/install cloudswift100[12] - https://phabricator.wikimedia.org/T289882 (10Jhancock.wm) [16:40:25] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10SRE, and 2 others: Q1:(Need By: TBD) rack/setup/install cloudswift100[12] - https://phabricator.wikimedia.org/T289882 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host cloudswift1002.eqiad.wmnet with OS... [16:40:39] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10SRE, and 2 others: Q1:(Need By: TBD) rack/setup/install cloudswift100[12] - https://phabricator.wikimedia.org/T289882 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host cloudswift1002.eqiad.wmnet with OS bul... [16:55:45] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10SRE, and 2 others: Q1:(Need By: TBD) rack/setup/install cloudswift100[12] - https://phabricator.wikimedia.org/T289882 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host cloudswift1002.eqiad.wmnet with OS... [16:55:55] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10SRE, and 2 others: Q1:(Need By: TBD) rack/setup/install cloudswift100[12] - https://phabricator.wikimedia.org/T289882 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host cloudswift1002.eqiad.wmnet with OS bul... [17:05:42] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10SRE, and 2 others: Q1:(Need By: TBD) rack/setup/install cloudswift100[12] - https://phabricator.wikimedia.org/T289882 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host cloudswift1002.eqiad.wmnet with OS... [17:05:51] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10SRE, and 2 others: Q1:(Need By: TBD) rack/setup/install cloudswift100[12] - https://phabricator.wikimedia.org/T289882 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host cloudswift1002.eqiad.wmnet with OS bul... [17:07:26] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10SRE, and 2 others: Q1:(Need By: TBD) rack/setup/install cloudswift100[12] - https://phabricator.wikimedia.org/T289882 (10Jhancock.wm) [17:23:32] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10SRE, and 2 others: Q1:(Need By: TBD) rack/setup/install cloudswift100[12] - https://phabricator.wikimedia.org/T289882 (10Jhancock.wm) @Jclark-ctr when you have a moment/back can you swap the ports on the NIC? thanks! [23:57:38] 10netops, 10Cloud-VPS, 10Infrastructure-Foundations, 10SRE, and 2 others: cloudservices[2004/2005]-dev & cloudweb2002-dev: connect them to cloudsw so they can have cloud-private vlan - https://phabricator.wikimedia.org/T336587 (10Andrew) I haven't dug much, but designate is currently failing on cloudservic...