[04:00:44] <Guest66>	 Karina
[08:06:31] <elukey>	 XioNoX: o/ around?
[08:06:44] <elukey>	 I am wondering if there is anything ongoing between eqiad and eqsin
[08:07:09] <elukey>	 cc: topranks
[08:08:11] <elukey>	 the purged instances on cp50xx nodes are showing some issues while connecting to kafka brokers in eqiad
[08:08:24] <elukey>	 I see some packet loss in pings between cp5017 -> kafka-main1001
[08:09:46] <XioNoX>	 elukey: I'm going to be on my laptop in 2h but you're right: https://grafana.wikimedia.org/d/m1LYjVjnz/network-icmp-probes?orgId=1&var-site=eqiad&var-target_site=eqsin&var-role=cr&var-family=All&from=now-3h&to=now
[08:10:08] <elukey>	 TIL about the dashboard, thanks!
[08:10:21] <XioNoX>	 elukey: you can drain the arelion link if you want
[08:11:06] <elukey>	 XioNoX: never done it before, don't want to cause an outage on monday morning :D
[08:11:26] <elukey>	 I can wait for you or Cathal, no problem
[08:11:51] <XioNoX>	 https://netbox.wikimedia.org/circuits/circuits/27/ state->drained then run homer on both sides' routers
[08:12:04] <topranks>	 elukey: I’m here I can have a look
[08:12:28] <XioNoX>	 could be worth an email to Arelion too
[08:12:34] <elukey>	 XioNoX: ah wow super simple
[08:12:50] <XioNoX>	 topranks: good morning :)
[08:13:27] <elukey>	 indeed :)
[08:13:56] <topranks>	 elukey: it is indeed simple so feel free if you want to run.  I best take a look anyway though so no probs either way
[08:14:29] <elukey>	 topranks: I can try yes, before that one qs - where can I find that the issue is related to Aerlion?
[08:14:32] <elukey>	 (as curiosity)
[08:15:26] <elukey>	 in the meantime - setting https://netbox.wikimedia.org/circuits/circuits/27/ to state drained
[08:16:09] <topranks>	 various ways depending, the graph XioNoX linked is our ICMP probes, usually a good sign of some kind of “partial” issue (i.e. not down co
[08:16:19] <topranks>	 *completely
[08:16:38] <XioNoX>	 yeah previous experience also, not the first time it happens
[08:16:55] <XioNoX>	 and knowing that the arelion link is the primary one
[08:17:04] <elukey>	 ah wait I am slow today, if I see problems with pings eqiad <-> eqsin then it is the transport link that causes problems, hence I check what it is
[08:17:36] <XioNoX>	 yeah correct 
[08:17:48] <XioNoX>	 eqiad-eqsin goes through codfw
[08:18:18] <elukey>	 I was about to say - homer on cr3-eqsin and cr1-codfw
[08:18:26] <elukey>	 does it sound good?
[08:18:30] <XioNoX>	 yeah that's correct 
[08:18:38] <XioNoX>	 both direct sides of the circuit 
[08:19:15] <XioNoX>	 you can click the "trace" button 
[08:19:20] <XioNoX>	 https://netbox.wikimedia.org/circuits/circuit-terminations/69/trace/
[08:19:21] <topranks>	 yep, sry that's the missing piece you need to know what sites it links
[08:20:01] <elukey>	 wow totally different since the last time I worked on it
[08:20:01] <XioNoX>	 one day I'll write a cookbook 
[08:20:11] <XioNoX>	 I actually started locally 
[08:20:11] <elukey>	 feels like being in the Minority Report movie
[08:20:17] <elukey>	 (compared to before)
[08:20:30] <elukey>	 good job folks
[08:20:50] <XioNoX>	 thanks :)
[08:21:15] <topranks>	 Also this circuit is more susceptible to these kind of issues than many of our others 
[08:21:39] <topranks>	 It's a virtual-ethernet circuit over MPLS, so if the carrier loses part of the normal path it will re-route 
[08:21:53] <topranks>	 and those kind of events can mean the carrier suffers congestion, we see this kind of thing
[08:22:08] <topranks>	 *most* of our circuits are unprotected wavelengths, meaning if path of the overall path is cut it just dies 
[08:22:12] <topranks>	 doesn't re-route automatically 
[08:23:54] <elukey>	 thanks :)
[08:24:04] <elukey>	 so homer shows me this diff for cr1-codfw
[08:24:04] <elukey>	 -     metric 2000;
[08:24:05] <elukey>	 +     metric 5000;
[08:24:17] <elukey>	 interface xe-1/0/1
[08:25:29] <elukey>	 that seems correct from what I can see
[08:25:38] <topranks>	 yep that makes sense 
[08:25:42] <elukey>	 shall I proceed codfw then eqsin?
[08:25:43] <topranks>	 just adjusts the OSPF cost 
[08:25:46] <topranks>	 please do 
[08:25:46] <elukey>	 (with the commits)
[08:25:47] <elukey>	 ack
[08:25:48] <topranks>	 thanks :)
[08:26:51] <elukey>	 thank you folks! Learning a lot
[08:28:26] <elukey>	 {{done}}
[08:29:46] <elukey>	 ok I see zero packet loss now in pings between cp5017 and kafka-main1001
[08:29:53] <topranks>	 Nice!
[08:30:08] <elukey>	 and purged metrics are recovering afaics
[08:30:20] <topranks>	 Yes the traffic should be going from Eqiad to Ulsfo and across the Pacific with another carrier now :)
[08:32:15] <elukey>	 purged is happy, all recovered afaics
[08:34:12] <topranks>	 great.  I opened T337220 and will chase up with Arelion 
[08:34:45] <topranks>	 On past experience it's probably some fibre cut that's pushed traffic between the sites over a new path which is congested 
[08:38:36] <elukey>	 makes sense yes
[09:38:07] <jbond>	 hi all back from vacation. just making my way through my bac log but if there is anything you would like me to push to the top feel free to ping
[10:14:34] <arturo>	 XioNoX: can you merge https://gerrit.wikimedia.org/r/c/operations/dns/+/914751 I'm on the airport with a bit of a spotty internet connection. Should be very safe to merge. I agreed with volans|off on getting it merged this monday
[10:23:09] <topranks>	 arturo: let me have a look, I think Arzhel has spotty connectivity too 
[10:24:04] <topranks>	 seems ok, I'll take care of it 
[10:26:06] <topranks>	 elukey: FYI it seems the drops on the eqsin link may have been usage related on our side 
[10:26:28] <topranks>	 a little bit "boy who cried wolf" probably jumped too quick to assume carrier issue because we've had that a bunch of times on that circuit 
[10:26:51] <topranks>	 I've un-drained it now, but we still have high-ish usage.  I'm keeping an eye and will flip back if it seems needed
[10:27:00] <topranks>	 Do let me know if you observe any app-layer problems 
[10:29:32] <arturo>	 thanks topranks !!
[10:31:01] <elukey>	 ack!
[10:41:14] <XioNoX>	 I had a quick look through https://turnilo.wikimedia.org/#webrequest_sampled_live/ but don't see any smoking gun on why we have those spikes
[10:43:43] <XioNoX>	 but there is indeed correlation, even though we're supposed to have a 4G CAP, so maybe it's still (or back at) 2.5G?
[10:44:15] <XioNoX>	 might be worth asking Arelion
[10:47:02] <topranks>	 XioNoX: yeah I asked them already 
[10:47:23] <topranks>	 the 5-min averaging may be smoothing out bigger spikes, but yep doesn't look like it's saturated all the time 
[10:47:45] <topranks>	 right now usage seems ok and no drops, I also was looking in turnilo but didn't turn up anything yet 
[10:48:24] <topranks>	 Hopefully the carrier can confirm their policer is set to the right level, and if they have counter drops on it 
[13:30:22] <elukey>	 topranks: just noticed that around an hour ago there was another round of issues for purged in eqsin, auto-resolved
[13:32:08] <topranks>	 elukey: yeah, I was on lunch but looking I see there was another burst of traffic 
[13:32:28] <topranks>	 we should be able to get more throughput out of the circuit, carrier have confirmed settings are ok 
[13:35:23] <topranks>	 elukey: for now I think I'll drain it again, at least until we can verify what's causing the bursts in traffic, and see if we can do anything else our side to help better use this link to its full capacity