[11:01:54] !incidents [11:01:55] 3410 (UNACKED) [FIRING:1] NELHigh (page thanos sre tcp.timed_out) [11:02:36] !ack 3410 [11:02:37] 3410 (ACKED) [FIRING:1] NELHigh (page thanos sre tcp.timed_out) [11:03:20] NEL's seems to mostly be reports from Iran and "Other" - going to text-lb.esams.wikimedia.org. [11:04:07] topranks: yeah [11:05:41] I get directed there too, wikipieda seems ok just browsing [11:08:01] !incidents [11:08:02] 3410 (RESOLVED) [FIRING:1] NELHigh (page thanos sre tcp.timed_out) [11:08:35] topranks: fa.m.w.o follows the IR NEL curve [11:09:02] "Others" been high for a while, but that's not surprising if we bin everything non geoloced to it [11:10:03] hmm yeah, IT showing. Wonder if these are the same thing or not. [11:21:13] as discussed on the other channel, seems to be DNS issues from IT and a NEL spike from IR right before. NELs have gone back down under threshold. [11:38:51] <_joe_> !incidents [11:38:51] 3411 (UNACKED) [FIRING:1] NELHigh (page thanos sre tcp.timed_out) [11:38:51] 3410 (RESOLVED) [FIRING:1] NELHigh (page thanos sre tcp.timed_out) [11:38:57] !ack 3411 [11:38:57] 3411 (ACKED) [FIRING:1] NELHigh (page thanos sre tcp.timed_out) [11:39:04] <_joe_> <3 [11:39:07] ;) [11:41:38] Iran again [11:44:38] And now IT again [13:12:30] jynus: Hola, vuelvo con el tema de las pistolas en es.wiki. Con la dummy edit del otro día parece que se solucionó en su mayoría, pero acaba de llegar un nuevo informe de error en el que persiste el problema https://es.wikipedia.org/w/index.php?title=Wikipedia%3AInformes_de_error&diff=149074140&oldid=149072698 [13:15:18] lol he isn't even online atm [14:20:50] !incidents [14:20:51] 3412 (ACKED) [FIRING:1] NELHigh (page thanos sre tcp.timed_out) [14:20:51] 3411 (RESOLVED) [FIRING:1] NELHigh (page thanos sre tcp.timed_out) [14:20:52] 3410 (RESOLVED) [FIRING:1] NELHigh (page thanos sre tcp.timed_out) [14:21:55] jhathaway: looks like italy again [14:22:16] nod, thanks for sending out that email earlier [14:22:22] np [14:22:34] anything else we can do? [14:22:36] I figured it would keep happening until they fix it, might as well document it [14:22:52] jhathaway: network's not my forte, but I don't think so [14:23:09] thanks [14:24:10] If you look at the NEL graph for the last 6 hours, you'll see there were spikes of NELs from IR and Others right before, t.opranks removed the peering with TIM, and this time only IT is affected [14:25:55] it switched the impact from esams to drmrs [15:36:53] topranks: fyi https://mobile.twitter.com/netblocks/status/1622227650386214913 [15:41:38] RhinosF1: thanks for the info! Indeed that does not look good. [22:27:57] Indications are that TIM have resolved their issues, so I'll re-enable the Seabone peering at AMS-IX