[03:43:57] (HAProxyEdgeTrafficDrop) firing: 52% request drop in text@esams during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=esams&var-cache_type=text - https://alerts.wikimedia.org/?q=alertname%3DHAProxyEdgeTrafficDrop [03:48:56] (HAProxyEdgeTrafficDrop) resolved: (4) 65% request drop in text@drmrs during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://alerts.wikimedia.org/?q=alertname%3DHAProxyEdgeTrafficDrop [06:30:06] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10SRE, 10netbox: Avoid ghost hosts on the network - https://phabricator.wikimedia.org/T306007 (10ayounsi) @wiki_willy See T304849 (and its description history), or T306129 @cmooney we can query LibreNMS as it also have the data, but I'd prefer to not hav... [06:35:10] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10SRE, 10ops-ulsfo: (Need By: TBD) rack/setup/install new mr1-ulsfo - https://phabricator.wikimedia.org/T294314 (10ayounsi) [06:35:24] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10SRE, 10ops-ulsfo: (Need By: TBD) rack/setup/install new mr1-ulsfo - https://phabricator.wikimedia.org/T294314 (10ayounsi) Cables can't be moved through the Netbox UI, they need to be deleted and re-created, which is cumbersome and error-prone. The curre... [08:16:46] 10Traffic: Improve handling/logging of HAproxy emergency log messages - https://phabricator.wikimedia.org/T306236 (10Vgutierrez) [08:16:55] 10Traffic: Improve handling/logging of HAproxy emergency log messages - https://phabricator.wikimedia.org/T306236 (10Vgutierrez) p:05Triage→03High [08:46:57] 10netops, 10Infrastructure-Foundations, 10netbox: Netbox Juniper report - https://phabricator.wikimedia.org/T306238 (10ayounsi) p:05Triage→03Low [10:14:57] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10SRE, 10netbox: Avoid ghost hosts on the network - https://phabricator.wikimedia.org/T306007 (10cmooney) > @cmooney we can query LibreNMS as it also have the data, but I'd prefer to not have the source of truth driven by production (thus the alert only f... [10:23:13] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10SRE, 10netbox: Avoid ghost hosts on the network - https://phabricator.wikimedia.org/T306007 (10cmooney) Actually one really ugly thing you could do is to make the Jinja templates add "disabled" config for every _possible_ interface name. For example fo... [13:29:02] 10Traffic, 10SRE: Improve handling/logging of HAproxy emergency log messages - https://phabricator.wikimedia.org/T306236 (10CDanis) Something else I'm wondering about is if we can do any rate-limiting of the generation of such messages within haproxy. I suspect it was spending a non-trivial amount of CPU time... [20:34:56] (HAProxyEdgeTrafficDrop) firing: 16% request drop in text@eqsin during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=eqsin&var-cache_type=text - https://alerts.wikimedia.org/?q=alertname%3DHAProxyEdgeTrafficDrop [20:39:56] (HAProxyEdgeTrafficDrop) firing: (6) 58% request drop in text@codfw during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://alerts.wikimedia.org/?q=alertname%3DHAProxyEdgeTrafficDrop [20:44:56] (HAProxyEdgeTrafficDrop) resolved: (6) 58% request drop in text@codfw during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://alerts.wikimedia.org/?q=alertname%3DHAProxyEdgeTrafficDrop