[02:32:55] FIRING: MaxConntrack: Max conntrack at 82.52% on krb1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack [02:37:55] RESOLVED: MaxConntrack: Max conntrack at 80.57% on krb1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack [04:20:55] FIRING: MaxConntrack: Max conntrack at 82.44% on krb1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack [04:25:55] RESOLVED: MaxConntrack: Max conntrack at 80.64% on krb1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack [06:00:55] FIRING: MaxConntrack: Max conntrack at 84.68% on krb1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack [06:05:55] RESOLVED: MaxConntrack: Max conntrack at 82.69% on krb1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack [07:54:55] is the MaxConntrack on krb1001 problem known? this alert is becoming quite spammy [08:19:02] side note, I've noticed that we have ~70~80k request per hour that report "Server not found in Kerberos database", and those are always the same 4 combinations of the entities [08:47:44] volans: o/ [08:48:04] checking it, it is almost surely a DE job (spark -> presto -> krb) [08:48:23] I think that Ben is working on fixing this issue for good, not sure what's the status [08:48:30] elukey: https://etherpad.wikimedia.org/p/volans-tmp2 [08:48:40] the 4 combinations that generates the "Server not found in Kerberos database" [08:48:43] fyi [08:49:09] no idea :( [08:53:26] that's about 20~25% of requests AFAICT doing some greps in the logs [09:04:26] already alerted DE [09:06:29] <3 [15:25:55] FIRING: MaxConntrack: Max conntrack at 84% on krb1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack [15:30:55] RESOLVED: MaxConntrack: Max conntrack at 84% on krb1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack [15:52:30] jhathaway: hi! any chance you could review https://gerrit.wikimedia.org/r/c/operations/puppet/+/1127066? thx! [15:52:39] sure [15:53:24] vgutierrez: done [15:53:34] the idea is to enable ECDSA certs on both Apache and Exim, check their adoption and consider the feasibility of dropping RSA at some point [15:53:37] thx [15:53:43] I'll send the exim one in a jiffy [16:06:21] sounds good [16:41:27] that's https://gerrit.wikimedia.org/r/c/operations/puppet/+/1127933 [16:47:57] thx for the review jhathaway, let's talk on Monday about deploying them [16:48:07] vgutierrez: sounds good [16:48:21] 🍻 enjoy your weekend! [17:13:25] you too! [17:27:56] FIRING: MaxConntrack: Max conntrack at 80.03% on krb1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack [17:32:55] RESOLVED: MaxConntrack: Max conntrack at 82.19% on krb1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack [17:46:55] FIRING: MaxConntrack: Max conntrack at 83.18% on krb1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack [17:51:55] RESOLVED: MaxConntrack: Max conntrack at 83.18% on krb1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack [18:33:55] FIRING: MaxConntrack: Max conntrack at 83.5% on krb1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack [18:38:55] RESOLVED: MaxConntrack: Max conntrack at 83.5% on krb1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack [19:00:55] FIRING: MaxConntrack: Max conntrack at 80.11% on krb1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack [19:05:55] RESOLVED: MaxConntrack: Max conntrack at 80.11% on krb1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack [19:13:55] FIRING: MaxConntrack: Max conntrack at 82.26% on krb1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack [19:18:55] RESOLVED: MaxConntrack: Max conntrack at 83.78% on krb1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack [19:19:55] FIRING: MaxConntrack: Max conntrack at 83.3% on krb1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack [19:24:55] RESOLVED: MaxConntrack: Max conntrack at 80.84% on krb1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack [19:38:55] FIRING: MaxConntrack: Max conntrack at 82.02% on krb1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack [19:43:55] RESOLVED: MaxConntrack: Max conntrack at 83.29% on krb1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack [20:03:56] FIRING: MaxConntrack: Max conntrack at 84.7% on krb1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack [20:08:55] RESOLVED: MaxConntrack: Max conntrack at 83.82% on krb1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack [20:19:55] FIRING: MaxConntrack: Max conntrack at 82.72% on krb1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack [20:24:55] RESOLVED: MaxConntrack: Max conntrack at 82.68% on krb1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack