[10:09:00] FIRING: PurgedHighEventLag: High event process lag with purged on cp5032:2112 - https://wikitech.wikimedia.org/wiki/Purged#Alerts - https://grafana.wikimedia.org/d/RvscY1CZk/purged?var-datasource=eqsin%20prometheus/ops&var-instance=cp5032 - https://alerts.wikimedia.org/?q=alertname%3DPurgedHighEventLag [10:14:00] RESOLVED: [14x] PurgedHighEventLag: High event process lag with purged on cp5020:2112 - https://wikitech.wikimedia.org/wiki/Purged#Alerts - https://alerts.wikimedia.org/?q=alertname%3DPurgedHighEventLag [12:14:07] Dear traffic, I will remove kafka1003 and replace with kafka1008 [12:25:22] effie: ack [12:32:18] tx [15:10:13] 06Traffic, 10Data Products (Data Products Sprint 23): Cookie % has been rejected because it is foreign and does not have the "Partitioned" attribute - https://phabricator.wikimedia.org/T375256#10371709 (10Milimetric) [16:52:03] 06Traffic, 10ircservserv, 06Security-Team, 10wikimedia-irc-libera, and 3 others: #wikimedia-traffic does not use wikimedia global bans list / wmopbot etc. - https://phabricator.wikimedia.org/T381110#10372286 (10sbassett) [16:52:05] 06Traffic, 10ircservserv, 06Security-Team, 10wikimedia-irc-libera, and 3 others: #wikimedia-traffic does not use wikimedia global bans list / wmopbot etc. - https://phabricator.wikimedia.org/T381110#10372280 (10sbassett) p:05Triage→03Low [16:52:07] 06Traffic, 10ircservserv, 06Security-Team, 10wikimedia-irc-libera, and 3 others: #wikimedia-traffic does not use wikimedia global bans list / wmopbot etc. - https://phabricator.wikimedia.org/T381110#10372287 (10sbassett) [16:52:16] 06Traffic, 10ircservserv, 06Security-Team, 10wikimedia-irc-libera, and 3 others: #wikimedia-traffic does not use wikimedia global bans list / wmopbot etc. - https://phabricator.wikimedia.org/T381110#10372288 (10sbassett) [18:07:00] FIRING: [5x] PurgedHighEventLag: High event process lag with purged on cp1110:2112 - https://wikitech.wikimedia.org/wiki/Purged#Alerts - https://alerts.wikimedia.org/?q=alertname%3DPurgedHighEventLag [18:32:00] FIRING: [7x] PurgedHighEventLag: High event process lag with purged on cp1107:2112 - https://wikitech.wikimedia.org/wiki/Purged#Alerts - https://alerts.wikimedia.org/?q=alertname%3DPurgedHighEventLag [19:27:00] FIRING: [6x] PurgedHighEventLag: High event process lag with purged on cp1107:2112 - https://wikitech.wikimedia.org/wiki/Purged#Alerts - https://alerts.wikimedia.org/?q=alertname%3DPurgedHighEventLag [19:28:00] FIRING: PurgedHighBacklogQueue: Large backlog queue for purged on cp1110:2112 - https://wikitech.wikimedia.org/wiki/Purged#Alerts - https://grafana.wikimedia.org/d/RvscY1CZk/purged?var-datasource=eqiad%20prometheus/ops&var-instance=cp1110 - https://alerts.wikimedia.org/?q=alertname%3DPurgedHighBacklogQueue [19:28:49] 06Traffic, 06SRE, 13Patch-For-Review: Disable acceptance of IPv6 router-advertisement on non-default LVS interface - https://phabricator.wikimedia.org/T358260#10373105 (10cmooney) 05Open→03Resolved a:03cmooney Happy we managed to get this sorted in the end and have the right settings on our LVS now... [19:38:00] RESOLVED: [2x] PurgedHighBacklogQueue: Large backlog queue for purged on cp1110:2112 - https://wikitech.wikimedia.org/wiki/Purged#Alerts - https://grafana.wikimedia.org/d/RvscY1CZk/purged?var-datasource=eqiad%20prometheus/ops&var-instance=cp1110 - https://alerts.wikimedia.org/?q=alertname%3DPurgedHighBacklogQueue [19:47:00] RESOLVED: [8x] PurgedHighEventLag: High event process lag with purged on cp1107:2112 - https://wikitech.wikimedia.org/wiki/Purged#Alerts - https://alerts.wikimedia.org/?q=alertname%3DPurgedHighEventLag [19:51:29] FIRING: HAProxyRestarted: HAProxy server restarted on cp3069:9100 - https://wikitech.wikimedia.org/wiki/HAProxy#HAProxy_for_edge_caching - https://grafana.wikimedia.org/d/gQblbjtnk/haproxy-drilldown?orgId=1&var-site=esams%20prometheus/ops&var-instance=cp3069&viewPanel=10 - https://alerts.wikimedia.org/?q=alertname%3DHAProxyRestarted [19:55:30] FIRING: [2x] PurgedHighBacklogQueue: Large backlog queue for purged on cp3069:2112 - https://wikitech.wikimedia.org/wiki/Purged#Alerts - https://alerts.wikimedia.org/?q=alertname%3DPurgedHighBacklogQueue [19:56:29] RESOLVED: HAProxyRestarted: HAProxy server restarted on cp3069:9100 - https://wikitech.wikimedia.org/wiki/HAProxy#HAProxy_for_edge_caching - https://grafana.wikimedia.org/d/gQblbjtnk/haproxy-drilldown?orgId=1&var-site=esams%20prometheus/ops&var-instance=cp3069&viewPanel=10 - https://alerts.wikimedia.org/?q=alertname%3DHAProxyRestarted [20:05:30] FIRING: [3x] PurgedHighBacklogQueue: Large backlog queue for purged on cp3069:2112 - https://wikitech.wikimedia.org/wiki/Purged#Alerts - https://alerts.wikimedia.org/?q=alertname%3DPurgedHighBacklogQueue [20:15:30] RESOLVED: [4x] PurgedHighBacklogQueue: Large backlog queue for purged on cp3069:2112 - https://wikitech.wikimedia.org/wiki/Purged#Alerts - https://alerts.wikimedia.org/?q=alertname%3DPurgedHighBacklogQueue [20:15:57] cool [22:44:21] 06Traffic, 13Patch-For-Review: Some domains listed as redirects for ncredir don't actually redirect properly. - https://phabricator.wikimedia.org/T380667#10373922 (10BCornwall) p:05Triage→03Low a:03BCornwall [22:44:24] 06Traffic: main project domains are improperly added/evaluated in ncmonitor patches - https://phabricator.wikimedia.org/T374640#10373938 (10BCornwall) 05In progress→03Resolved [22:47:45] 06Traffic: Performance optimization for ncmonitor's NCRedirRedirects class - https://phabricator.wikimedia.org/T368106#10373946 (10BCornwall) 05Open→03Declined The savings are not great enough to warrant the time, though if anyone is interested they are welcome to submit a patch :) [23:05:53] 10netops, 06Traffic, 06Infrastructure-Foundations, 06SRE: Heavy usage (possible scraping) of ceb.wikipedia.org from AS54801 (Dec 2 2024) - https://phabricator.wikimedia.org/T381347 (10cmooney) 03NEW p:05Triage→03High [23:06:28] 10netops, 06Traffic, 06Infrastructure-Foundations, 06SRE: Heavy usage (possible scraping) of ceb.wikipedia.org from AS54801 (Dec 2 2024) - https://phabricator.wikimedia.org/T381347#10374057 (10cmooney)