[05:48:21] FIRING: SLOMetricAbsent: varnish-combined codfw - https://slo.wikimedia.org/?search=varnish-combined - https://alerts.wikimedia.org/?q=alertname%3DSLOMetricAbsent [05:48:52] FIRING: [2x] SLOMetricAbsent: haproxy-combined - https://slo.wikimedia.org/?search=haproxy-combined - https://alerts.wikimedia.org/?q=alertname%3DSLOMetricAbsent [05:49:06] FIRING: SLOMetricAbsent: varnish-combined codfw - https://slo.wikimedia.org/?search=varnish-combined - https://alerts.wikimedia.org/?q=alertname%3DSLOMetricAbsent [06:04:06] FIRING: [2x] SLOMetricAbsent: varnish-combined codfw - https://slo.wikimedia.org/?search=varnish-combined - https://alerts.wikimedia.org/?q=alertname%3DSLOMetricAbsent [06:08:21] FIRING: [2x] SLOMetricAbsent: varnish-combined codfw - https://slo.wikimedia.org/?search=varnish-combined - https://alerts.wikimedia.org/?q=alertname%3DSLOMetricAbsent [07:49:10] FIRING: [2x] SLOMetricAbsent: varnish-combined codfw - https://slo.wikimedia.org/?search=varnish-combined - https://alerts.wikimedia.org/?q=alertname%3DSLOMetricAbsent [07:53:21] 06Traffic, 13Patch-For-Review: haproxykafka service isn't restarted when upgraded - https://phabricator.wikimedia.org/T393016#10791068 (10Fabfur) [07:53:21] FIRING: [2x] SLOMetricAbsent: varnish-combined codfw - https://slo.wikimedia.org/?search=varnish-combined - https://alerts.wikimedia.org/?q=alertname%3DSLOMetricAbsent [07:58:21] RESOLVED: [2x] SLOMetricAbsent: varnish-combined codfw - https://slo.wikimedia.org/?search=varnish-combined - https://alerts.wikimedia.org/?q=alertname%3DSLOMetricAbsent [07:58:52] RESOLVED: [2x] SLOMetricAbsent: haproxy-combined - https://slo.wikimedia.org/?search=haproxy-combined - https://alerts.wikimedia.org/?q=alertname%3DSLOMetricAbsent [07:59:06] RESOLVED: [2x] SLOMetricAbsent: varnish-combined codfw - https://slo.wikimedia.org/?search=varnish-combined - https://alerts.wikimedia.org/?q=alertname%3DSLOMetricAbsent [11:01:14] 06Traffic, 06[Archived]Wikidata Dev Team, 10Prod-Kubernetes, 06SRE, and 5 others: Frequent 500 Errors and Timeouts When Adding Statements to New Item or Lexeme-typed Properties - https://phabricator.wikimedia.org/T374230#10791672 (10Silvan_WMDE) True, the deployment had not actually happened when [[ https:... [11:21:04] 06Traffic, 06Infrastructure-Foundations, 10Spicerack, 10SRE-tools, 13Patch-For-Review: Spicerack's Icinga module should provide a way to skip specific services in sub-optimal but desired state - https://phabricator.wikimedia.org/T392848#10791709 (10Volans) Have you considered just downtiming the affected... [13:27:04] 06Traffic, 06serviceops, 10Content-Transform-Team (Work In Progress), 07Essential-Work, 13Patch-For-Review: Purging edge caches doesn't work for articles with ":" in their title - https://phabricator.wikimedia.org/T392849#10792134 (10MSantos) [13:27:21] 06Traffic, 06serviceops, 10Content-Transform-Team (Work In Progress), 07Essential-Work, 13Patch-For-Review: Purging edge caches doesn't work for articles with ":" in their title - https://phabricator.wikimedia.org/T392849#10792135 (10MSantos) a:03Jgiannelos [13:41:25] FIRING: SystemdUnitFailed: haproxykafka.service on cp7001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [13:46:48] 06Traffic, 10conftool: FY 24/25 WE 4.3.11 Define a policy for maintenance of requestctl rules - https://phabricator.wikimedia.org/T393381 (10Joe) 03NEW [13:50:42] ^^ haproxykafka alert is me [14:01:09] FIRING: LVSHighRX: Excessive RX traffic on lvs3008:9100 (eno12399np0) - https://bit.ly/wmf-lvsrx - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=lvs3008 - https://alerts.wikimedia.org/?q=alertname%3DLVSHighRX [14:01:12] 06Traffic, 06serviceops, 10Content-Transform-Team (Work In Progress), 07Essential-Work, 13Patch-For-Review: Purging edge caches doesn't work for articles with ":" in their title - https://phabricator.wikimedia.org/T392849#10792370 (10Jgiannelos) After a bit of investigation here is were I am at: * For a... [14:01:37] 06Traffic, 06serviceops, 10Content-Transform-Team (Work In Progress), 07Essential-Work, 13Patch-For-Review: Purging edge caches doesn't work for articles with ":" in their title - https://phabricator.wikimedia.org/T392849#10792385 (10Jgiannelos) cc @hnowlan [14:06:09] RESOLVED: LVSHighRX: Excessive RX traffic on lvs3008:9100 (eno12399np0) - https://bit.ly/wmf-lvsrx - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=lvs3008 - https://alerts.wikimedia.org/?q=alertname%3DLVSHighRX [14:09:44] 06Traffic, 06Infrastructure-Foundations, 10Spicerack, 10SRE-tools, 13Patch-For-Review: Spicerack's Icinga module should provide a way to skip specific services in sub-optimal but desired state - https://phabricator.wikimedia.org/T392848#10792415 (10ssingh) >>! In T392848#10791709, @Volans wrote: > Have y... [14:11:25] RESOLVED: SystemdUnitFailed: haproxykafka.service on cp7001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [14:17:30] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: Upgrade codfw E/F Juniper equipment to Junos 23.x - https://phabricator.wikimedia.org/T393001#10792478 (10Volans) p:05Triage→03Medium [15:06:58] 06Traffic, 13Patch-For-Review: haproxykafka service isn't restarted when upgraded - https://phabricator.wikimedia.org/T393016#10792799 (10Fabfur) 05Open→03Resolved p:05Triage→03Low [15:21:27] 06Traffic, 06Infrastructure-Foundations, 10Spicerack, 10SRE-tools, 13Patch-For-Review: Spicerack's Icinga module should provide a way to skip specific services in sub-optimal but desired state - https://phabricator.wikimedia.org/T392848#10792865 (10Volans) No, you're right, `current_state` in the icinga... [15:42:27] 06Traffic, 06Infrastructure-Foundations, 10Spicerack, 10SRE-tools, 13Patch-For-Review: Spicerack's Icinga module should provide a way to skip specific services in sub-optimal but desired state - https://phabricator.wikimedia.org/T392848#10792981 (10ssingh) >>! In T392848#10792865, @Volans wrote: > No, yo... [15:45:34] 06Traffic, 06serviceops, 10Content-Transform-Team (Work In Progress), 07Essential-Work, 13Patch-For-Review: Purging edge caches doesn't work for articles with ":" in their title - https://phabricator.wikimedia.org/T392849#10793002 (10Jgiannelos) 05Open→03Resolved I just verified this change in pr... [16:15:12] nemo-yiannis: ^^ nice to see that one resolved [16:19:40] 06Traffic, 10Data-Engineering (Q4 2025 April 1st - June 30th), 10DPE HAProxy Migration, 13Patch-For-Review: Add HAproxy termination field to webrequest - https://phabricator.wikimedia.org/T387454#10793139 (10JAllemandou) [16:25:16] vgutierrez: yeah, it took me abit [16:25:19] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: Upgrade codfw E/F Juniper equipment to Junos 23.x - https://phabricator.wikimedia.org/T393001#10793160 (10Papaul) 05Open→03Resolved a:03Papaul @ayounsi the solution here was to start a shell and run the commands below ` star... [21:04:49] 06Traffic, 06[Archived]Wikidata Dev Team, 10Prod-Kubernetes, 06SRE, and 5 others: Frequent 500 Errors and Timeouts When Adding Statements to New Item or Lexeme-typed Properties - https://phabricator.wikimedia.org/T374230#10793975 (10ArthurPSmith) Yes, P13552 did not result in delay, although I did wait som...