[06:13:18] 06Traffic, 10Data-Engineering (Q4 2025 April 1st - June 30th), 10DPE HAProxy Migration: Add HAproxy termination field to webrequest - https://phabricator.wikimedia.org/T387454#10784953 (10JAllemandou) a:03JAllemandou [06:24:18] 06Traffic, 10Data-Engineering (Q4 2025 April 1st - June 30th), 10DPE HAProxy Migration: Add HAproxy termination field to webrequest - https://phabricator.wikimedia.org/T387454#10784964 (10JAllemandou) Hi @Fabfur , sorry for the late answer :) I'm adding a PR for you to validate. There are multiple places whe... [08:19:43] 06Traffic, 13Patch-For-Review: haproxykafka service isn't restarted when upgraded - https://phabricator.wikimedia.org/T393016#10785224 (10MoritzMuehlenhoff) >>! In T393016#10780769, @CodeReviewBot wrote: > fabfur opened https://gitlab.wikimedia.org/repos/sre/haproxykafka/-/merge_requests/84 > > use systemd un... [08:22:24] 06Traffic, 13Patch-For-Review: haproxykafka service isn't restarted when upgraded - https://phabricator.wikimedia.org/T393016#10785231 (10Fabfur) Sorry that was a wrong MR that ended up here... Relevant changes are on the `bullseye-wikimedia` branch with https://gitlab.wikimedia.org/repos/sre/haproxykafka/-/co... [08:49:26] 10netops, 06Infrastructure-Foundations, 10Observability-Alerting, 13Patch-For-Review: Migrate network icinga alerts to gNMI/prometheus - https://phabricator.wikimedia.org/T388641#10785296 (10ayounsi) a (better for now) alternative to `/components/` is to use Juniper's `/junos/system/linecard/optics/` Bett... [09:27:15] 06Traffic, 06serviceops, 10Content-Transform-Team (Work In Progress): Purging edge caches doesn't work for articles with ":" in their title - https://phabricator.wikimedia.org/T392849#10785506 (10Jgiannelos) After some debugging it turns out that the problem is (a race condition ?) that `rest.php` `page/ 06Traffic, 06serviceops, 10Content-Transform-Team (Work In Progress), 13Patch-For-Review: Purging edge caches doesn't work for articles with ":" in their title - https://phabricator.wikimedia.org/T392849#10785510 (10Jgiannelos) Also the caching middleware part: https://gitlab.wikimedia.org/repos/content-tr... [13:27:49] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: Upgrade codfw E/F Juniper equipment to Junos 23.x - https://phabricator.wikimedia.org/T393001#10786078 (10Papaul) on lsw1-f1-codfw we are having this error ` ERROR: There is pending upgrade. upgrade_in_progress=appupgrade_stage ERROR:... [13:30:58] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: Upgrade codfw E/F Juniper equipment to Junos 23.x - https://phabricator.wikimedia.org/T393001#10786092 (10ayounsi) Thx, I tried this, let's see if it helps: ` lsw1-f1-eqiad> request system software rollback localre: ----------... [15:06:12] FIRING: SLOMetricAbsent: varnish-combined magru - https://slo.wikimedia.org/?search=varnish-combined - https://alerts.wikimedia.org/?q=alertname%3DSLOMetricAbsent [15:06:27] ^ I am assuming this is due to the prom issues, olly is looking into it [15:06:32] FIRING: SLOMetricAbsent: haproxy-combined - https://slo.wikimedia.org/?search=haproxy-combined - https://alerts.wikimedia.org/?q=alertname%3DSLOMetricAbsent [15:07:42] FIRING: SLOMetricAbsent: varnish-combined magru - https://slo.wikimedia.org/?search=varnish-combined - https://alerts.wikimedia.org/?q=alertname%3DSLOMetricAbsent [15:11:32] FIRING: [2x] SLOMetricAbsent: haproxy-combined - https://slo.wikimedia.org/?search=haproxy-combined - https://alerts.wikimedia.org/?q=alertname%3DSLOMetricAbsent [15:16:12] RESOLVED: SLOMetricAbsent: varnish-combined magru - https://slo.wikimedia.org/?search=varnish-combined - https://alerts.wikimedia.org/?q=alertname%3DSLOMetricAbsent [15:16:32] RESOLVED: [2x] SLOMetricAbsent: haproxy-combined - https://slo.wikimedia.org/?search=haproxy-combined - https://alerts.wikimedia.org/?q=alertname%3DSLOMetricAbsent [15:17:42] RESOLVED: SLOMetricAbsent: varnish-combined magru - https://slo.wikimedia.org/?search=varnish-combined - https://alerts.wikimedia.org/?q=alertname%3DSLOMetricAbsent [16:57:24] 10netops, 06Infrastructure-Foundations, 06SRE: Do we need prometheus-ethtool-exporter? - https://phabricator.wikimedia.org/T371375#10786790 (10jhathaway) We packaged `prometheus-ethtool-exporter` ourselves, so removing it from our stack would also remove a small maintenance burden, e.g. we haven't packaged i... [22:38:17] 06Traffic, 07PM: Create useful Phabricator task workflow for Traffic - https://phabricator.wikimedia.org/T370557#10787566 (10BCornwall) 05Open→03Stalled