[00:19:09] FIRING: LVSHighCPU: The host lvs7001:9100 has at least its CPU 36 saturated - https://bit.ly/wmf-lvscpu - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=lvs7001 - https://alerts.wikimedia.org/?q=alertname%3DLVSHighCPU [00:24:09] RESOLVED: LVSHighCPU: The host lvs7001:9100 has at least its CPU 36 saturated - https://bit.ly/wmf-lvscpu - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=lvs7001 - https://alerts.wikimedia.org/?q=alertname%3DLVSHighCPU [03:45:09] FIRING: LVSHighRX: Excessive RX traffic on lvs2013:9100 (eno12399np0) - https://bit.ly/wmf-lvsrx - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=lvs2013 - https://alerts.wikimedia.org/?q=alertname%3DLVSHighRX [03:50:09] RESOLVED: LVSHighRX: Excessive RX traffic on lvs2013:9100 (eno12399np0) - https://bit.ly/wmf-lvsrx - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=lvs2013 - https://alerts.wikimedia.org/?q=alertname%3DLVSHighRX [07:46:36] 06Traffic, 06collaboration-services, 10MinT, 10LPL Essential (LPL Essential 2025 Feb-Mar), 13Patch-For-Review: MinT: Fails to download models/files from peopleweb.discovery.wmnet - https://phabricator.wikimedia.org/T383750#10594909 (10KartikMistry) >>! In T383750#10591779, @Dzahn wrote: > Just to cla... [08:33:10] 06Traffic: Gather site pooled/depooled information for Grafana - https://phabricator.wikimedia.org/T376876#10594995 (10Fabfur) I think this could be interesting for @CDobbins , we can sync up on the work already done, if you're interested (cc @ssingh ) [09:43:38] 06Traffic, 10Data-Engineering (Q3 2024 January 1st - March 31th), 07Essential-Work, 10Experimentation Lab Radar: Cookie % has been rejected because it is foreign and does not have the "Partitioned" attribute - https://phabricator.wikimedia.org/T375256#10595181 (10Aklapper) p:05Unbreak!→03High Resetting... [13:36:48] 06Traffic, 13Patch-For-Review: Allow acmecerts to deploy certificates in tmpfs storage - https://phabricator.wikimedia.org/T384227#10595935 (10Fabfur) [13:45:31] 10netops, 06Infrastructure-Foundations, 10Observability-Alerting, 13Patch-For-Review, 10SRE Observability (FY2024/2025-Q3): Prometheus: attach host's BGP/interface remote side metrics - https://phabricator.wikimedia.org/T387287#10595966 (10fgiunchedi) >>! In T387287#10583672, @cmooney wrote: >>>! In T387... [13:50:23] 10Wikimedia-Apache-configuration, 10Observability-Logging, 06SRE: Gain visibility into httpd mod_proxy actions - https://phabricator.wikimedia.org/T188601#10595978 (10fgiunchedi) 05Open→03Declined >>! In T188601#10446174, @andrea.denisse wrote: > Is this related to T187434 ? It is not, the work is t... [14:15:09] FIRING: LVSHighRX: Excessive RX traffic on lvs2013:9100 (eno12399np0) - https://bit.ly/wmf-lvsrx - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=lvs2013 - https://alerts.wikimedia.org/?q=alertname%3DLVSHighRX [14:25:09] RESOLVED: LVSHighRX: Excessive RX traffic on lvs2013:9100 (eno12399np0) - https://bit.ly/wmf-lvsrx - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=lvs2013 - https://alerts.wikimedia.org/?q=alertname%3DLVSHighRX [14:30:09] FIRING: LVSHighRX: Excessive RX traffic on lvs2013:9100 (eno12399np0) - https://bit.ly/wmf-lvsrx - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=lvs2013 - https://alerts.wikimedia.org/?q=alertname%3DLVSHighRX [14:45:09] RESOLVED: LVSHighRX: Excessive RX traffic on lvs2013:9100 (eno12399np0) - https://bit.ly/wmf-lvsrx - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=lvs2013 - https://alerts.wikimedia.org/?q=alertname%3DLVSHighRX [16:07:44] 10netops, 06Infrastructure-Foundations, 10Observability-Alerting, 13Patch-For-Review, 10SRE Observability (FY2024/2025-Q3): Prometheus: attach host's BGP/interface remote side metrics - https://phabricator.wikimedia.org/T387287#10596744 (10ayounsi) p:05Triage→03Medium a:03ayounsi [16:34:09] FIRING: LVSHighRX: Excessive RX traffic on lvs2013:9100 (eno12399np0) - https://bit.ly/wmf-lvsrx - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=lvs2013 - https://alerts.wikimedia.org/?q=alertname%3DLVSHighRX [16:44:09] RESOLVED: LVSHighRX: Excessive RX traffic on lvs2013:9100 (eno12399np0) - https://bit.ly/wmf-lvsrx - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=lvs2013 - https://alerts.wikimedia.org/?q=alertname%3DLVSHighRX [17:59:51] 06Traffic, 13Patch-For-Review: Upgrade Varnish from 6.0 to 7.1 - https://phabricator.wikimedia.org/T378737#10597478 (10BCornwall) [18:07:02] 10netops, 06Infrastructure-Foundations: Different BFD settings on direct connected links - https://phabricator.wikimedia.org/T387773 (10Papaul) 03NEW [18:30:11] 10netops, 06Infrastructure-Foundations: Different BFD settings on direct connected links - https://phabricator.wikimedia.org/T387773#10597633 (10cmooney) Hmm good spot. I would say it's not absolutely essential to have it, as OSPF timers are quite short anyway, but also it doesn't really hurt. There may also... [18:56:09] FIRING: LVSHighRX: Excessive RX traffic on lvs2013:9100 (eno12399np0) - https://bit.ly/wmf-lvsrx - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=lvs2013 - https://alerts.wikimedia.org/?q=alertname%3DLVSHighRX [18:56:37] 06Traffic, 13Patch-For-Review: Update South America geo-maps - https://phabricator.wikimedia.org/T387774#10597851 (10Aklapper) [19:06:09] RESOLVED: LVSHighRX: Excessive RX traffic on lvs2013:9100 (eno12399np0) - https://bit.ly/wmf-lvsrx - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=lvs2013 - https://alerts.wikimedia.org/?q=alertname%3DLVSHighRX [19:20:34] 06Traffic, 13Patch-For-Review: Update South America geo-maps - https://phabricator.wikimedia.org/T387774#10597929 (10ssingh) p:05Triage→03Medium [19:36:09] FIRING: LVSHighRX: Excessive RX traffic on lvs2013:9100 (eno12399np0) - https://bit.ly/wmf-lvsrx - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=lvs2013 - https://alerts.wikimedia.org/?q=alertname%3DLVSHighRX [19:41:09] RESOLVED: LVSHighRX: Excessive RX traffic on lvs2013:9100 (eno12399np0) - https://bit.ly/wmf-lvsrx - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=lvs2013 - https://alerts.wikimedia.org/?q=alertname%3DLVSHighRX [21:26:49] 06Traffic, 13Patch-For-Review: Allow acmecerts to deploy certificates in tmpfs storage - https://phabricator.wikimedia.org/T384227#10598340 (10Fabfur) [23:22:51] 06Traffic, 06Infrastructure-Foundations: Create systemd path unit - https://phabricator.wikimedia.org/T387799 (10Fabfur) 03NEW [23:51:09] FIRING: LVSHighRX: Excessive RX traffic on lvs2013:9100 (eno12399np0) - https://bit.ly/wmf-lvsrx - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=lvs2013 - https://alerts.wikimedia.org/?q=alertname%3DLVSHighRX