[06:52:03] 10Traffic, 10Observability-Metrics, 10Patch-For-Review: Add prometheus-https load balancer - https://phabricator.wikimedia.org/T326657 (10fgiunchedi) >>! In T326657#9102218, @herron wrote: >>>! In T326657#9095309, @fgiunchedi wrote: >> I think we'll need two distinct services there (i.e. to be able to change... [08:14:50] 10netops, 10Infrastructure-Foundations, 10SRE: Default allowed SSH parameters on upgraded Juniper mgmt routers prevent some connections - https://phabricator.wikimedia.org/T320272 (10ayounsi) For the record, another possible workaround: ` mr1-esams> start shell % ssh root@10.80.128.6 -m hmac-... [08:17:33] 10netops, 10Infrastructure-Foundations, 10SRE: Implement better filter on BGP_Customer_out - https://phabricator.wikimedia.org/T340448 (10ayounsi) 05Open→03Resolved All done. [08:33:12] 10Traffic, 10MW-on-K8s, 10SRE, 10serviceops, 10Release-Engineering-Team (Seen): Serve production traffic via Kubernetes - https://phabricator.wikimedia.org/T290536 (10JMeybohm) [09:38:13] (HAProxyRestarted) firing: (2) HAProxy server restarted on cp3066:9100 - https://wikitech.wikimedia.org/wiki/HAProxy#HAProxy_for_edge_caching - https://alerts.wikimedia.org/?q=alertname%3DHAProxyRestarted [09:48:13] (HAProxyRestarted) firing: (2) HAProxy server restarted on cp3066:9100 - https://wikitech.wikimedia.org/wiki/HAProxy#HAProxy_for_edge_caching - https://alerts.wikimedia.org/?q=alertname%3DHAProxyRestarted [09:53:13] (HAProxyRestarted) firing: (3) HAProxy server restarted on cp3066:9100 - https://wikitech.wikimedia.org/wiki/HAProxy#HAProxy_for_edge_caching - https://alerts.wikimedia.org/?q=alertname%3DHAProxyRestarted [09:58:23] ^^ this is me [10:08:11] (SystemdUnitFailed) firing: clean-confd-rundir.service Failed on cp3068:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [10:08:13] (HAProxyRestarted) firing: (3) HAProxy server restarted on cp3067:9100 - https://wikitech.wikimedia.org/wiki/HAProxy#HAProxy_for_edge_caching - https://alerts.wikimedia.org/?q=alertname%3DHAProxyRestarted [10:18:13] (HAProxyRestarted) firing: (4) HAProxy server restarted on cp3067:9100 - https://wikitech.wikimedia.org/wiki/HAProxy#HAProxy_for_edge_caching - https://alerts.wikimedia.org/?q=alertname%3DHAProxyRestarted [10:28:14] (HAProxyRestarted) firing: (5) HAProxy server restarted on cp3067:9100 - https://wikitech.wikimedia.org/wiki/HAProxy#HAProxy_for_edge_caching - https://alerts.wikimedia.org/?q=alertname%3DHAProxyRestarted [10:33:11] (SystemdUnitFailed) resolved: clean-confd-rundir.service Failed on cp3068:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [10:43:11] (SystemdUnitFailed) firing: (2) clean-confd-rundir.service Failed on cp3068:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [10:43:13] (HAProxyRestarted) firing: (6) HAProxy server restarted on cp3067:9100 - https://wikitech.wikimedia.org/wiki/HAProxy#HAProxy_for_edge_caching - https://alerts.wikimedia.org/?q=alertname%3DHAProxyRestarted [10:48:14] (HAProxyRestarted) firing: (7) HAProxy server restarted on cp3067:9100 - https://wikitech.wikimedia.org/wiki/HAProxy#HAProxy_for_edge_caching - https://alerts.wikimedia.org/?q=alertname%3DHAProxyRestarted [10:58:13] (HAProxyRestarted) firing: (8) HAProxy server restarted on cp3067:9100 - https://wikitech.wikimedia.org/wiki/HAProxy#HAProxy_for_edge_caching - https://alerts.wikimedia.org/?q=alertname%3DHAProxyRestarted [11:03:11] (SystemdUnitFailed) resolved: clean-confd-rundir.service Failed on cp3071:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [11:03:13] (HAProxyRestarted) firing: (8) HAProxy server restarted on cp3067:9100 - https://wikitech.wikimedia.org/wiki/HAProxy#HAProxy_for_edge_caching - https://alerts.wikimedia.org/?q=alertname%3DHAProxyRestarted [11:08:14] (HAProxyRestarted) resolved: (8) HAProxy server restarted on cp3067:9100 - https://wikitech.wikimedia.org/wiki/HAProxy#HAProxy_for_edge_caching - https://alerts.wikimedia.org/?q=alertname%3DHAProxyRestarted [11:35:43] 10netops, 10Infrastructure-Foundations, 10SRE: Add non-EVPN L3 Switch routing policy definitions to Homer - https://phabricator.wikimedia.org/T344601 (10cmooney) p:05Triage→03Low [11:35:54] 10netops, 10Infrastructure-Foundations, 10SRE: Add non-EVPN L3 Switch routing policy definitions to Homer - https://phabricator.wikimedia.org/T344601 (10cmooney) [11:36:01] 10netops, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: Consolidate Automation Templates for DC Switches - https://phabricator.wikimedia.org/T312635 (10cmooney) [12:53:35] 10Traffic, 10SRE: NIC autonegotiation takes 4s in esams - https://phabricator.wikimedia.org/T344604 (10Fabfur) [12:56:51] 10Traffic, 10SRE: NIC autonegotiation takes 4s in esams - https://phabricator.wikimedia.org/T344604 (10cmooney) Hi @Fabfur it won't be anything related to configuration on either side I would expect. Possibly the different JunOS version or NIC firmware (I'd say latter more likely). Or maybe to do with the pa... [13:33:11] (SystemdUnitFailed) firing: clean-confd-rundir.service Failed on cp3075:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [14:03:10] (SystemdUnitFailed) resolved: clean-confd-rundir.service Failed on cp3075:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [14:07:56] 10Traffic, 10Infrastructure-Foundations: NIC autonegotiation takes 4s in esams - https://phabricator.wikimedia.org/T344604 (10ayounsi) [15:47:03] 10Traffic, 10Content-Transform-Team-WIP, 10Mobile-Content-Service, 10RESTbase Sunsetting, and 3 others: Setup allowed list for MCS decom - https://phabricator.wikimedia.org/T340036 (10MSantos) [19:16:42] (SystemdUnitFailed) firing: clean-confd-rundir.service Failed on cp5017:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [19:21:42] (SystemdUnitFailed) resolved: clean-confd-rundir.service Failed on cp5017:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed