[00:41:45] (HAProxyRestarted) firing: (2) HAProxy server restarted on cp2031:9100 - https://wikitech.wikimedia.org/wiki/HAProxy#HAProxy_for_edge_caching - https://alerts.wikimedia.org/?q=alertname%3DHAProxyRestarted [00:51:45] (HAProxyRestarted) firing: (2) HAProxy server restarted on cp2031:9100 - https://wikitech.wikimedia.org/wiki/HAProxy#HAProxy_for_edge_caching - https://alerts.wikimedia.org/?q=alertname%3DHAProxyRestarted [04:51:45] (HAProxyRestarted) firing: HAProxy server restarted on cp2035:9100 - https://wikitech.wikimedia.org/wiki/HAProxy#HAProxy_for_edge_caching - https://grafana.wikimedia.org/d/gQblbjtnk/haproxy-drilldown?orgId=1&var-site=codfw%20prometheus/ops&var-instance=cp2035&viewPanel=10 - https://alerts.wikimedia.org/?q=alertname%3DHAProxyRestarted [06:56:45] (HAProxyRestarted) firing: (2) HAProxy server restarted on cp2035:9100 - https://wikitech.wikimedia.org/wiki/HAProxy#HAProxy_for_edge_caching - https://alerts.wikimedia.org/?q=alertname%3DHAProxyRestarted [08:14:27] 10HTTPS, 10Traffic, 10SRE, 10serviceops, 10Abstract Wikipedia team (Phase λ – Launch): Get new edge & internal HTTPS certificates expanded to add wikifunctions.org and *.wikifunctions.org - https://phabricator.wikimedia.org/T313227 (10Clement_Goubert) [08:15:29] 10HTTPS, 10Traffic, 10SRE, 10serviceops, 10Abstract Wikipedia team (Phase λ – Launch): Get new edge & internal HTTPS certificates expanded to add wikifunctions.org and *.wikifunctions.org - https://phabricator.wikimedia.org/T313227 (10Clement_Goubert) New internal certs now include `wikifunctions.org` an... [08:48:42] 10netops, 10Cloud-Services, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: Modify Bird module to allow source IP to be passed to template - https://phabricator.wikimedia.org/T335760 (10ayounsi) > The obvious solution is to allow passing of the specific IP to use, and default to $facts['ipaddre... [09:31:06] 10Traffic, 10Infrastructure-Foundations, 10netbox, 10Patch-For-Review: gdnsd failures when converting services from active/passive to active/active - https://phabricator.wikimedia.org/T330084 (10jbond) 05Open→03Stalled Setting to stalled as i need to test the procedure in https://phabricator.wikimedia.... [10:56:45] (HAProxyRestarted) firing: (2) HAProxy server restarted on cp2035:9100 - https://wikitech.wikimedia.org/wiki/HAProxy#HAProxy_for_edge_caching - https://alerts.wikimedia.org/?q=alertname%3DHAProxyRestarted [11:04:37] 10Traffic, 10MW-on-K8s, 10SRE, 10serviceops, and 3 others: Migrate internal traffic to k8s - https://phabricator.wikimedia.org/T333120 (10Clement_Goubert) [11:04:48] 10Traffic, 10MW-on-K8s, 10SRE, 10serviceops, and 3 others: Migrate internal traffic to k8s - https://phabricator.wikimedia.org/T333120 (10Clement_Goubert) [13:09:16] 10Traffic, 10SRE, 10envoy, 10serviceops: Upgrade Envoy to supported version - https://phabricator.wikimedia.org/T300324 (10JMeybohm) [13:09:27] 10Traffic, 10SRE, 10envoy, 10serviceops: Upgrade Envoy to supported version - https://phabricator.wikimedia.org/T300324 (10JMeybohm) [13:09:38] 10Traffic, 10SRE, 10envoy, 10serviceops: Upgrade Envoy to supported version - https://phabricator.wikimedia.org/T300324 (10JMeybohm) [13:13:32] 10Traffic, 10MW-on-K8s, 10SRE, 10serviceops, and 3 others: Migrate internal traffic to k8s - https://phabricator.wikimedia.org/T333120 (10Clement_Goubert) [13:20:13] 10Traffic, 10SRE, 10envoy, 10serviceops: Upgrade Envoy to supported version - https://phabricator.wikimedia.org/T300324 (10JMeybohm) [13:20:41] 10Traffic, 10envoy, 10serviceops: Upgrade Envoy to supported version - https://phabricator.wikimedia.org/T300324 (10JMeybohm) a:05RLazarus→03JMeybohm [13:45:15] 10Traffic, 10SRE, 10envoy, 10serviceops, 10Patch-For-Review: Refactor envoy HTTP protocol options to new version - https://phabricator.wikimedia.org/T303230 (10JMeybohm) a:05RLazarus→03JMeybohm [13:46:43] 10Traffic, 10envoy, 10serviceops, 10Patch-For-Review: Refactor envoy access_log_path to access loggers - https://phabricator.wikimedia.org/T303231 (10JMeybohm) a:05RLazarus→03JMeybohm [14:08:40] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10SRE, and 2 others: Q1:(Need By: TBD) rack/setup/install cloudswift100[12] - https://phabricator.wikimedia.org/T289882 (10Andrew) @papaul, note that these hosts are still pending some trial work in codfw1dev so you shouldn't spend any effort on these hosts... [14:11:45] (HAProxyRestarted) firing: (2) HAProxy server restarted on cp2035:9100 - https://wikitech.wikimedia.org/wiki/HAProxy#HAProxy_for_edge_caching - https://alerts.wikimedia.org/?q=alertname%3DHAProxyRestarted [14:46:24] 10Traffic, 10ops-codfw, 10Patch-For-Review: Q4:rack/decom codfw unified decommission task - https://phabricator.wikimedia.org/T335777 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by sukhe@cumin2002 for hosts: `lvs2007.codfw.wmnet` - lvs2007.codfw.wmnet (**WARN**) - Downtimed host on Ici... [14:51:34] 10Traffic, 10ops-codfw, 10Patch-For-Review: Q4:rack/decom codfw unified decommission task - https://phabricator.wikimedia.org/T335777 (10ssingh) [15:02:41] 10netops, 10Infrastructure-Foundations: Store network users in Bitu/LDAP - https://phabricator.wikimedia.org/T335870 (10ayounsi) [15:03:57] 10netops, 10Infrastructure-Foundations: Store network users in Bitu/LDAP - https://phabricator.wikimedia.org/T335870 (10SLyngshede-WMF) p:05Triage→03Low a:03SLyngshede-WMF [16:01:45] (HAProxyRestarted) resolved: HAProxy server restarted on cp3062:9100 - https://wikitech.wikimedia.org/wiki/HAProxy#HAProxy_for_edge_caching - https://grafana.wikimedia.org/d/gQblbjtnk/haproxy-drilldown?orgId=1&var-site=esams%20prometheus/ops&var-instance=cp3062&viewPanel=10 - https://alerts.wikimedia.org/?q=alertname%3DHAProxyRestarted [16:43:47] 10Traffic, 10DC-Ops, 10SRE, 10ops-codfw, 10Patch-For-Review: Q4:rack/setup/install lvs2011, lvs2012, lvs2013, lvs2014 - https://phabricator.wikimedia.org/T326767 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host lvs2011.codfw.wmnet with OS bullseye [17:22:38] 10Traffic, 10DC-Ops, 10SRE, 10ops-codfw, 10Patch-For-Review: Q4:rack/setup/install lvs2011, lvs2012, lvs2013, lvs2014 - https://phabricator.wikimedia.org/T326767 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host lvs2011.codfw.wmnet with OS bullseye completed... [17:23:13] 10Traffic, 10DC-Ops, 10SRE, 10ops-codfw, 10Patch-For-Review: Q4:rack/setup/install lvs2011, lvs2012, lvs2013, lvs2014 - https://phabricator.wikimedia.org/T326767 (10Papaul) [17:55:15] 10Traffic, 10ops-codfw, 10Patch-For-Review: Q4:rack/decom codfw unified decommission task - https://phabricator.wikimedia.org/T335777 (10Papaul) @Jhancock.wm can you run the netbox offline script and get lvs2007 out of the rack and into storage ? Thanks [22:51:43] 10Traffic, 10Patch-For-Review: Switch Source Hashing ('sh') scheduling on LVS hosts to Maglev hashing ('mh') - https://phabricator.wikimedia.org/T263797 (10BCornwall)