[13:53:12] (HAProxyRestarted) firing: HAProxy server restarted on cp1105:9100 - https://wikitech.wikimedia.org/wiki/HAProxy#HAProxy_for_edge_caching - https://grafana.wikimedia.org/d/gQblbjtnk/haproxy-drilldown?orgId=1&var-site=eqiad%20prometheus/ops&var-instance=cp1105&viewPanel=10 - https://alerts.wikimedia.org/?q=alertname%3DHAProxyRestarted [13:53:13] (PuppetConstantChange) firing: Puppet performing a change on every puppet run on cp1102:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange [13:59:21] ^ silencing [14:23:21] due to some bugs in the AQS2 services, we need to roll back the ATS changes partially for some services - this is a safer change (as it brings us back to what we had beforehand), shouldn't need a slow rollout https://gerrit.wikimedia.org/r/c/operations/puppet/+/972822 [14:31:13] hnowlan: lgtm! [14:31:53] thanks! [15:27:21] 10Traffic, 10DC-Ops, 10SRE, 10ops-eqiad: Q1:Install cp11[00-15] and rotate into production - https://phabricator.wikimedia.org/T349244 (10Fabfur) [16:33:06] 10Traffic, 10Patch-For-Review: Provide a TCP MSS clamping mechanism for real servers - https://phabricator.wikimedia.org/T350462 (10CodeReviewBot) vgutierrez opened https://gitlab.wikimedia.org/vgutierrez/tcp-mss-clamper/-/merge_requests/1 Provide basic functionality [17:09:12] 10Traffic, 10DC-Ops, 10SRE, 10ops-eqiad: Q1:Install cp11[00-15] and rotate into production - https://phabricator.wikimedia.org/T349244 (10Fabfur) [19:49:25] 10Traffic, 10SRE, 10Patch-For-Review: Upgrade Traffic hosts to bookworm - https://phabricator.wikimedia.org/T342154 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by brett@cumin2002 for host acmechief-test1001.eqiad.wmnet with OS bookworm [19:52:41] 10Traffic, 10SRE, 10Patch-For-Review: Upgrade Traffic hosts to bookworm - https://phabricator.wikimedia.org/T342154 (10BCornwall) [20:12:48] 10Traffic, 10DC-Ops, 10SRE, 10ops-eqiad: Q1:Install cp11[00-15] and rotate into production - https://phabricator.wikimedia.org/T349244 (10Fabfur) Problems encountered while checking and current situation (just a note for bookeeping): * Virtualization is enabled on cp1102 (`egrep -q "vmx|svm" /proc/cpuinf... [20:28:50] 10Traffic, 10SRE, 10Patch-For-Review: Upgrade Traffic hosts to bookworm - https://phabricator.wikimedia.org/T342154 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by brett@cumin2002 for host acmechief-test1001.eqiad.wmnet with OS bookworm completed: - acmechief-test1001 (**WARN**) - Do... [20:37:21] 10Traffic, 10SRE, 10Patch-For-Review: Upgrade Traffic hosts to bookworm - https://phabricator.wikimedia.org/T342154 (10BCornwall) [20:39:03] 10Traffic, 10SRE, 10GitLab (Project Migration): Migrate Traffic repositories from Gerrit to Gitlab - https://phabricator.wikimedia.org/T347623 (10BCornwall) [20:49:34] 10Traffic, 10SRE, 10GitLab (Project Migration): Migrate Traffic repositories from Gerrit to Gitlab - https://phabricator.wikimedia.org/T347623 (10BCornwall) Yes, indeed! Thanks for pointing that out.