[00:41:05] 10Traffic, 10DC-Ops, 10SRE, 10ops-eqsin, 10Patch-For-Review: Q2:rack/setup/install eqsin refresh - https://phabricator.wikimedia.org/T322048 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin2002 for host cp5021.eqsin.wmnet with OS buster [01:48:16] 10Traffic, 10DC-Ops, 10SRE, 10ops-eqsin, 10Patch-For-Review: Q2:rack/setup/install eqsin refresh - https://phabricator.wikimedia.org/T322048 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin2002 for host cp5021.eqsin.wmnet with OS buster completed: - cp5021 (**WARN**) -... [01:48:42] 10Traffic, 10DC-Ops, 10SRE, 10ops-eqsin, 10Patch-For-Review: Q2:rack/setup/install eqsin refresh - https://phabricator.wikimedia.org/T322048 (10ssingh) [08:16:02] 10netops, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: ICMPv6 'TTL Exceeded' messages are not generated by row E/F switches due to loopback filter - https://phabricator.wikimedia.org/T324033 (10ayounsi) Thanks! Patch reviewed, it's always great to remove unnecessary config! :) For the record a... [09:11:03] 10netops, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: ICMPv6 'TTL Exceeded' messages are not generated by row E/F switches due to loopback filter - https://phabricator.wikimedia.org/T324033 (10cmooney) 05Open→03Resolved Thanks for the review @ayounsi >>! In T324033#8430782, @ayounsi wrot... [09:15:31] 10netops, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: ICMPv6 'TTL Exceeded' messages are not generated by row E/F switches due to loopback filter - https://phabricator.wikimedia.org/T324033 (10taavi) [14:03:29] godog: /var/run/reload-vcl-state contains the old OK string.. hence the issue [14:03:46] yeah, I'll send a patch to check for both old and new status [14:03:52] seems the easiest [14:05:17] that or keep OK/KO and map that to 1/0 for the .prom file [14:06:18] yeah since we'll be ditching the nagios script altogether might as well change that [14:06:21] https://gerrit.wikimedia.org/r/c/operations/puppet/+/862266 [14:08:01] godog: https://www.shellcheck.net/wiki/SC2166 [14:08:55] fixing [14:09:17] since CI does run shell check, should we be failing on warnings too ? [14:12:04] I don't know how picky is shellcheck regarding warnings [14:12:06] but probably [14:12:44] ack, fixed SC2166 [14:12:55] +1ed [14:14:40] cheers, will check on an host and do the rest, thanks vgutierrez for the assist [14:14:46] np [14:15:02] I also assisted in breaking it 😅 [14:15:20] heheh fair enough [14:19:40] yep verified the fix on cp3051 and reenabled puppet [14:19:44] the irony of the fact that I'm fixing the irc alert spam while causing more spam is not lost on me [14:20:13] :) [14:20:25] handling alerts can trigger alerts spam [14:21:07] indeed, quite easily [14:30:20] oh ok, so the confd already are known and from that change, good :) [14:30:26] I was worried there for a second ha [15:33:32] FYI I'm deploying https://gerrit.wikimedia.org/r/c/operations/puppet/+/862258 and will roll-restart / !log pybal in eqiad/codfw [15:34:38] godog: see PM :) [15:49:40] all good FWIW [15:50:00] thanks! [16:33:38] (LVSHighCPU) firing: (8) The host lvs5002:9100 has at least its CPU 0 saturated - https://bit.ly/wmf-lvscpu - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=lvs5002 - https://alerts.wikimedia.org/?q=alertname%3DLVSHighCPU [16:38:38] (LVSHighCPU) resolved: (8) The host lvs5002:9100 has at least its CPU 0 saturated - https://bit.ly/wmf-lvscpu - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=lvs5002 - https://alerts.wikimedia.org/?q=alertname%3DLVSHighCPU [17:32:11] 10Traffic, 10DC-Ops, 10SRE, 10ops-eqsin, 10Patch-For-Review: Q2:rack/setup/install eqsin refresh - https://phabricator.wikimedia.org/T322048 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin2002 for host cp5022.eqsin.wmnet with OS buster [18:34:23] 10Traffic, 10DC-Ops, 10SRE, 10ops-eqsin, 10Patch-For-Review: Q2:rack/setup/install eqsin refresh - https://phabricator.wikimedia.org/T322048 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin2002 for host cp5022.eqsin.wmnet with OS buster completed: - cp5022 (**PASS**) -... [18:36:32] 10Traffic, 10DC-Ops, 10SRE, 10ops-eqsin, 10Patch-For-Review: Q2:rack/setup/install eqsin refresh - https://phabricator.wikimedia.org/T322048 (10ssingh) [19:09:09] 10Traffic, 10DC-Ops, 10SRE, 10ops-eqsin, 10Patch-For-Review: Q2:rack/setup/install eqsin refresh - https://phabricator.wikimedia.org/T322048 (10RobH) [19:20:19] 10netops, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review, 10cloud-services-team (Kanban): Move WMCS servers to 1 NIC - https://phabricator.wikimedia.org/T319184 (10cmooney) On the back of the meeting earlier and our discussion around Ceph I decided to look a little bit closer into the heartbeat... [19:38:34] 10Traffic, 10DC-Ops, 10SRE, 10ops-eqsin, 10Patch-For-Review: Q2:rack/setup/install eqsin refresh - https://phabricator.wikimedia.org/T322048 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by brett@cumin1001 for host cp5023.eqsin.wmnet with OS buster [19:48:59] 10Traffic, 10DC-Ops, 10SRE, 10ops-eqsin, 10Patch-For-Review: Q2:rack/setup/install eqsin refresh - https://phabricator.wikimedia.org/T322048 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by robh@cumin2002 for host ganeti5004.eqsin.wmnet with OS bullseye [20:41:51] 10Traffic, 10DC-Ops, 10SRE, 10ops-eqsin, 10Patch-For-Review: Q2:rack/setup/install eqsin refresh - https://phabricator.wikimedia.org/T322048 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by robh@cumin2002 for host ganeti5004.eqsin.wmnet with OS bullseye completed: - ganeti5004 (**PA... [20:42:13] 10Traffic, 10DC-Ops, 10SRE, 10ops-eqsin, 10Patch-For-Review: Q2:rack/setup/install eqsin refresh - https://phabricator.wikimedia.org/T322048 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by brett@cumin1001 for host cp5023.eqsin.wmnet with OS buster completed: - cp5023 (**PASS**) -... [20:42:58] 10Traffic, 10DC-Ops, 10SRE, 10ops-eqsin, 10Patch-For-Review: Q2:rack/setup/install eqsin refresh - https://phabricator.wikimedia.org/T322048 (10RobH) [20:44:53] 10Traffic, 10DC-Ops, 10SRE, 10ops-eqsin, 10Patch-For-Review: Q2:rack/setup/install eqsin refresh - https://phabricator.wikimedia.org/T322048 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by brett@cumin1001 for host cp5024.eqsin.wmnet with OS buster [21:32:02] 10Traffic, 10DC-Ops, 10SRE, 10ops-eqsin, 10Patch-For-Review: Q2:rack/setup/install eqsin refresh - https://phabricator.wikimedia.org/T322048 (10RobH) [21:47:19] 10Traffic, 10DC-Ops, 10SRE, 10ops-eqsin, 10Patch-For-Review: Q2:rack/setup/install eqsin refresh - https://phabricator.wikimedia.org/T322048 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by brett@cumin1001 for host cp5024.eqsin.wmnet with OS buster completed: - cp5024 (**PASS**) -... [22:02:10] 10Traffic, 10DC-Ops, 10SRE, 10ops-eqsin, 10Patch-For-Review: Q2:rack/setup/install eqsin refresh - https://phabricator.wikimedia.org/T322048 (10RobH) [22:18:40] 10Traffic, 10DC-Ops, 10SRE, 10ops-eqsin, 10Patch-For-Review: Q2:rack/setup/install eqsin refresh - https://phabricator.wikimedia.org/T322048 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by brett@cumin1001 for host cp5025.eqsin.wmnet with OS buster [22:19:58] 10Traffic, 10DC-Ops, 10SRE, 10ops-eqsin, 10Patch-For-Review: Q2:rack/setup/install eqsin refresh - https://phabricator.wikimedia.org/T322048 (10RobH) [23:22:12] 10Traffic, 10DC-Ops, 10SRE, 10ops-eqsin, 10Patch-For-Review: Q2:rack/setup/install eqsin refresh - https://phabricator.wikimedia.org/T322048 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by brett@cumin1001 for host cp5025.eqsin.wmnet with OS buster completed: - cp5025 (**WARN**) -...