[00:23:59] 10Traffic, 10DC-Ops, 10SRE, 10ops-eqsin, 10Patch-For-Review: Q2:rack/setup/install eqsin refresh - https://phabricator.wikimedia.org/T322048 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin2002 for host cp5029.eqsin.wmnet with OS buster [00:50:38] 10Traffic, 10DC-Ops, 10SRE, 10ops-eqsin, 10Patch-For-Review: Q2:rack/setup/install eqsin refresh - https://phabricator.wikimedia.org/T322048 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin2002 for host cp5029.eqsin.wmnet with OS buster [00:50:49] 10Traffic, 10DC-Ops, 10SRE, 10ops-eqsin, 10Patch-For-Review: Q2:rack/setup/install eqsin refresh - https://phabricator.wikimedia.org/T322048 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin2002 for host cp5029.eqsin.wmnet with OS buster executed with errors: - cp5029 (**... [00:51:39] 10Traffic, 10DC-Ops, 10SRE, 10ops-eqsin, 10Patch-For-Review: Q2:rack/setup/install eqsin refresh - https://phabricator.wikimedia.org/T322048 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin2002 for host cp5029.eqsin.wmnet with OS buster [01:08:40] 10Traffic, 10DC-Ops, 10SRE, 10ops-eqsin, 10Patch-For-Review: Q2:rack/setup/install eqsin refresh - https://phabricator.wikimedia.org/T322048 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin2002 for host cp5029.eqsin.wmnet with OS buster executed with errors: - cp5029 (**... [10:36:08] 10netops, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review, 10cloud-services-team (Kanban): Move WMCS servers to 1 NIC - https://phabricator.wikimedia.org/T319184 (10aborrero) [10:38:18] 10netops, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review, 10cloud-services-team (Kanban): Move WMCS servers to 1 NIC - https://phabricator.wikimedia.org/T319184 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by aborrero@cumin1001 for host cloudvirt1053.eqiad.wmnet with O... [11:22:04] 10netops, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review, 10cloud-services-team (Kanban): Move WMCS servers to 1 NIC - https://phabricator.wikimedia.org/T319184 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by aborrero@cumin1001 for host cloudvirt1053.eqiad.wmnet with OS bu... [11:22:14] 10netops, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review, 10cloud-services-team (Kanban): Move WMCS servers to 1 NIC - https://phabricator.wikimedia.org/T319184 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by aborrero@cumin1001 for host cloudvirt1053.eqiad.wmnet with OS bu... [12:20:41] 10netops, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review, 10cloud-services-team (Kanban): Move WMCS servers to 1 NIC - https://phabricator.wikimedia.org/T319184 (10aborrero) [12:27:50] 10netops, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review, 10cloud-services-team (Kanban): Move WMCS servers to 1 NIC - https://phabricator.wikimedia.org/T319184 (10aborrero) [12:45:05] 10Traffic, 10MW-on-K8s, 10SRE, 10serviceops, and 3 others: Stop spamming SAL with helmfile on scap deployments - https://phabricator.wikimedia.org/T323296 (10Clement_Goubert) 05In progress→03Resolved [12:45:17] 10Traffic, 10MW-on-K8s, 10SRE, 10serviceops, and 3 others: Deploy mediawiki kubernetes services - https://phabricator.wikimedia.org/T321786 (10Clement_Goubert) [12:54:00] 10netops, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review, 10cloud-services-team (Kanban): Move WMCS servers to 1 NIC - https://phabricator.wikimedia.org/T319184 (10aborrero) [13:10:13] 10netops, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review, 10cloud-services-team (Kanban): Move WMCS servers to 1 NIC - https://phabricator.wikimedia.org/T319184 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by aborrero@cumin1001 for host cloudvirt1052.eqiad.wmnet with O... [13:53:54] 10netops, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review, 10cloud-services-team (Kanban): Move WMCS servers to 1 NIC - https://phabricator.wikimedia.org/T319184 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by aborrero@cumin1001 for host cloudvirt1052.eqiad.wmnet with OS bu... [14:47:56] godog: T321678 triggers a puppet message on every run, "Info: /Stage[main]/Confd/Tidy[/var/run/confd-template]: File does not exist" [14:47:56] T321678: Clean up stale/old confd errors automatically - https://phabricator.wikimedia.org/T321678 [14:48:00] is this expected? [14:48:08] aka we should just ignore the message? :) [14:49:07] ah mmmh yeah because that directory is created by confd on error [14:49:36] safe to ignore for now but I'm thinking of adding a tmpfiles entry to create the directory ourselves [14:49:49] shouldn't hurt anyways [14:50:03] what do you think vgutierrez ? [14:51:51] (meeting in 10) [14:54:18] hmm /var/run/conf-template exists [14:54:28] but it's empty [14:59:15] ah! mmhh on all hosts? I tried just now on puppetmaster1001 and I'm not getting that info from puppet [14:59:48] meeting, bbiab [15:00:59] godog: hmm right.. just on cp hosts without confd.. aka WMCS instances [15:01:01] forget about it :) [15:03:29] or not... [15:03:32] cp5020 just issued Info: /Stage[main]/Confd/Tidy[/var/run/confd-template]: File does not exist [15:04:55] ls: cannot access '/var/run/confd-template': No such file or directory [15:04:59] weird [15:06:26] 10netops, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review, 10cloud-services-team (Kanban): Move WMCS servers to 1 NIC - https://phabricator.wikimedia.org/T319184 (10aborrero) [15:15:27] 10Traffic, 10DC-Ops, 10SRE, 10ops-eqsin, 10Patch-For-Review: Q2:rack/setup/install eqsin refresh - https://phabricator.wikimedia.org/T322048 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin2002 for host cp5030.eqsin.wmnet with OS buster [15:20:13] 10netops, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review, 10cloud-services-team (Kanban): Move WMCS servers to 1 NIC - https://phabricator.wikimedia.org/T319184 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by aborrero@cumin1001 for host cloudvirt1051.eqiad.wmnet with O... [15:35:19] yeah that's because confd creates the directory on demand, so at reboot it is unlikely to exist [15:35:27] I'll send the tmpfiles patch, might as well [15:59:11] godog: I've cleaned the /srv/prometheus/ops/targets/ directory a little bit.. there were some pieces there from ats-tls and envoy :_) [16:00:57] cheers! yeah there's some cruft in there alright [16:01:07] victim of its on success (?) [16:04:24] 10netops, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review, 10cloud-services-team (Kanban): Move WMCS servers to 1 NIC - https://phabricator.wikimedia.org/T319184 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by aborrero@cumin1001 for host cloudvirt1051.eqiad.wmnet with OS bu... [16:04:47] and as expected https://gerrit.wikimedia.org/r/c/operations/puppet/+/859090 was a noop on prometheus3001 (just ran puppet there) [16:05:17] 10Traffic, 10SRE, 10Patch-For-Review: Rename role::cache::(text|upload)_haproxy to role::cache::(text|upload) - https://phabricator.wikimedia.org/T323365 (10Vgutierrez) 05Open→03Resolved [16:05:26] 10Traffic, 10SRE, 10Patch-For-Review, 10Performance-Team (Radar): Test haproxy as a WMF's CDN TLS terminator with real traffic - https://phabricator.wikimedia.org/T290005 (10Vgutierrez) [16:07:02] neato [16:17:10] 10Traffic, 10DC-Ops, 10SRE, 10ops-eqsin, 10Patch-For-Review: Q2:rack/setup/install eqsin refresh - https://phabricator.wikimedia.org/T322048 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin2002 for host cp5030.eqsin.wmnet with OS buster completed: - cp5030 (**PASS**) -... [16:50:01] 10netops, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review, 10cloud-services-team (Kanban): Move WMCS servers to 1 NIC - https://phabricator.wikimedia.org/T319184 (10aborrero) [17:22:33] 10Traffic, 10DC-Ops, 10SRE, 10ops-ulsfo, 10Patch-For-Review: Q1:rack/setup/install ulsfo misc class hosts - https://phabricator.wikimedia.org/T317247 (10RobH) [17:30:33] 10Traffic, 10DC-Ops, 10SRE, 10ops-ulsfo, 10Patch-For-Review: Q1:rack/setup/install ulsfo misc class hosts - https://phabricator.wikimedia.org/T317247 (10RobH) [18:28:27] 10Traffic, 10WMF-NDA: Unblock port 80 on AS 35039 - https://phabricator.wikimedia.org/T323525 (10BCornwall) [19:04:24] 10Traffic, 10DC-Ops, 10SRE, 10ops-eqsin, 10Patch-For-Review: Q2:rack/setup/install eqsin refresh - https://phabricator.wikimedia.org/T322048 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by brett@cumin1001 for host cp5031.eqsin.wmnet with OS buster [20:06:13] 10Traffic, 10DC-Ops, 10SRE, 10ops-eqsin, 10Patch-For-Review: Q2:rack/setup/install eqsin refresh - https://phabricator.wikimedia.org/T322048 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by brett@cumin1001 for host cp5031.eqsin.wmnet with OS buster completed: - cp5031 (**PASS**) -... [20:11:11] 10Traffic, 10DC-Ops, 10SRE, 10ops-eqsin, 10Patch-For-Review: Q2:rack/setup/install eqsin refresh - https://phabricator.wikimedia.org/T322048 (10ssingh) [20:13:18] 10Traffic, 10DC-Ops, 10SRE, 10ops-eqsin, 10Patch-For-Review: Q2:rack/setup/install eqsin refresh - https://phabricator.wikimedia.org/T322048 (10BCornwall) [20:14:00] 10Traffic, 10DC-Ops, 10SRE, 10ops-eqsin, 10Patch-For-Review: Q2:rack/setup/install eqsin refresh - https://phabricator.wikimedia.org/T322048 (10ssingh)