[00:37:01] (SystemdUnitFailed) firing: (2) update-ubuntu-mirror.service on mirror1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [01:50:07] (PuppetConstantChange) firing: (2) Puppet performing a change on every puppet run on testvm2005:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange [04:37:16] (SystemdUnitFailed) firing: (2) update-ubuntu-mirror.service on mirror1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [05:50:07] (PuppetConstantChange) firing: (2) Puppet performing a change on every puppet run on testvm2005:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange [06:32:01] (SystemdUnitFailed) firing: (2) update-ubuntu-mirror.service on mirror1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [09:50:07] (PuppetConstantChange) firing: (2) Puppet performing a change on every puppet run on testvm2005:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange [10:32:16] (SystemdUnitFailed) firing: generate_os_reports.service on puppetdb2003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [13:06:55] 10SRE-tools, 10Cloud-VPS, 10Spicerack: spicerack.puppet.PuppetHostsError: Unable to find CSR fingerprints for all hosts, detected errors are: Another puppet instance is already running and the waitforlock setting is set to 0; exiting - https://phabricator.wikimedia.org/T361218 (10taavi) 03NEW [13:19:23] 10netops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: Decom asw-a-codfw switch stack - https://phabricator.wikimedia.org/T358244#9669160 (10Papaul) [13:21:02] 10netops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: 14Decom asw-a-codfw switch stack - 14https://phabricator.wikimedia.org/T358244#9669161 (10Papaul) 05Open→03Resolved a:03Papaul 14complete  [13:50:08] (PuppetConstantChange) firing: (2) Puppet performing a change on every puppet run on testvm2005:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange [14:32:16] (SystemdUnitFailed) firing: generate_os_reports.service on puppetdb2003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [15:12:30] 10SRE-tools, 06Infrastructure-Foundations, 10Spicerack, 13Patch-For-Review: gNMI module in Spicerack - https://phabricator.wikimedia.org/T344325#9669742 (10ayounsi) [16:19:18] 10netops, 06Infrastructure-Foundations: Replace Rancid with Oxidized - https://phabricator.wikimedia.org/T361252 (10ayounsi) 03NEW p:05Triage→03Low [16:30:37] hi folks, I think (though I'm not sure, I lost the scrollback) that my latest puppet-merge failed to sync on some hosts, is there a procedure to re-sync or easier/simpler to either wait or send another review and merge that ? [16:31:21] the patch being https://gerrit.wikimedia.org/r/c/operations/puppet/+/1015326 though I'm still getting the same error on alert2001 as if the change wasn't merged [16:46:02] godog: taking a look [16:46:59] cdanis: thank you, in the meantime I did verify that indeed the change isn't fully propagated AFAICS [16:47:02] cumin 'puppetmaster* or puppetserver*' 'ls -la /srv/puppet_code/environments/production/modules/icing [16:47:05] a/manifests/naggen.pp /var/lib/git/operations/puppet/modules/icinga/manifests/naggen.pp 2>/dev/null || true' [16:47:08] sigh ok you get the idea [16:56:58] godog: https://phabricator.wikimedia.org/P59002 I think is fixed now [16:57:18] that's just the inner sync body of puppet-merge [16:57:49] I think submitting a dummy patch would have worked as well [16:58:17] undecided if that's a better idea than what I did, or, if that should be the 'usual' solution instead of adding a flag to the script [16:58:50] thank you cdanis ! yeah can confirm [16:59:14] indeed I don't know either tbh, I didn't want to send a dummy patch though that's probably the easiest option [16:59:54] I also thought we had puppet-merge logs somewhere central, but maybe not [17:03:35] very very anticlimactic, I actually need to revert that patch [17:04:32] ahaha [17:04:52] yeah sounds about right [17:05:00] let me know if you can reproduce the trouble! [17:05:14] haha! thank you, I will [17:50:08] (PuppetConstantChange) firing: (2) Puppet performing a change on every puppet run on testvm2005:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange [18:32:16] (SystemdUnitFailed) firing: generate_os_reports.service on puppetdb2003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [19:06:59] 10Mail, 06Infrastructure-Foundations, 06SRE: Access to DMARCIAN - https://phabricator.wikimedia.org/T356920#9670979 (10Aklapper) T330944 is a task which requires being a member of #WMF-NDA on Phab. @DBu-WMF: I've made you a member now (after verifying your account via https://meta.wikimedia.org/wiki/Special:... [20:29:51] 10Mail, 06Infrastructure-Foundations, 06SRE: Access to DMARCIAN - https://phabricator.wikimedia.org/T356920#9671363 (10Dzahn) fwiw, I don't see how it's related to S4 [20:38:53] 10Mail, 06Infrastructure-Foundations, 06SRE: Access to DMARCIAN - https://phabricator.wikimedia.org/T356920#9671428 (10Aklapper) @dzahn: The Space is displayed as a prefix of the task title, separated by a pipeline character. [21:50:08] (PuppetConstantChange) firing: (2) Puppet performing a change on every puppet run on testvm2005:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange [22:32:16] (SystemdUnitFailed) firing: generate_os_reports.service on puppetdb2003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed