[02:00:11] (SystemdUnitFailed) firing: upload_puppet_facts.service Failed on puppetmaster1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [02:36:46] (NTPNoSynced) firing: NTP not synced - https://wikitech.wikimedia.org/wiki/NTP - TODO - https://alerts.monitoring.wmflabs.org/?q=alertname%3DNTPNoSynced [06:00:12] (SystemdUnitFailed) firing: upload_puppet_facts.service Failed on puppetmaster1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [06:36:46] (NTPNoSynced) firing: NTP not synced - https://wikitech.wikimedia.org/wiki/NTP - TODO - https://alerts.monitoring.wmflabs.org/?q=alertname%3DNTPNoSynced [08:48:21] 10SRE-tools, 10DBA, 10Infrastructure-Foundations, 10Puppet-Core, and 3 others: puppet7 on cumin breaks database connections - https://phabricator.wikimedia.org/T352974 (10ABran-WMF) {F41573747} testing `db-mysql` commands directly in context with the 2 CA reproduces this issue, it is possible that there is... [09:40:44] 10SRE-tools, 10DBA, 10Infrastructure-Foundations, 10Puppet-Core, and 3 others: puppet7 on cumin breaks database connections - https://phabricator.wikimedia.org/T352974 (10ABran-WMF) one other interesting fact: a puppet 7 host >>! In T352974#9389926, @Marostegui wrote: > db1124 can be used for testing. It... [10:00:12] (SystemdUnitFailed) firing: upload_puppet_facts.service Failed on puppetmaster1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [10:36:46] (NTPNoSynced) firing: NTP not synced - https://wikitech.wikimedia.org/wiki/NTP - TODO - https://alerts.monitoring.wmflabs.org/?q=alertname%3DNTPNoSynced [10:37:10] 10SRE-tools, 10DBA, 10Infrastructure-Foundations, 10Puppet-Core, and 3 others: puppet7 on cumin breaks database connections - https://phabricator.wikimedia.org/T352974 (10ABran-WMF) it appears that most of our hosts are still using `/etc/ssl/certs/Puppet_Internal_CA.pem` and should be migrated to use `/etc... [11:00:12] (SystemdUnitFailed) resolved: upload_puppet_facts.service Failed on puppetmaster1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [11:10:09] moritzm: When/if you have the time, can you make a gerrit repo for the debmonitor-client? [11:11:16] on it [11:12:06] Thanks [11:12:25] operations/software/debs/debmonitor-client (like the other debs) or operations/software/debmonitor-client (like current repo)? [11:14:06] I think like the current repo, it's the code + the debian directory in one [11:28:14] ok [11:29:23] slyngs: created: https://gerrit.wikimedia.org/r/admin/repos/operations/software/debmonitor-client [13:42:36] 10SRE-tools, 10Infrastructure-Foundations, 10Patch-For-Review: Automation to change a server's vlan - https://phabricator.wikimedia.org/T350152 (10ayounsi) >>! In T350152#9355720, @Volans wrote: > * I would probably add a grep for the IP on at least `/etc` on the host too to check if it's hardcoded somewhere... [14:36:46] (NTPNoSynced) firing: NTP not synced - https://wikitech.wikimedia.org/wiki/NTP - TODO - https://alerts.monitoring.wmflabs.org/?q=alertname%3DNTPNoSynced [14:37:39] is that a WMCS alert? ^ the line also points to a 503 error /cc godog [14:39:19] ah pontoon [14:43:03] https://phabricator.wikimedia.org/T353060 [15:43:14] 10SRE-tools, 10DBA, 10Infrastructure-Foundations, 10Puppet-Core, and 2 others: puppet7 on cumin breaks database connections - https://phabricator.wikimedia.org/T352974 (10LSobanski) I believe the collab tag was added automatically from the parent task so removing it.