[04:04:36] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: Install new MPC10E-10C line cards on cr1-eqiad and cr2-eqiad slot 0. - https://phabricator.wikimedia.org/T426343#12016920 (10Papaul) @cmooney I took a look at the steps all look good to me for the RE. However I didn't see the steps to... [07:54:57] 10netops, 06Infrastructure-Foundations, 10Toolforge, 06tools-infrastructure-team: Plan networking for Toolforge-on-Metal experiment - https://phabricator.wikimedia.org/T407140#12017322 (10fgiunchedi) 05Stalled→03Declined I'm boldly declining the task, we can reopen as/if needed [07:56:24] 10netops, 06cloud-services-team, 06Infrastructure-Foundations, 10Toolforge: Create new VRF and networks for Toolforge-on-Metal - https://phabricator.wikimedia.org/T409309#12017328 (10fgiunchedi) 05Open→03Declined I'm boldly declining the task, we can reopen as/if needed [08:08:36] 10SRE-tools, 06Infrastructure-Foundations, 06SRE: Terminal configuration for cookbooks - https://phabricator.wikimedia.org/T429129 (10MoritzMuehlenhoff) 03NEW [10:04:39] FIRING: SystemdUnitFailed: stunnel4.service on cumin1003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [10:27:34] RESOLVED: SystemdUnitFailed: stunnel4.service on cumin1003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [11:14:02] dunno who would be best suited to review https://gerrit.wikimedia.org/r/c/operations/software/homer/deploy/+/1302127 [12:42:34] FIRING: SystemdUnitFailed: requestctl-credential-refresh.service on puppetserver2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [12:42:35] 10SRE-tools, 06Infrastructure-Foundations, 06SRE: Terminal configuration for cookbooks - https://phabricator.wikimedia.org/T429129#12018455 (10Volans) It seems to be a mix of things, from one side sudo is executing without bash that sets TERM, with: ` sudo cumin 'A:puppetserver' 'echo $TERM' ` you get `dumb`... [13:44:39] FIRING: [2x] SystemdUnitFailed: requestctl-credential-refresh.service on puppetserver2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [13:52:34] FIRING: [3x] SystemdUnitFailed: requestctl-credential-refresh.service on puppetserver2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [13:57:34] FIRING: [4x] SystemdUnitFailed: update-ubuntu-mirror.service on mirror1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [14:12:57] 10homer, 10netops, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: Homer should abort on filter rules applied on non-existent or disabled interfaces - https://phabricator.wikimedia.org/T428886#12019220 (10taavi) I'm not a huge fan of relying on the exact formatting of the interface description,... [14:37:34] FIRING: [4x] SystemdUnitFailed: update-ubuntu-mirror.service on mirror1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [14:42:34] FIRING: [4x] SystemdUnitFailed: update-ubuntu-mirror.service on mirror1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [14:49:39] FIRING: [4x] SystemdUnitFailed: update-ubuntu-mirror.service on mirror1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [17:58:33] 10netops, 06Infrastructure-Foundations, 06SRE: Network device tls certs: alerting niggles - https://phabricator.wikimedia.org/T429242 (10cmooney) 03NEW p:05Triage→03Medium [18:52:34] FIRING: SystemdUnitFailed: update-ubuntu-mirror.service on mirror1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [20:50:02] 10SRE-tools, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: Terminal configuration for cookbooks - https://phabricator.wikimedia.org/T429129#12021336 (10jhathaway) Since we don't have a PTY, I would disable colors. Patch sent! [22:52:34] FIRING: SystemdUnitFailed: update-ubuntu-mirror.service on mirror1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed