[07:48:40] <jinxer-wm>	 (SystemdUnitFailed) firing: update-ubuntu-mirror.service Failed on mirror1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[11:33:00] <wikibugs>	 10SRE-tools, 10Ganeti, 10Spicerack: cookbook sre.ganeti.makevm calls wrong netbox_ganeti_codfw_sync.service - https://phabricator.wikimedia.org/T344812 (10ayounsi)
[11:34:53] <wikibugs>	 10SRE-tools, 10Ganeti, 10Infrastructure-Foundations: cookbook sre.ganeti.makevm fails when no group is set - https://phabricator.wikimedia.org/T344813 (10ayounsi)
[11:44:01] <wikibugs>	 10SRE-tools, 10Ganeti, 10Infrastructure-Foundations, 10User-MoritzMuehlenhoff: cookbook sre.ganeti.makevm fails when no group is set - https://phabricator.wikimedia.org/T344813 (10MoritzMuehlenhoff)
[11:44:19] <wikibugs>	 10SRE-tools, 10Ganeti, 10Infrastructure-Foundations, 10Spicerack, 10User-MoritzMuehlenhoff: cookbook sre.ganeti.makevm calls wrong netbox_ganeti_codfw_sync.service - https://phabricator.wikimedia.org/T344812 (10MoritzMuehlenhoff)
[11:47:16] <wikibugs>	 10CAS-SSO, 10Data-Platform-SRE, 10Infrastructure-Foundations: Switch DataHub authentication to OIDC - https://phabricator.wikimedia.org/T305874 (10Stevemunene) Added the `AUTH_OIDC_CLIENT_AUTHENTICATION_METHOD ` method and retested, the idp seems okay with everything, my user is authenticated and provided a...
[11:48:41] <jinxer-wm>	 (SystemdUnitFailed) firing: update-ubuntu-mirror.service Failed on mirror1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[12:21:21] <wikibugs>	 10CAS-SSO, 10Data-Platform-SRE, 10Infrastructure-Foundations: Switch DataHub authentication to OIDC - https://phabricator.wikimedia.org/T305874 (10BTullis) Thanks @Stevemunene - Good work. I can confirm that when I attempt to log into staging.
[13:48:41] <jinxer-wm>	 (SystemdUnitFailed) resolved: update-ubuntu-mirror.service Failed on mirror1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[14:08:41] <jinxer-wm>	 (SystemdUnitFailed) firing: check_netbox_uncommitted_dns_changes.service Failed on netbox1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[14:13:41] <jinxer-wm>	 (SystemdUnitFailed) resolved: check_netbox_uncommitted_dns_changes.service Failed on netbox1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[14:47:31] <wikibugs>	 10CAS-SSO, 10Data-Platform-SRE, 10Infrastructure-Foundations: Switch DataHub authentication to OIDC - https://phabricator.wikimedia.org/T305874 (10Stevemunene) Found some info on what we might be missing, We have so far verified that authentication on the IDP side is okay and that we do receive a signed id t...
[16:01:53] <wikibugs>	 10CAS-SSO, 10Data-Platform-SRE, 10Infrastructure-Foundations: Switch DataHub authentication to OIDC - https://phabricator.wikimedia.org/T305874 (10BTullis) In case it helps, I did a little digging into the CAS logs on idp-test1002 and stumbled upon this, which might help. ` root@idp-test1002:/var/log/cas# gr...