[06:13:06] XioNoX: looks like RIPE are about to assign us a /24 on first LIR waiting list :) [06:13:31] yeah saw the email, was wondering if you woke up at 3am to ask for more IPs [07:03:38] (SystemdUnitFailed) firing: nginx.service Failed on install3003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [07:04:29] (SystemdUnitFailed) resolved: nginx.service Failed on install3003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [09:13:38] (SystemdUnitFailed) firing: netbox_ganeti_esams_sync.service Failed on netbox1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [09:35:28] 10CAS-SSO, 10Data-Platform-SRE, 10Infrastructure-Foundations, 10Patch-For-Review: Switch DataHub authentication to OIDC - https://phabricator.wikimedia.org/T305874 (10Stevemunene) These are the values added for our initial idp test and a brief explanation on each. ` - name: AUTH_OIDC_ENABLED val... [09:59:41] 10CAS-SSO, 10Data-Platform-SRE, 10Infrastructure-Foundations, 10Patch-For-Review: Switch DataHub authentication to OIDC - https://phabricator.wikimedia.org/T305874 (10BTullis) I have a feeling that for `AUTH_OIDC_USER_NAME_CLAIM` we may want to use `cn`. [10:19:17] 10CAS-SSO, 10Data-Platform-SRE, 10Infrastructure-Foundations, 10Patch-For-Review: Switch DataHub authentication to OIDC - https://phabricator.wikimedia.org/T305874 (10jbond) Just adding a bit more context >>! In T305874#9098561, @BTullis wrote: > I have a feeling that for `AUTH_OIDC_USER_NAME_CLAIM` we ma... [10:32:30] 10CAS-SSO, 10Data-Platform-SRE, 10Infrastructure-Foundations, 10Patch-For-Review: Switch DataHub authentication to OIDC - https://phabricator.wikimedia.org/T305874 (10BTullis) That's really helpful. Thanks @jbond > preferred_username maps to uid Great, so we can keep with the default of `preferred_usernam... [11:23:47] 10CAS-SSO, 10Data-Platform-SRE, 10Infrastructure-Foundations, 10Patch-For-Review: Switch DataHub authentication to OIDC - https://phabricator.wikimedia.org/T305874 (10jbond) >>! In T305874#9098663, @BTullis wrote: > Great, so we can keep with the default of `preferred_username` and this will map to the `ui... [12:28:38] (SystemdUnitFailed) firing: (2) apache2.service Failed on config-master1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [12:29:30] (SystemdUnitFailed) firing: (2) apache2.service Failed on config-master1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [12:46:10] 10CAS-SSO, 10Data-Platform-SRE, 10Infrastructure-Foundations, 10Patch-For-Review: Switch DataHub authentication to OIDC - https://phabricator.wikimedia.org/T305874 (10jbond) > Initiate a connection to staging datahub front-end via ssh -N -L 30443:k8s-ingress-staging.svc.eqiad.wmnet:30443 deploy1002.eqiad.w... [13:24:11] 10Packaging, 10Infrastructure-Foundations, 10MW-on-K8s, 10Scap: scap backport fails to build a image for k8s deployment - https://phabricator.wikimedia.org/T344438 (10Clement_Goubert) [13:26:42] 10Packaging, 10Infrastructure-Foundations, 10MW-on-K8s, 10Scap: scap backport fails to build a image for k8s deployment - https://phabricator.wikimedia.org/T344438 (10Urbanecm_WMF) >>! In T344438#9099184, @Clement_Goubert wrote: > Tagging #infrastructure-foundations #packaging because that looks like it ma... [13:38:36] 10Packaging, 10Infrastructure-Foundations, 10MW-on-K8s, 10Scap: scap backport fails to build a image for k8s deployment - https://phabricator.wikimedia.org/T344438 (10Clement_Goubert) 05Open→03In progress p:05High→03Unbreak! Raising to UBN, this breaks all mw-on-k8s deployments. [13:40:41] 10Packaging, 10Infrastructure-Foundations, 10MW-on-K8s, 10Scap: scap backport fails to build a image for k8s deployment - https://phabricator.wikimedia.org/T344438 (10Clement_Goubert) @Joe is investigating [14:02:18] 10CAS-SSO, 10Data-Platform-SRE, 10Infrastructure-Foundations, 10Patch-For-Review: Switch DataHub authentication to OIDC - https://phabricator.wikimedia.org/T305874 (10BTullis) >>! In T305874#9099070, @jbond wrote: > At this point you are already hijacking all 443 traffic so it might be easier to use `CAP_N... [15:31:34] 10Packaging, 10Infrastructure-Foundations, 10MW-on-K8s, 10Scap: scap backport fails to build a image for k8s deployment - https://phabricator.wikimedia.org/T344438 (10Clement_Goubert) 05In progress→03Resolved a:03Clement_Goubert We encountered another issue while fixing, which delayed resolution a bi... [16:33:38] (SystemdUnitFailed) firing: netbox_ganeti_esams_sync.service Failed on netbox1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [17:43:38] (SystemdUnitFailed) firing: (2) netbox_ganeti_esams_sync.service Failed on netbox1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [17:48:38] (SystemdUnitFailed) firing: (2) netbox_ganeti_esams_sync.service Failed on netbox1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [18:28:38] (SystemdUnitFailed) firing: (2) check_netbox_uncommitted_dns_changes.service Failed on netbox1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [18:33:38] (SystemdUnitFailed) firing: (2) check_netbox_uncommitted_dns_changes.service Failed on netbox1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [22:33:39] (SystemdUnitFailed) firing: netbox_ganeti_esams_sync.service Failed on netbox1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed