[09:18:25] FIRING: [24x] SystemdUnitFailed: cfssl-ocsprefresh-Wikimedia_Internal_Root_CA.service on pki1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [09:28:25] RESOLVED: [24x] SystemdUnitFailed: cfssl-ocsprefresh-Wikimedia_Internal_Root_CA.service on pki1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [09:33:25] FIRING: [24x] SystemdUnitFailed: cfssl-ocsprefresh-Wikimedia_Internal_Root_CA.service on pki1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [09:47:07] 10CAS-SSO, 06Data-Platform-SRE, 06Infrastructure-Foundations: Establish LDAP access groups and login tooling for human access to the Data Lake on S3 - https://phabricator.wikimedia.org/T435473#12270233 (10LSobanski) 05Open→03Stalled p:05Triage→03Medium Setting to stalled as decisions still remain to... [09:47:17] 10CAS-SSO, 06Data-Platform-SRE, 06Infrastructure-Foundations: Design and create the LDAP groups for the Data Lake access tiers - https://phabricator.wikimedia.org/T435601#12270238 (10LSobanski) 05Open→03Stalled p:05Triage→03Medium Setting to stalled as decisions still remain to be made before this is... [09:47:28] 10CAS-SSO, 06Data-Platform-SRE, 06Infrastructure-Foundations: Enable the OAuth device authorization grant in CAS and register a wmf-s3-login client - https://phabricator.wikimedia.org/T435602#12270243 (10LSobanski) 05Open→03Stalled p:05Triage→03Medium Setting to stalled as decisions still remain to b... [10:33:25] RESOLVED: [24x] SystemdUnitFailed: cfssl-ocsprefresh-Wikimedia_Internal_Root_CA.service on pki1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [13:14:34] 10netops, 06Infrastructure-Foundations: Create a 3rd Netflow instance in eqiad/codfw - https://phabricator.wikimedia.org/T436517 (10ayounsi) 03NEW p:05Triage→03High [14:38:36] 10netbox, 06Infrastructure-Foundations: Add missing fields like CPU info to Netbox data - https://phabricator.wikimedia.org/T436028#12271299 (10LSobanski) p:05Triage→03Low @andrea.denisse could I ask you to try and write up a more specific business case for this so that we can have an idea what data makes... [15:34:28] FIRING: KeyholderUnarmed: 2 unarmed Keyholder key(s) on cumin1004:9100 - https://wikitech.wikimedia.org/wiki/Keyholder - TODO - https://alerts.wikimedia.org/?q=alertname%3DKeyholderUnarmed [15:44:28] RESOLVED: KeyholderUnarmed: 1 unarmed Keyholder key(s) on cumin1004:9100 - https://wikitech.wikimedia.org/wiki/Keyholder - TODO - https://alerts.wikimedia.org/?q=alertname%3DKeyholderUnarmed [15:55:55] 10netops, 06Infrastructure-Foundations: Consider storing the entire chain in network device's gRPC certificate - https://phabricator.wikimedia.org/T375513#12271748 (10elukey) @ayounsi IIUC in this case the gRPC certificate is not validated by prometheus and other tools? If so it should be sufficient to expose... [17:09:48] FIRING: PuppetFailure: Puppet has failed on cumin1004:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [20:19:42] 10CFSSL-PKI, 10Beta-Cluster-Infrastructure, 06Infrastructure-Foundations: profile::pki::client does not create symlinks using hashes of CAs - https://phabricator.wikimedia.org/T436562 (10Southparkfan) 03NEW [20:36:14] 10Mail, 06Infrastructure-Foundations, 06MediaWiki-Core-Platform-Team, 10MediaWiki-Email, and 9 others: Could not send confirmation email: Unknown error in PHP's mail() function. - https://phabricator.wikimedia.org/T383047#12272781 (10Tgr) Yeah, it's basically back to the pre-July 26 rate (which is about 20... [21:10:03] FIRING: PuppetFailure: Puppet has failed on cumin1004:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure