[12:40:20] 10SRE-tools, 06DC-Ops, 06Infrastructure-Foundations: Create cookbook to rebuild an MD RAID array upon disk replacement - https://phabricator.wikimedia.org/T364540 (10Volans) 03NEW p:05Triage→03Medium [12:59:19] o/ [12:59:40] hey there :) [13:08:36] I am moving the remaining swift nodes to PKI [13:09:40] yep,noticed it [13:09:41] great [13:09:46] I'll try to work with ServiceOps/Olly/DataPersistence to also move thanos-swift to PKI, the code is the same.. When the attempted it in the past tegola (maps) started to show an horrible perf regression, hopefully it is gone now [13:10:01] otherwise thanos-swift will block the cergen deprecation [13:11:19] perf regression due to the cert algo/hash? [13:11:42] otherwise I don't see what changed to the serivice who issued the cert [13:15:50] yep it must be (at least from reading https://phabricator.wikimedia.org/T344324#9117628). IIRC tegola doesn't use the envoy local sidecar to connect to swift [13:19:15] elukey: might be a good change tbh [13:20:05] worse case could we tell acmechief to issue the cert with the same settings they have now? [13:20:09] cdanis: I agree yes, Joe warned people in the task that the local ssl proxy may be a little tricky to tune due to the swift SDK etc.. but it would definitely make things better [13:27:17] volans: in theory yes but it would be super nice if we didn't have any weird use case :( [13:27:48] indeed [16:23:54] 10netops, 06cloud-services-team, 06Infrastructure-Foundations, 10ops-codfw: Create (or teach Andrew how to create) private connections+dns entries for new cloudcontrols - https://phabricator.wikimedia.org/T364559 (10Andrew) 03NEW [16:33:55] 10netops, 06cloud-services-team, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: Create (or teach Andrew how to create) private connections+dns entries for new cloudcontrols - https://phabricator.wikimedia.org/T364559#9783893 (10cmooney) Hey Andrew, Yeah this is on me, I'd not completed the work to ma... [17:45:04] 10Mail, 06collaboration-services, 06Infrastructure-Foundations, 10Znuny: Clean up OTRS/Znuny addresses handles by gsuite - https://phabricator.wikimedia.org/T284145#9784159 (10Quiddity) >>! In T284145#9743789, @LSobanski wrote: > 1. Let's review if the new Znuny version enabled removal of unused emails and... [17:53:11] FIRING: SystemdUnitFailed: update-ubuntu-mirror.service on mirror1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [19:24:04] 10netops, 06cloud-services-team, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: Create (or teach Andrew how to create) private connections+dns entries for new cloudcontrols - https://phabricator.wikimedia.org/T364559#9784420 (10Andrew) Reimaging cloudcontrol2006-dev works now, thanks! Bonus points: I... [21:53:11] FIRING: SystemdUnitFailed: update-ubuntu-mirror.service on mirror1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [23:50:27] RESOLVED: SystemdUnitFailed: update-ubuntu-mirror.service on mirror1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed