[06:35:16] I am going to failover m3 proxy [08:40:21] s7 has now no special groups, if you notice something strange with the traffic there please ping me [09:56:25] Is there a way to split out a task from phab? https://phabricator.wikimedia.org/T162122 got updated because of a failure on ms-fe1005, but it's an unrelated issue. Is there something more elegant than "make new issue, add links to it"? [09:56:57] Emperor: you can create a subtask, or a parent task, but that's essentially creating a new task [09:57:18] OK, new task then :) [10:00:28] my current theory is that the key rollover we did for T296767 hasn't propagated to the swiftrepl config, but still digging [10:00:29] T296767: Rotate swift auth key for mw:media account - https://phabricator.wikimedia.org/T296767 [10:06:01] -rw------- 1 swiftrepl wikidev 142 May 3 2017 /srv/software/swiftrepl/swiftrepl.conf [10:06:58] # TODO(filippo) credentials for both source and destination sites are needed [10:06:58] #file { "${basedir}/swiftrepl/swiftrepl.conf": [10:07:16] (from modules/swift/manifests/swiftrepl.pp) [10:09:10] So I _think_ that swiftrepl.conf is currently manually edited/deployed [10:09:35] and the key rollover we did for mw:media is now biting [10:12:38] I think godog may still be off with Plague... [10:17:24] Yeah, the eqiad key in swiftrepl.conf is the old key [10:19:11] So I can unbreak-now by editing swiftrepl.conf (and presumably the equivalent on the first codfw frontend), but this really needs puppeting [10:23:49] ms-fe1005 now running swiftrepl OK [10:28:14] es1022 (es4) has been reimaged to Bullseye and is now starting to serve traffic, if you notice something strange on es4 traffic, let me know [10:37:21] I've made T299125 to track the need to puppetize the .conf [10:37:22] T299125: `swiftrepl.conf` should be puppet-managed - https://phabricator.wikimedia.org/T299125