[10:17:06] <jinxer-wm>	 FIRING: SystemdUnitFailed: swift_ring_manager.service on ms-fe2009:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[10:51:57] <Amir1>	 ms-fe2009 is not happy
[10:57:03] <hnowlan>	 everything swift in codfw was unhappy for a bit just now
[11:12:06] <jinxer-wm>	 RESOLVED: SystemdUnitFailed: swift_ring_manager.service on ms-fe2009:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[11:33:15] <Amir1>	 hnowlan: thank you for taking care of it
[15:15:00] <Amir1>	 > Last_IO_Error: Got fatal error 1236 from master when reading data from binary log: 'Error: connecting slave requested to start from GTID 171970606-171970606-33444,
[15:15:07] <Amir1>	 s3 wikireplicas broken 
[15:15:32] <arnaudb>	 oof
[15:15:46] <dhinus>	 :(
[15:15:49] <arnaudb>	 checkin
[15:15:55] <Amir1>	 arnaudb: I'm on it
[15:16:14] <dhinus>	 thanks Amir1 
[15:16:27] <Amir1>	 I think the schema change broke it
[15:16:46] <Amir1>	 very likely it ran the schema change with replication and that wasn't needed or something like that
[15:17:17] <arnaudb>	 so in that case the course of action would be to resync the replica from a "proper" source?
[15:17:38] <Amir1>	 I need to check
[15:17:55] <Amir1>	 I don't think we can resync the replica from a proper source