[00:07:04] <icinga-wm>	 PROBLEM - MariaDB sustained replica lag on s1 on db2216 is CRITICAL: 2.4 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2216&var-port=9104
[00:07:04] <icinga-wm>	 PROBLEM - MariaDB sustained replica lag on s1 on db2188 is CRITICAL: 2.2 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2188&var-port=9104
[00:08:04] <icinga-wm>	 RECOVERY - MariaDB sustained replica lag on s1 on db2216 is OK: (C)2 ge (W)1 ge 0.2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2216&var-port=9104
[00:08:04] <icinga-wm>	 RECOVERY - MariaDB sustained replica lag on s1 on db2188 is OK: (C)2 ge (W)1 ge 0.4 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2188&var-port=9104
[05:03:08] <marostegui>	 I am going to disable writes on es5 
[07:26:07] <marostegui>	 I am switching s7 codfw
[07:26:27] <marostegui>	 mmm not yet
[09:02:18] <marostegui>	 Starting s7 codfw switchover
[09:49:05] <Amir1>	 if you see anything with m5: https://phabricator.wikimedia.org/T303930#9752341
[09:49:17] <marostegui>	 thanks
[09:54:23] <jynus>	 are s7 and s4 soon to be completed?
[09:54:41] <marostegui>	 s7 yes
[09:54:45] <marostegui>	 codfw is done actually
[09:54:58] <marostegui>	 and I will finish eqiad probably tomorrow or wed
[09:55:05] <jynus>	 thanks, then switching backups this week
[09:55:10] <marostegui>	 great thank you
[11:01:27] <jinxer-wm>	 (SystemdUnitFailed) firing: prometheus-mysqld-exporter.service on es1038:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[11:01:36] <marostegui>	 that's expected
[13:03:53] <marostegui>	 arnaudb: did you just switch s4 codfw?
[13:04:02] <arnaudb>	 I did
[13:04:09] <marostegui>	 the old master is still serving api
[13:04:11] <marostegui>	 please remove it from there
[13:04:13] <arnaudb>	 oh
[14:52:30] <arnaudb>	 performing s6 codfw switchover T363713
[14:52:30] <stashbot>	 T363713: Switchover s6 master (db2114 -> db2129) - https://phabricator.wikimedia.org/T363713
[15:07:41] <arnaudb>	 hm I've had an issue at runtime with the cookbook
[15:08:10] <arnaudb>	 for future ref https://www.irccloud.com/pastebin/4PG3Chpn/
[15:11:31] <arnaudb>	 Amir1: marostegui I don't want to break something by taking any uninformed action
[15:11:48] <arnaudb>	 given the situation, and the topology that orchestrator sees
[15:11:58] <arnaudb>	 I'm not sure what's the right move here
[15:17:16] <arnaudb>	 I'll start replication again to avoid taking too much lag
[15:21:28] <jynus>	 reenable gtid too, according to that log if disabled
[15:28:43] <arnaudb>	 I've depooled db2151 who's still lagging. GTID reenabled on db2151 and db2129
[15:30:24] <arnaudb>	 node is back on track, will left things as they are until marostegui and/or Amir1 are back here, I think the situation is stabilized 
[15:48:00] <arnaudb>	 will retry the script upon advice
[15:55:34] <marostegui>	 arnaudb: if the host is back in sync you can probably repool and retry. It maybe just timedout
[15:55:38] <marostegui>	 Sometimes it happens 
[16:02:28] <arnaudb>	 no idea what went wrong as I had no more information than the provided log haha but the switchover has been resumed thanks to jynus' and Amir1's help! - T363713
[16:02:29] <stashbot>	 T363713: Switchover s6 master (db2114 -> db2129) - https://phabricator.wikimedia.org/T363713
[16:49:45] <urandom>	 Emperor: T358830 is amusing
[16:49:46] <stashbot>	 T358830: Uploads fail due to 401 error from swift on wednesdays - https://phabricator.wikimedia.org/T358830
[17:12:48] <Emperor>	 good, isn't it? :)
[18:17:44] <andrewbogott>	 milimetric: Are you able to see https://phabricator.wikimedia.org/T363633?  And if so, can you plan to take a look sometime soon?
[20:24:41] <milimetric>	 done @andrewbogott, left a comment
[20:24:52] <andrewbogott>	 thanks!