[00:34:32] Amir1: marostegui: jynus: I don't suppose any of you are around [00:36:48] FIRING: [8x] MysqlReplicationLagPtHeartbeat: MySQL instance es1033:9104 has too large replication lag (11m 32s) - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLagPtHeartbeat [00:37:54] cdanis: I am [00:37:57] one sec [00:38:00] ty [00:39:42] FIRING: SystemdUnitFailed: pt-heartbeat-wikimedia.service on es1039:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [00:51:48] RESOLVED: [8x] MysqlReplicationLagPtHeartbeat: MySQL instance es1033:9104 has too large replication lag (25m 8s) - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLagPtHeartbeat [01:40:30] Didyi [01:40:41] Downtime expired ^ [10:08:37] Doing s4 codfw switchover shortly [10:27:24] ^ done [10:46:42] @cezmunsta do you have other s* codfw switchovers planned? [10:47:11] Nope [12:48:00] ok I'm switching s6 in codfw then [13:55:10] switching s7 as well if nobody is touching it @marostegui @cezmunsta [13:55:19] not me [14:03:26] nor me [14:22:25] marostegui: federico3: planning for s4 primary in the morning, as no others seem to be scheduled [14:22:46] cezmunsta: I was planning to do m3, but i will do it before yours, so feel free to go ahead [14:22:57] ok! [14:23:15] ack && OK [15:06:52] federico3: I have to go but I've seen s7 codfw master laggig behind because it still has the old entry from heartbeat, are you sure you cleaned up the heartbeat there? [15:08:12] *old master that is [15:09:08] @marostegui I see the cleanup in my logs 2109 sudo db-mysql db2159 heartbeat -e "delete from heartbeat where file like 'db2220%';" ... but maybe it created a new entry when I bounced the pt service by hand [15:09:08] [15:09:28] yes, if you did that, then yes [15:09:34] so you'd have to clean it again on that host [15:10:44] I'm rerunning it however the host itself looks healty [15:11:05] yes, it was just the heartbeat [15:11:18] RESOLVED: MysqlReplicationLagPtHeartbeat: MySQL instance db2220:9104 has too large replication lag (50m 26s) - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica - https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&refresh=1m&var-job=All&var-server=db2220&var-port=9104 - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLagPtHeartbeat [15:12:13] heh that was it, yet it's not showing in https://grafana-rw.wikimedia.org/d/000000303/mysql-replication-lag?forceLogin=true&from=now-1h&orgId=1&timezone=utc&to=now&var-section=s7&var-site=eqiad [15:12:28] actually yes it's showing [16:47:25] PROBLEM - MariaDB sustained replica lag on s2 on db2226 is CRITICAL: 183 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2226&var-port=9104 [16:47:43] PROBLEM - MariaDB sustained replica lag on s1 on db2203 is CRITICAL: 204 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2203&var-port=9104 [16:48:21] looking [16:48:25] RECOVERY - MariaDB sustained replica lag on s2 on db2226 is OK: (C)10 ge (W)5 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2226&var-port=9104 [16:50:19] this should be due to https://phabricator.wikimedia.org/T429861 [17:15:48] RECOVERY - MariaDB sustained replica lag on s1 on db2203 is OK: (C)10 ge (W)5 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2203&var-port=9104