[05:55:50] hello, fyi for next's week maintenance, there is only db2202 that we can't depool ourself and need your team's attention - https://phabricator.wikimedia.org/T429861 [05:56:22] XioNoX: that host shouldn't require anything [05:56:35] XioNoX: you can simply downtime and then proceed [05:56:58] cool! just curious, why? [05:57:04] because it is a testing host [05:57:15] and in fact, I am going to create a task for it to decide what to do for it [05:57:19] so thanks for bringing it up [05:57:26] sounds good! [05:57:29] thanks! [10:39:40] FIRING: SystemdUnitFailed: wmf_auto_restart_prometheus-mysqld-exporter@m2.service on db1217:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [10:40:05] fixing that ^ it is due to some maintenance I am doing [10:44:40] RESOLVED: SystemdUnitFailed: wmf_auto_restart_prometheus-mysqld-exporter@m2.service on db1217:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [12:40:53] FYI I can't repool db2155 after my maintenance, but I think it's because of another issue [12:40:59] [1/15, retrying in 3.00s] Attempt to run 'spicerack.icinga.IcingaHosts.wait_for_optimal..check' raised: Not all services are recovered: db2155:Dell PowerEdge or Supermicro Broadcom RAID Controller,MariaDB Replica Lag: s4 #page,MariaDB sustained replica lag on s4 [12:41:07] XioNoX: checking [12:41:17] db2156 repool is in progress [12:42:24] XioNoX: I will take care of db2155, you can forget about if you want. It should be repoolable soon [12:42:37] great, thx! [12:42:42] all done with A6 then [12:42:57] XioNoX: merci [12:53:01] PROBLEM - MariaDB sustained replica lag on s7 on db1194 is CRITICAL: 28.8 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1194&var-port=9104 [12:53:03] PROBLEM - MariaDB sustained replica lag on s7 on db1170 is CRITICAL: 123 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1170&var-port=9104 [12:53:07] PROBLEM - MariaDB sustained replica lag on s7 on db1181 is CRITICAL: 30.8 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1181&var-port=9104 [12:53:07] PROBLEM - MariaDB sustained replica lag on s7 on db2220 is CRITICAL: 196 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2220&var-port=9104 [12:55:01] RECOVERY - MariaDB sustained replica lag on s7 on db1194 is OK: (C)10 ge (W)5 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1194&var-port=9104 [12:55:05] RECOVERY - MariaDB sustained replica lag on s7 on db1181 is OK: (C)10 ge (W)5 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1181&var-port=9104 [12:57:09] RECOVERY - MariaDB sustained replica lag on s7 on db1170 is OK: (C)10 ge (W)5 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1170&var-port=9104 [12:58:47] PROBLEM - MariaDB sustained replica lag on s7 on db2182 is CRITICAL: 148.4 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2182&var-port=9104 [12:59:11] RECOVERY - MariaDB sustained replica lag on s7 on db2220 is OK: (C)10 ge (W)5 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2220&var-port=9104 [12:59:11] PROBLEM - MariaDB sustained replica lag on s7 on db2208 is CRITICAL: 11.2 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2208&var-port=9104 [12:59:41] PROBLEM - MariaDB sustained replica lag on s7 on db2168 is CRITICAL: 92 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2168&var-port=9104 [12:59:47] PROBLEM - MariaDB sustained replica lag on s7 on db2218 is CRITICAL: 217.6 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2218&var-port=9104 [12:59:47] PROBLEM - MariaDB sustained replica lag on s7 on db2221 is CRITICAL: 12 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2221&var-port=9104 [12:59:48] FIRING: [3x] MysqlReplicationLagPtHeartbeat: MySQL instance db2159:9104 has too large replication lag (11m 34s) - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLagPtHeartbeat [13:00:11] RECOVERY - MariaDB sustained replica lag on s7 on db2208 is OK: (C)10 ge (W)5 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2208&var-port=9104 [13:00:11] PROBLEM - MariaDB sustained replica lag on s7 on db2222 is CRITICAL: 13.6 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2222&var-port=9104 [13:00:11] PROBLEM - MariaDB sustained replica lag on s7 on db2159 is CRITICAL: 202 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2159&var-port=9104 [13:00:41] RECOVERY - MariaDB sustained replica lag on s7 on db2168 is OK: (C)10 ge (W)5 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2168&var-port=9104 [13:00:47] RECOVERY - MariaDB sustained replica lag on s7 on db2221 is OK: (C)10 ge (W)5 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2221&var-port=9104 [13:01:11] RECOVERY - MariaDB sustained replica lag on s7 on db2222 is OK: (C)10 ge (W)5 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2222&var-port=9104 [13:02:52] what was the s7 thing? is it related to maintenance? [13:03:11] yep [13:03:27] ah, sustained [13:03:31] nevermind [13:04:47] RECOVERY - MariaDB sustained replica lag on s7 on db2218 is OK: (C)10 ge (W)5 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2218&var-port=9104 [13:04:48] RESOLVED: [3x] MysqlReplicationLagPtHeartbeat: MySQL instance db2159:9104 has too large replication lag (6m 28s) - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLagPtHeartbeat [13:05:11] RECOVERY - MariaDB sustained replica lag on s7 on db2159 is OK: (C)10 ge (W)5 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2159&var-port=9104 [13:05:47] RECOVERY - MariaDB sustained replica lag on s7 on db2182 is OK: (C)10 ge (W)5 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2182&var-port=9104 [14:28:07] federico3: to confirm, the add instance on zarcillo UI won't add it to servers, right? [15:30:43] marostegui: now it does [15:31:30] (but please check if it worked)