[02:45:25] FIRING: [2x] SystemdUnitFailed: wmf_auto_restart_prometheus-mysqld-exporter.service on es1039:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [05:57:04] FIRING: MysqlPredictiveFreeDiskSpace: Host es1039:9100 predictive low disk space on root - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting - https://grafana.wikimedia.org/goto/Jdz2PnLNg?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DMysqlPredictiveFreeDiskSpace [06:01:54] ^ expected [06:03:48] FIRING: MysqlReplicationLag: MySQL instance es1039:9104@es7 has too large replication lag (10h 0m 27s). Its replication source is es1035.eqiad.wmnet. - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica - https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&refresh=1m&var-job=All&var-server=es1039&var-port=9104 - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLag [06:04:48] FIRING: MysqlReplicationLagPtHeartbeat: MySQL instance es1039:9104 has too large replication lag (1h 54m 4s) - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica - https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&refresh=1m&var-job=All&var-server=es1039&var-port=9104 - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLagPtHeartbeat [06:08:48] RESOLVED: MysqlReplicationLag: MySQL instance es1039:9104@es7 has too large replication lag (4h 30m 42s). Its replication source is es1035.eqiad.wmnet. - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica - https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&refresh=1m&var-job=All&var-server=es1039&var-port=9104 - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLag [06:09:54] RESOLVED: MysqlReplicationLagPtHeartbeat: MySQL instance es1039:9104 has too large replication lag (1h 54m 4s) - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica - https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&refresh=1m&var-job=All&var-server=es1039&var-port=9104 - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLagPtHeartbeat [06:12:04] RESOLVED: MysqlPredictiveFreeDiskSpace: Host es1039:9100 predictive low disk space on root - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting - https://grafana.wikimedia.org/goto/Jdz2PnLNg?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DMysqlPredictiveFreeDiskSpace [06:36:13] @marostegui pc1024 is not in zarcillo it seems but it's live in pc4. I'm seeing https://gerrit.wikimedia.org/r/c/operations/puppet/+/1293842 [06:37:17] Would you mind adding it please? [06:42:59] ok, added [06:45:25] FIRING: [2x] SystemdUnitFailed: wmf_auto_restart_prometheus-mysqld-exporter.service on es1039:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [07:19:49] (the rclone failure was losing a deletion race yesterday) [07:24:22] FIRING: [2x] SystemdUnitFailed: wmf_auto_restart_prometheus-mysqld-exporter.service on es1039:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [07:44:22] RESOLVED: SystemdUnitFailed: wmf_auto_restart_prometheus-mysqld-exporter.service on es1039:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [08:50:53] marostegui: are you doing anything with pc2106? [08:51:09] @cezmunsta I am [08:51:23] it's the rolling restart for pc* and ms* [08:52:14] There doesn't seem to be anything in the server admin log, is that expected? [08:53:06] It also seemed to alert in Icinga too [08:53:54] there should be depooling logged in -operations as usual but the io alert seems gone now [08:54:30] yet I'm reviewing the status [08:55:44] I'm a bit puzzled as the script was operating on pc1016, and pc2016 is still todo [08:57:04] Why is that puzzling? [09:35:16] cezmunsta: sorry, no [09:41:15] federico3: are you on ms3 now? That seems to be firing [09:41:21] yes [09:42:03] What is adding the downtime? [09:42:17] but it's silenced or did it expire? [09:42:50] rolling_restart_pc_ms.py is doing alerting.downtime for 999 hours [09:43:23] db2252 is firing on replication [09:43:45] that's not me [09:43:48] FIRING: MysqlReplicationLagPtHeartbeat: MySQL instance db2252:9104 has too large replication lag (11m 6s) - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica - https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&refresh=1m&var-job=All&var-server=db2252&var-port=9104 - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLagPtHeartbeat [09:44:10] Why isn't it you? You said you are working on ms3 [09:45:31] marostegui: np :) [09:46:47] federico3: I only see downtime for db1153 [09:46:53] @cezmunsta you are right, the script seems stuck now, I'm chasing db2252 by hand [09:50:11] and I'll add better logging to the script later on [09:51:11] https://grafana.wikimedia.org/d/55b4cbb1-961a-42ae-92ff-28d8f6307585/mariadb-weights-and-pooling?from=now-1h&orgId=1&showCategory=Cell%20display&timezone=utc&to=now&var-section_name=ms3 is showing the silencing as well (but be aware it's not "hard real time") [09:52:18] Need any help? [09:53:51] me? [10:00:42] federico3: Not sure, if there's anything going on as I am seeing db1153 down for 30 mins [10:00:46] if that's expected then cool! [10:01:40] i'm pending the reboot of its companion 2252, should be back in a minute now [13:18:47] I think this should be good to go now https://gitlab.wikimedia.org/repos/data_persistence/dbtools/scripts/-/merge_requests/19 [14:25:44] I'm doing a quick replication switch ot test-s4 [14:27:20] federico3: OK, please confirm once done as I have been using it [14:34:18] interesting, it's not restarting replication 🤷 [14:34:39] What do you mean? They all appear to be replicating [14:35:23] the repl status looks ok but orch is showing a 4 min lag... maybe pt-hb is dead on 1176? [14:35:24] Whatever you did with db-test2001 left a bad heartbeat entry [14:35:42] It is not db1176 that you should be looking at [14:36:26] aha the truncation of the hb table on 2230 maybe [14:38:14] When you made db-test2001 the primary of codfw its heartbeat then appears on db2230 and now it is stale after the subsequent reorganisation [14:41:05] federico3: are you done now? [14:41:27] yep, i cleaned it up just now and updated https://gerrit.wikimedia.org/r/c/operations/cookbooks/+/1238368 to make it more practical [14:41:54] How did you "clean it up"? I ask as it was pristine a little earlier ;) [14:43:24] I mean clean up the content of the heartbeat table and restore the repl chain to a healthy state [14:44:33] Please can you do a paste so that I know the details of what you have done, in case it affects what I am doing?: [14:46:40] mostly I used the cb from https://gerrit.wikimedia.org/r/c/operations/cookbooks/+/1238368 [14:47:45] (ah and if you want to test the same cb while doing your tests, it should be good to go) [22:44:22] FIRING: SystemdUnitFailed: wmf_auto_restart_prometheus-mysqld-exporter.service on es1039:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed