[00:28:37] (SystemdUnitFailed) firing: (5) wmf_auto_restart_prometheus-mysqld-exporter@s2.service on db1170:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [01:28:55] (SystemdUnitFailed) firing: (6) wmf_auto_restart_prometheus-mysqld-exporter@s2.service on db1170:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [03:08:38] (SystemdUnitFailed) firing: (7) wmf_auto_restart_prometheus-mysqld-exporter@s2.service on db1170:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [04:13:55] (SystemdUnitFailed) firing: (8) wmf_auto_restart_prometheus-mysqld-exporter@s2.service on db1170:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [05:38:38] (SystemdUnitFailed) firing: (8) wmf_auto_restart_prometheus-mysqld-exporter@s2.service on db1170:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [05:38:55] (SystemdUnitFailed) firing: (8) wmf_auto_restart_prometheus-mysqld-exporter@s2.service on db1170:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [08:41:51] :sadpanda: [09:23:55] Most of those should be gone by now [13:59:15] I have upgraded clouddb1020 to 10.6, all went fine, this is not the first clouddb* we upgraded, but I am going to give it 24h before upgrade the next one [13:59:29] Upgrading clouddb hosts blocks the upgrade to any section to 10.6 of course [13:59:36] In eqiad at least [14:54:07] PROBLEM - MariaDB sustained replica lag on s6 on db1231 is CRITICAL: 20.6 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1231&var-port=9104 [14:55:07] RECOVERY - MariaDB sustained replica lag on s6 on db1231 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1231&var-port=9104 [15:06:40] PROBLEM - MariaDB sustained replica lag on s6 on db2114 is CRITICAL: 73.4 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2114&var-port=9104 [15:08:40] RECOVERY - MariaDB sustained replica lag on s6 on db2114 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2114&var-port=9104 [15:14:08] PROBLEM - MariaDB sustained replica lag on s1 on db2146 is CRITICAL: 75.5 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2146&var-port=9104 [15:16:08] RECOVERY - MariaDB sustained replica lag on s1 on db2146 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2146&var-port=9104 [15:17:12] PROBLEM - MariaDB sustained replica lag on s6 on db2151 is CRITICAL: 199.5 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2151&var-port=9104 [15:20:12] RECOVERY - MariaDB sustained replica lag on s6 on db2151 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2151&var-port=9104 [15:22:52] Anyone up to an aptrepo CR review, please? https://gerrit.wikimedia.org/r/c/operations/puppet/+/1005110 Adds Ceph upstream's reef packages [15:23:26] done! [15:24:20] Thanks :) [15:46:30] PROBLEM - MariaDB sustained replica lag on s8 on db1226 is CRITICAL: 41 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1226&var-port=9104 [15:46:34] PROBLEM - MariaDB sustained replica lag on s2 on db1233 is CRITICAL: 64.4 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1233&var-port=9104 [15:47:28] PROBLEM - MariaDB sustained replica lag on s6 on db1168 is CRITICAL: 16.8 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1168&var-port=9104 [15:48:28] RECOVERY - MariaDB sustained replica lag on s6 on db1168 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1168&var-port=9104 [15:48:30] RECOVERY - MariaDB sustained replica lag on s8 on db1226 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1226&var-port=9104 [15:49:34] RECOVERY - MariaDB sustained replica lag on s2 on db1233 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1233&var-port=9104 [16:39:06] PROBLEM - MariaDB sustained replica lag on s7 on db2122 is CRITICAL: 4.4 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2122&var-port=9104 [16:39:46] PROBLEM - MariaDB sustained replica lag on s7 on db1227 is CRITICAL: 3 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1227&var-port=9104 [16:40:46] RECOVERY - MariaDB sustained replica lag on s7 on db1227 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1227&var-port=9104 [16:42:08] RECOVERY - MariaDB sustained replica lag on s7 on db2122 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2122&var-port=9104 [22:29:19] PROBLEM - MariaDB sustained replica lag on s4 on db1249 is CRITICAL: 2.8 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1249&var-port=9104 [22:30:19] PROBLEM - MariaDB sustained replica lag on s4 on db1238 is CRITICAL: 22.4 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1238&var-port=9104 [22:30:21] RECOVERY - MariaDB sustained replica lag on s4 on db1249 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1249&var-port=9104 [22:31:21] PROBLEM - MariaDB sustained replica lag on s4 on db1221 is CRITICAL: 7.2 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1221&var-port=9104 [22:31:23] PROBLEM - MariaDB sustained replica lag on s4 on db1248 is CRITICAL: 3 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1248&var-port=9104 [22:32:21] RECOVERY - MariaDB sustained replica lag on s4 on db1221 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1221&var-port=9104 [22:35:19] RECOVERY - MariaDB sustained replica lag on s4 on db1238 is OK: (C)2 ge (W)1 ge 0.6 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1238&var-port=9104 [22:35:23] RECOVERY - MariaDB sustained replica lag on s4 on db1248 is OK: (C)2 ge (W)1 ge 0.6 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1248&var-port=9104 [22:44:21] PROBLEM - MariaDB sustained replica lag on s4 on db1238 is CRITICAL: 2.6 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1238&var-port=9104 [22:45:21] RECOVERY - MariaDB sustained replica lag on s4 on db1238 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1238&var-port=9104