[06:06:07] Amir1: I will reboot db1123 for kernel upgrade first [06:06:54] ping me once you're done. I go back to sleep :D [06:06:59] XDDDD [06:58:46] PROBLEM - MariaDB sustained replica lag on x2 on db2142 is CRITICAL: 4.2 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2142&var-port=9104 [07:03:07] Amir1: db1123 is all yours [07:30:34] RECOVERY - MariaDB sustained replica lag on x2 on db2142 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2142&var-port=9104 [07:42:32] PROBLEM - MariaDB sustained replica lag on x2 on db2142 is CRITICAL: 19.8 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2142&var-port=9104 [07:43:26] PROBLEM - MariaDB sustained replica lag on x2 on db2143 is CRITICAL: 2.8 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2143&var-port=9104 [08:16:42] PROBLEM - MariaDB sustained replica lag on x2 on db2144 is CRITICAL: 35 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2144&var-port=9104 [08:19:10] RECOVERY - MariaDB sustained replica lag on x2 on db2144 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2144&var-port=9104 [08:31:48] PROBLEM - MariaDB sustained replica lag on x2 on db2144 is CRITICAL: 43 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2144&var-port=9104 [08:42:36] RECOVERY - MariaDB sustained replica lag on x2 on db2143 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2143&var-port=9104 [08:55:22] PROBLEM - MariaDB sustained replica lag on x2 on db2143 is CRITICAL: 16.8 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2143&var-port=9104 [09:35:38] marostegui: awesome, thanks [09:44:02] RECOVERY - MariaDB sustained replica lag on x2 on db2143 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2143&var-port=9104 [09:48:14] RECOVERY - MariaDB sustained replica lag on x2 on db2144 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2144&var-port=9104 [09:56:46] PROBLEM - MariaDB sustained replica lag on x2 on db2143 is CRITICAL: 81.2 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2143&var-port=9104 [10:00:50] PROBLEM - MariaDB sustained replica lag on x2 on db2144 is CRITICAL: 86.8 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2144&var-port=9104 [10:05:03] RECOVERY - MariaDB sustained replica lag on x2 on db2144 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2144&var-port=9104 [10:11:55] PROBLEM - MariaDB sustained replica lag on x2 on db2144 is CRITICAL: 132.4 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2144&var-port=9104 [10:13:57] RECOVERY - MariaDB sustained replica lag on x2 on db2143 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2143&var-port=9104 [10:23:03] PROBLEM - MariaDB sustained replica lag on x2 on db2143 is CRITICAL: 27.2 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2143&var-port=9104 [10:25:09] RECOVERY - MariaDB sustained replica lag on x2 on db2143 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2143&var-port=9104 [10:32:44] RECOVERY - MariaDB sustained replica lag on x2 on db2144 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2144&var-port=9104 [10:36:34] PROBLEM - MariaDB sustained replica lag on x2 on db2143 is CRITICAL: 53.6 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2143&var-port=9104 [10:38:42] PROBLEM - MariaDB sustained replica lag on x2 on db2144 is CRITICAL: 107.4 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2144&var-port=9104 [10:52:10] RECOVERY - MariaDB sustained replica lag on x2 on db2143 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2143&var-port=9104 [11:01:16] (PrometheusMysqldExporterFailed) firing: Prometheus-mysqld-exporter failed (db2164:9104) - https://grafana.wikimedia.org/d/000000278/mysql-aggregated - https://alerts.wikimedia.org/?q=alertname%3DPrometheusMysqldExporterFailed [11:01:35] PROBLEM - MariaDB sustained replica lag on x2 on db2143 is CRITICAL: 24.6 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2143&var-port=9104 [11:01:47] The prometheus exporter alert is me [11:05:39] RECOVERY - MariaDB sustained replica lag on x2 on db2144 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2144&var-port=9104 [11:16:33] PROBLEM - MariaDB sustained replica lag on x2 on db2144 is CRITICAL: 26 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2144&var-port=9104 [11:43:07] RECOVERY - MariaDB sustained replica lag on x2 on db2143 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2143&var-port=9104 [11:44:35] RECOVERY - MariaDB sustained replica lag on x2 on db2144 is OK: (C)2 ge (W)1 ge 0.6 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2144&var-port=9104 [11:48:29] RECOVERY - MariaDB sustained replica lag on x2 on db2142 is OK: (C)2 ge (W)1 ge 0.2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2142&var-port=9104 [11:56:31] (PrometheusMysqldExporterFailed) resolved: Prometheus-mysqld-exporter failed (db2164:9104) - https://grafana.wikimedia.org/d/000000278/mysql-aggregated - https://alerts.wikimedia.org/?q=alertname%3DPrometheusMysqldExporterFailed [12:18:53] PROBLEM - MariaDB sustained replica lag on x2 on db2142 is CRITICAL: 40 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2142&var-port=9104 [12:19:59] PROBLEM - MariaDB sustained replica lag on x2 on db2144 is CRITICAL: 27.2 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2144&var-port=9104 [12:26:05] PROBLEM - MariaDB sustained replica lag on x2 on db2143 is CRITICAL: 14.2 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2143&var-port=9104 [12:27:33] RECOVERY - MariaDB sustained replica lag on x2 on db2144 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2144&var-port=9104 [12:28:11] we really need to do something about this ^ [12:28:19] I have pinged Timo [12:28:21] in SRE [12:28:25] oh thanks [12:42:45] PROBLEM - MariaDB sustained replica lag on x2 on db2144 is CRITICAL: 27.8 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2144&var-port=9104 [12:43:49] RECOVERY - MariaDB sustained replica lag on x2 on db2143 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2143&var-port=9104 [12:45:18] Can I get a review https://gerrit.wikimedia.org/r/c/operations/puppet/+/813234/ ? [12:45:32] 👀 [12:45:36] haha thanks [12:46:59] thanks jynus (that was fast!) and Emperor! [12:47:40] Count von Count taught me well [12:56:33] PROBLEM - MariaDB sustained replica lag on x2 on db2143 is CRITICAL: 14.4 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2143&var-port=9104 [14:07:45] RECOVERY - MariaDB sustained replica lag on x2 on db2143 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2143&var-port=9104 [14:14:59] Notifications disabled for lag for x2 codfw hosts [15:52:18] PROBLEM - pt-heartbeat-wikimedia service on db2104 is CRITICAL: CRITICAL - Expecting active but unit pt-heartbeat-wikimedia is inactive https://wikitech.wikimedia.org/wiki/MariaDB/pt-heartbeat [15:53:04] PROBLEM - pt-heartbeat-wikimedia service on db2079 is CRITICAL: CRITICAL - Expecting active but unit pt-heartbeat-wikimedia is inactive https://wikitech.wikimedia.org/wiki/MariaDB/pt-heartbeat [15:53:40] PROBLEM - pt-heartbeat-wikimedia service on db2121 is CRITICAL: CRITICAL - Expecting active but unit pt-heartbeat-wikimedia is inactive https://wikitech.wikimedia.org/wiki/MariaDB/pt-heartbeat [15:57:12] PROBLEM - pt-heartbeat-wikimedia service on pc2011 is CRITICAL: CRITICAL - Expecting active but unit pt-heartbeat-wikimedia is inactive https://wikitech.wikimedia.org/wiki/MariaDB/pt-heartbeat [15:58:22] RECOVERY - pt-heartbeat-wikimedia service on db2121 is OK: OK - pt-heartbeat-wikimedia is active https://wikitech.wikimedia.org/wiki/MariaDB/pt-heartbeat [15:58:47] Amir1: m1 paged in all that too [15:59:00] PROBLEM - MariaDB sustained replica lag on s7 on db2120 is CRITICAL: 9.5 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2120&var-port=9104 [15:59:12] PROBLEM - MariaDB sustained replica lag on s2 on db2126 is CRITICAL: 19.67 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2126&var-port=9104 [15:59:18] RECOVERY - pt-heartbeat-wikimedia service on db2104 is OK: OK - pt-heartbeat-wikimedia is active https://wikitech.wikimedia.org/wiki/MariaDB/pt-heartbeat [15:59:39] I check once I'm done with s8 [15:59:46] PROBLEM - MariaDB sustained replica lag on s2 on db2125 is CRITICAL: 138.5 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2125&var-port=9104 [15:59:48] PROBLEM - MariaDB sustained replica lag on s7 on db2118 is CRITICAL: 82.67 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2118&var-port=9104 [16:00:08] PROBLEM - MariaDB sustained replica lag on s7 on db2108 is CRITICAL: 213.7 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2108&var-port=9104 [16:00:16] RECOVERY - pt-heartbeat-wikimedia service on db2079 is OK: OK - pt-heartbeat-wikimedia is active https://wikitech.wikimedia.org/wiki/MariaDB/pt-heartbeat [16:00:50] PROBLEM - MariaDB sustained replica lag on s7 on db2121 is CRITICAL: 124.3 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2121&var-port=9104 [16:00:50] PROBLEM - MariaDB sustained replica lag on s8 on db2152 is CRITICAL: 413 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2152&var-port=9104 [16:01:34] RECOVERY - MariaDB sustained replica lag on s7 on db2120 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2120&var-port=9104 [16:01:42] PROBLEM - MariaDB sustained replica lag on s8 on db2079 is CRITICAL: 214 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2079&var-port=9104 [16:01:46] PROBLEM - pt-heartbeat-wikimedia service on db2132 is CRITICAL: CRITICAL - Expecting active but unit pt-heartbeat-wikimedia is inactive https://wikitech.wikimedia.org/wiki/MariaDB/pt-heartbeat [16:01:48] RECOVERY - MariaDB sustained replica lag on s2 on db2126 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2126&var-port=9104 [16:01:54] PROBLEM - MariaDB sustained replica lag on s8 on db2161 is CRITICAL: 193.7 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2161&var-port=9104 [16:02:22] RECOVERY - MariaDB sustained replica lag on s2 on db2125 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2125&var-port=9104 [16:02:24] RECOVERY - MariaDB sustained replica lag on s7 on db2118 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2118&var-port=9104 [16:02:45] RECOVERY - MariaDB sustained replica lag on s7 on db2108 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2108&var-port=9104 [16:03:28] RECOVERY - MariaDB sustained replica lag on s7 on db2121 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2121&var-port=9104 [16:04:20] PROBLEM - MariaDB sustained replica lag on s8 on db2163 is CRITICAL: 61 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2163&var-port=9104 [16:05:58] RECOVERY - MariaDB sustained replica lag on s8 on db2152 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2152&var-port=9104 [16:06:50] PROBLEM - MariaDB sustained replica lag on m1 on db2132 is CRITICAL: 863 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2132&var-port=9104 [16:06:54] RECOVERY - MariaDB sustained replica lag on s8 on db2079 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2079&var-port=9104 [16:06:54] RECOVERY - MariaDB sustained replica lag on s8 on db2163 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2163&var-port=9104 [16:06:58] RECOVERY - pt-heartbeat-wikimedia service on db2132 is OK: OK - pt-heartbeat-wikimedia is active https://wikitech.wikimedia.org/wiki/MariaDB/pt-heartbeat [16:07:04] RECOVERY - MariaDB sustained replica lag on s8 on db2161 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2161&var-port=9104 [16:07:30] RECOVERY - pt-heartbeat-wikimedia service on pc2011 is OK: OK - pt-heartbeat-wikimedia is active https://wikitech.wikimedia.org/wiki/MariaDB/pt-heartbeat [16:08:36] PROBLEM - MariaDB sustained replica lag on m1 on db2160 is CRITICAL: 7.333 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2160&var-port=13321 [16:11:54] PROBLEM - MariaDB sustained replica lag on s1 on db2085 is CRITICAL: 1147 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2085&var-port=13311 [16:11:54] RECOVERY - MariaDB sustained replica lag on m1 on db2132 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2132&var-port=9104 [16:12:06] PROBLEM - MariaDB sustained replica lag on s8 on db2085 is CRITICAL: 468.3 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2085&var-port=13318 [16:13:40] RECOVERY - MariaDB sustained replica lag on m1 on db2160 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2160&var-port=13321 [16:14:20] PROBLEM - MariaDB sustained replica lag on s1 on db2145 is CRITICAL: 1272 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2145&var-port=9104 [16:14:38] PROBLEM - MariaDB sustained replica lag on s1 on db2153 is CRITICAL: 1288 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2153&var-port=9104 [16:17:14] RECOVERY - MariaDB sustained replica lag on s8 on db2085 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2085&var-port=13318 [16:19:34] RECOVERY - MariaDB sustained replica lag on s1 on db2085 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2085&var-port=13311 [16:22:04] RECOVERY - MariaDB sustained replica lag on s1 on db2145 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2145&var-port=9104 [16:24:54] RECOVERY - MariaDB sustained replica lag on s1 on db2153 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2153&var-port=9104