[00:05:47] (MysqlReplicationLag) firing: MySQL instance db2173:9104 has too large replication lag (10d 17h 48m 7s) - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica - https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&refresh=1m&var-job=All&var-server=db2173&var-port=9104 - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLag [01:06:07] PROBLEM - MariaDB sustained replica lag on m1 on db2160 is CRITICAL: 5.8 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2160&var-port=13321 [01:08:25] RECOVERY - MariaDB sustained replica lag on m1 on db2160 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2160&var-port=13321 [04:05:47] (MysqlReplicationLag) firing: MySQL instance db2173:9104 has too large replication lag (8d 15h 55m 55s) - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica - https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&refresh=1m&var-job=All&var-server=db2173&var-port=9104 - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLag [04:50:43] PROBLEM - MariaDB sustained replica lag on m1 on db1117 is CRITICAL: 36.2 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1117&var-port=13321 [04:51:25] PROBLEM - MariaDB sustained replica lag on m1 on db2160 is CRITICAL: 5.8 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2160&var-port=13321 [04:52:21] RECOVERY - MariaDB sustained replica lag on m1 on db1117 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1117&var-port=13321 [04:53:01] RECOVERY - MariaDB sustained replica lag on m1 on db2160 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2160&var-port=13321 [05:19:36] PROBLEM - MariaDB sustained replica lag on m1 on db1117 is CRITICAL: 32.8 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1117&var-port=13321 [05:21:16] RECOVERY - MariaDB sustained replica lag on m1 on db1117 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1117&var-port=13321 [05:30:26] PROBLEM - MariaDB sustained replica lag on m1 on db1117 is CRITICAL: 230.8 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1117&var-port=13321 [05:33:16] PROBLEM - MariaDB sustained replica lag on m1 on db2160 is CRITICAL: 152.4 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2160&var-port=13321 [05:34:24] RECOVERY - MariaDB sustained replica lag on m1 on db1117 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1117&var-port=13321 [05:35:16] RECOVERY - MariaDB sustained replica lag on m1 on db2160 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2160&var-port=13321 [07:07:16] (PrometheusMysqldExporterFailed) firing: Prometheus-mysqld-exporter failed (db1206:9104) - TODO - https://grafana.wikimedia.org/d/000000278/mysql-aggregated - https://alerts.wikimedia.org/?q=alertname%3DPrometheusMysqldExporterFailed [07:07:27] ^known [07:45:32] (MysqlReplicationLag) firing: (2) MySQL instance db1134:9104 has too large replication lag (1h 13m 35s) - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLag [07:47:16] (PrometheusMysqldExporterFailed) resolved: Prometheus-mysqld-exporter failed (db1206:9104) - TODO - https://grafana.wikimedia.org/d/000000278/mysql-aggregated - https://alerts.wikimedia.org/?q=alertname%3DPrometheusMysqldExporterFailed [07:50:32] (MysqlReplicationLag) firing: (3) MySQL instance db1134:9104 has too large replication lag (49m 41s) - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLag [07:55:32] (MysqlReplicationLag) firing: (3) MySQL instance db1134:9104 has too large replication lag (9m 55s) - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLag [08:05:32] (MysqlReplicationLag) firing: (3) MySQL instance db1134:9104 has too large replication lag (9m 55s) - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLag [12:02:07] marostegui: what's the right mariadb server version to do a load test? [12:02:20] 10.6? [12:02:25] yes, sorry [12:02:31] db1133 if you like [12:02:37] no, I mean the minor version [12:02:44] I think that has 10.6.10 [12:02:50] which version you want? [12:03:09] the one you want me to test, I think there was a regresion on one, so skipping that [12:03:16] but 10.4? [12:03:37] no, I want to test 10.6, but maybe the regression was in 10.4? idk [12:03:51] we have no hosts with any version regressions [12:03:54] ok [12:03:55] you can use db1133 [12:04:19] so 10.6.10 is a good test, right? [12:04:23] yep [12:04:29] we are skipping 10.6.11 [12:04:30] ok, sorry for the confussion [12:04:35] ah, I see [12:04:44] so 10.6.10 and later 10.6.12 [12:04:51] correct [12:04:55] I knew there was something I missed [12:05:47] (MysqlReplicationLag) firing: MySQL instance db2173:9104 has too large replication lag (3d 21h 48m 44s) - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica - https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&refresh=1m&var-job=All&var-server=db2173&var-port=9104 - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLag [12:06:21] can I break also db1133? I want to do the test on a more realistic scenario, without it replicating and with unsafe options [12:06:26] yeah [12:06:27] go for it [12:06:30] so I would want to wipe it [12:06:40] later I can reload it with anything you want [12:06:50] don't worry, do anything you like with it [12:07:00] ok, taking it for T301879 [12:07:01] T301879: Test MariaDB 10.6 on Bullseye - https://phabricator.wikimedia.org/T301879 [12:07:05] yep [12:07:09] actually T319383 [12:07:10] T319383: Mydumper incompatibility with MariaDB 10.6 (was: Logical recoveries (myloader) to db2098:s7 are failing with "Lock wait timeout exceeded; try restarting transaction") - https://phabricator.wikimedia.org/T319383 [12:25:37] one last FYI: I will enable for db1133 the change buffer (among other performance-related unsecure options)- I am ok with the loading getting corrupted for a performance test [16:05:47] (MysqlReplicationLag) firing: MySQL instance db2173:9104 has too large replication lag (2d 0h 8m 9s) - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica - https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&refresh=1m&var-job=All&var-server=db2173&var-port=9104 - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLag [17:02:18] * Emperor finishes the week by endearing themself to the tar maintainer [17:02:22] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1025331 [20:05:47] (MysqlReplicationLag) firing: MySQL instance db2173:9104 has too large replication lag (1h 42m 47s) - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica - https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&refresh=1m&var-job=All&var-server=db2173&var-port=9104 - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLag [20:15:32] (MysqlReplicationLag) resolved: MySQL instance db2173:9104 has too large replication lag (5m 16s) - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica - https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&refresh=1m&var-job=All&var-server=db2173&var-port=9104 - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLag