[06:05:57] Hello, I just did s5 switchover. Having to do switchovers alone isn't nice, it is risky and it is definitely not ok. Next time if there is no one around I will re-schedule it [07:19:01] sorry, 06:00 UTC is quite a bit before my usual start time, and I didn't think I was generally needed at switchovers [10:11:36] PROBLEM - MariaDB sustained replica lag on s7 on db1181 is CRITICAL: 2.134e+04 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1181&var-port=9104 [10:32:13] RECOVERY - MariaDB sustained replica lag on s7 on db1181 is OK: (C)2 ge (W)1 ge 0.6 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1181&var-port=9104 [13:11:03] https://wikitech.wikimedia.org/wiki/Swift/How_To also now updated to note the new process [13:15:02] https://gerrit.wikimedia.org/r/c/operations/software/swift-ring/+/775856 update to the old repo saying not to use it any more [14:07:18] Amir1: is it possible for auto_schema to run a schema change against a primary instance without replication? [14:54:41] kormat: primary of active dc? it does it but better done manually though. Primary of inactive dc? if you add a skip= argument that includes a replica under it, it would skip it [14:55:08] (=would run it without replication and does each one of replicas one by one except the skipped one) [15:49:39] * kormat looks at s3, and glares at the idea of doing it manually [15:50:45] hmm, kormat, is it the flaggedrevs things? [15:50:57] for that the list should be small [15:51:03] Amir1: https://phabricator.wikimedia.org/T300774#7782153 [15:51:35] ok so in that case s3 is already done [15:51:44] oh for that [15:51:59] kormat: the new master is done, you just need to run it on the replica [15:55:02] yeah, i got that :) [15:55:15] can you explain why you're suggesting running schema changes manually in the case above? [15:56:15] I don't know exactly in which cases but it might basically lock the table and writes pile up [15:56:30] Manuel is saying it won't so that's good [15:56:39] but don't do it generally [15:57:17] Amir1: i'm familiar with the reasoning for running or not running a given schema change against a primary instance. i'm unclear on the auto_schema-related factors [15:58:18] there is nothing special about auto_schema in this regard [15:58:30] it runs it without replication [15:58:50] so it automatically detects that the instance has replicas, and disables replication? [16:00:31] it detects that it's the master of primary dc and disabled replication [16:01:03] it actually uses if instances has replicas to "enable" replication with exception of master of active dc [16:01:41] ok [16:01:48] and it won't try to depool the primary instance? [16:02:27] it won't [16:02:35] you can run it with dry run and check [16:04:49] Amir1: ok cool, thanks :) [16:05:11] ^^