[08:59:27] marostegui: re: https://phabricator.wikimedia.org/T294295#7457483, table check completed successfully. do you mean let it catch up on replication, and then run db-compare against it? [09:23:16] PROBLEM - MariaDB sustained replica lag on s3 on db1154 is CRITICAL: 4.413e+04 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1154&var-port=13313 [09:25:18] ^ acking [09:25:18] ACKNOWLEDGEMENT - MariaDB sustained replica lag on s3 on db1154 is CRITICAL: 4.413e+04 ge 2 Kormat db1112 is catching up on replication T294295 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1154&var-port=13313 [09:41:16] RECOVERY - MariaDB sustained replica lag on s3 on db1154 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1154&var-port=13313 [09:57:43] kormat: yeah [09:58:37] codfw-swift replication has caught up, so I'll repool it after our meeting [10:32:00] sobanski: ah, i was wrong. ops@ gets a ton of automated email from the puppet private repo. i still find that human emails stand out very clearly from it, but yeah i got that the wrong way around. [10:32:10] https://lists.wikimedia.org/hyperkitty/list/ops@lists.wikimedia.org/ is the archives at least [10:33:16] Got it [10:34:52] I'd prefer automated and non-automated emails went to different lists WLOG, but... [10:35:01] sorry WLOG = Without Loss Of Generality [10:35:17] (there's no way i was going to guess that one, thanks for expanding :D) [10:35:27] and agreed [10:36:00] kormat: you've clearly never dated a mathmo ;-p [13:44:54] Emperor: not according to my calculations, no