[01:09:31] <icinga-wm>	 PROBLEM - MariaDB sustained replica lag on m1 on db2160 is CRITICAL: 15.2 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2160&var-port=13321
[01:09:37] <icinga-wm>	 PROBLEM - MariaDB sustained replica lag on m1 on db1117 is CRITICAL: 15 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1117&var-port=13321
[01:11:17] <icinga-wm>	 RECOVERY - MariaDB sustained replica lag on m1 on db2160 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2160&var-port=13321
[01:11:25] <icinga-wm>	 RECOVERY - MariaDB sustained replica lag on m1 on db1117 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1117&var-port=13321
[06:30:50] <marostegui>	 Amir1: are you touching s3?                 Last_SQL_Error: Error 'Multiple primary key defined' on query. Default database: 'acewiki'. Query: 'alter table site_stats ADD PRIMARY KEY(ss_row_id), drop index if exists ss_row_id' that's on db2094:3313
[07:31:33] <Amir1>	 marostegui: ah. Yeah. Let me fix it
[07:35:24] <marostegui>	 no rush 
[07:35:33] <marostegui>	 it's sanitarium in codfw, which isn't used 
[07:59:52] <Amir1>	 the good thing is that I already have the bash script for the clean up
[08:00:24] <Amir1>	 it's a set of codfw dirfts that exists on the sanitarium master in codfw but not in sanitarium (missing PK)
[08:06:54] <Amir1>	 it is flowing back but I think it'll break again with the new set of tables that got fixed
[09:01:48] <Emperor>	 Platonides: TYVM :)
[15:48:53] <Emperor>	 Huh, deleting a ghost objects causes a row to appear in the databases that previously knew nothing of the ghost object.
[15:50:45] <jynus>	 :-O
[15:56:43] <Emperor>	 so "consistency can be achieved by deletion"
[16:33:02] <Emperor>	 part of me wants to try deleting something that really doesn't exist, see if that makes new database rows or not...