[05:07:32] PROBLEM - MariaDB sustained replica lag on s8 on db1214 is CRITICAL: 2 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1214&var-port=9104 [05:08:32] RECOVERY - MariaDB sustained replica lag on s8 on db1214 is OK: (C)2 ge (W)1 ge 0.4 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1214&var-port=9104 [05:34:50] Going to be switching s8 codfw [05:34:54] master [07:13:41] were all those lag alerts over the weekend expected? [07:14:19] They are very small spikes, we are investigating why they are showing up [07:25:20] 👍 [07:26:45] FYI "Last snapshot for s6 at codfw (db2197) taken on 2024-05-20 01:02:42 is 473 GiB, but the previous one was 571 GiB, a change of -17.1 %" [07:28:33] nice [11:43:44] zabe: I will get that one for wikidata done right now [11:43:47] Is that the only urgent wiki? [11:45:04] yep [11:45:09] thanks [11:45:13] zabe: confirm revtag table? [11:45:17] Your select is for revision [11:46:10] yes, revtag table is correct. it stores revision id. my select was only to show that revision ids are so large, that a signed int is no longer enough. [11:46:15] Ah ok :) [11:46:31] bloody FK references ;) [11:47:15] ok it is done in wikidata [11:47:20] that was quick [11:47:43] Query OK, 437275 rows affected (11.273 sec) [11:47:53] the table is tiny, so I did it without replication [11:47:57] *with [11:49:06] zabe: I have changed it from UBN to Medium, that ok with you? [11:49:16] yes [11:49:19] thank you:) [11:49:51] isn't that revtag table on everywiki? [11:50:08] nope, only where there's translate extension [11:50:20] Is there a dblist for that? [11:50:46] unfortunately not [11:50:49] If it is this small everywhere I can just run it on masters [11:50:54] Reedy: Ok, I will look for it then :) [11:51:01] It's "only" 40 ish wikis [11:51:09] ah great [11:51:20] I suspect meta/mediawiki.org might be the bigger tables [11:51:30] Ok, I will get a list on the task [11:51:32] Thanks :) [11:51:54] meta is 3270654 [11:52:11] mw.o is 1840745 [11:52:28] so more than wikidata, but... [11:52:35] oh commons [11:52:36] * Reedy looks [11:52:54] that's only 600K [11:53:06] yeah commons can be done directly [11:53:37] meta is around 400M, might be risky on the master [11:54:16] row count of the revtag table :P [11:54:19] not revision [11:54:26] just for wikidatawiki it was because BIG NUMBER [11:54:30] I meant 400 MB on disk [11:54:34] ahh [11:54:46] haha sorry! [11:58:31] zabe: do we have a maintenance script to fix the rows? [11:58:53] I am going to a meeting, if something urgent comes up, ping me here [11:58:56] probably not (yet?) [11:59:13] After the meeting I will get s3 and s4 done [11:59:21] Those are small enough to be run on the master [11:59:47] There's a populateFuzzy... [12:00:00] Which does [12:00:00] $dbw->replace( 'revtag', [ [ 'rt_type', 'rt_page', 'rt_revision' ] ], $inserts, __METHOD__ ); [12:00:19] * Reedy looks [12:01:34] batch size of 5K feels a bit big [12:01:56] yeah it's big [12:04:02] would that replace statement actually remove the corrupted rows? since the revision id changes, they have a different pk. [12:04:27] I guess we probably have to manually drop the rows with rt_revision = 2147483647 [12:04:57] https://gerrit.wikimedia.org/r/1034060 drops 5K to 500 [12:06:44] PROBLEM - MariaDB sustained replica lag on s4 on db1160 is CRITICAL: 3.8 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1160&var-port=9104 [12:06:46] it's only 1000 rows [12:07:13] I guess the real trick here is running the fuzzyTags script only against the pages that need it running against [12:07:27] the script only takes a list of NS [12:07:44] RECOVERY - MariaDB sustained replica lag on s4 on db1160 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1160&var-port=9104 [12:07:47] Feed it a list of page ids or something? [12:09:05] most of the 1000 rows are actually not rt_type='fuzzy' [12:10:01] yay ;P [12:44:42] PROBLEM - MariaDB sustained replica lag on s3 on db2149 is CRITICAL: 4.8 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2149&var-port=9104 [12:45:42] RECOVERY - MariaDB sustained replica lag on s3 on db2149 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2149&var-port=9104 [13:24:46] PROBLEM - MariaDB sustained replica lag on s8 on db1214 is CRITICAL: 3 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1214&var-port=9104 [13:27:46] PROBLEM - MariaDB sustained replica lag on s8 on db1214 is CRITICAL: 3.2 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1214&var-port=9104 [13:35:45] RECOVERY - MariaDB sustained replica lag on s8 on db1214 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1214&var-port=9104 [13:45:05] I'm looking at https://gerrit.wikimedia.org/r/c/operations/puppet/+/1032809 -- is it best/common practice to update both maintain-views and filtered-tables in the same patch, or should they be modified and deployed in two separate patches? [13:49:23] dhinus: It depends on the operation. If it is an existing table that goes from being OK to being filtered, so it is better to update tables filetered, and then we need to restart replication and then clean the data, so better two separate patches [13:49:55] dhinus: In this case, if you are just deprecating things, then a single patch is fine [13:50:47] dhinus: are those columns removed from the table? [13:51:24] If the columns are still on the table and won't be removed, there is no need to remove them from filtered tables, you can just remove the views [13:51:43] We only remove fields from there once they are dropped [13:55:27] marostegui: thanks, I imagine the plan is to remove the columns from the table /after/ the views have been updated [13:55:51] dhinus: In that case, let's not include filtered_tables in the patch just yet [13:55:56] so filtered_tables can be updated later if/when the columns are dropped [13:56:00] I can do that whenever the schema change ticket arrives to us [13:56:04] dhinus: yep [13:56:06] sounds good [13:56:17] thanks [13:57:51] PROBLEM - MariaDB sustained replica lag on s8 on db1203 is CRITICAL: 2.2 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1203&var-port=9104 [13:59:49] RECOVERY - MariaDB sustained replica lag on s8 on db1203 is OK: (C)2 ge (W)1 ge 0.8 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1203&var-port=9104 [14:31:45] PROBLEM - MariaDB sustained replica lag on s8 on db1192 is CRITICAL: 2.8 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1192&var-port=9104 [14:34:53] PROBLEM - MariaDB sustained replica lag on s8 on db1193 is CRITICAL: 2.2 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1193&var-port=9104 [14:36:45] RECOVERY - MariaDB sustained replica lag on s8 on db1192 is OK: (C)2 ge (W)1 ge 0.8 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1192&var-port=9104 [14:37:51] PROBLEM - MariaDB sustained replica lag on s8 on db1214 is CRITICAL: 3.2 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1214&var-port=9104 [14:39:53] RECOVERY - MariaDB sustained replica lag on s8 on db1214 is OK: (C)2 ge (W)1 ge 0.8 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1214&var-port=9104 [14:39:53] RECOVERY - MariaDB sustained replica lag on s8 on db1193 is OK: (C)2 ge (W)1 ge 0.8 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1193&var-port=9104 [14:44:53] PROBLEM - MariaDB sustained replica lag on s8 on db1214 is CRITICAL: 3.6 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1214&var-port=9104 [14:53:53] PROBLEM - MariaDB sustained replica lag on s8 on db1214 is CRITICAL: 2.4 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1214&var-port=9104 [14:59:53] RECOVERY - MariaDB sustained replica lag on s8 on db1214 is OK: (C)2 ge (W)1 ge 0.8 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1214&var-port=9104 [15:04:53] PROBLEM - MariaDB sustained replica lag on s8 on db1214 is CRITICAL: 2.2 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1214&var-port=9104 [15:07:53] PROBLEM - MariaDB sustained replica lag on s8 on db1214 is CRITICAL: 2.4 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1214&var-port=9104 [15:10:53] PROBLEM - MariaDB sustained replica lag on s8 on db1214 is CRITICAL: 2.2 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1214&var-port=9104 [15:11:53] RECOVERY - MariaDB sustained replica lag on s8 on db1214 is OK: (C)2 ge (W)1 ge 0.8 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1214&var-port=9104 [15:36:54] PROBLEM - MariaDB sustained replica lag on s8 on db1203 is CRITICAL: 3.2 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1203&var-port=9104 [15:39:54] RECOVERY - MariaDB sustained replica lag on s8 on db1203 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1203&var-port=9104 [15:54:32] PROBLEM - MariaDB sustained replica lag on s8 on db1178 is CRITICAL: 2.2 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1178&var-port=9104 [15:56:32] PROBLEM - MariaDB sustained replica lag on s8 on db1178 is CRITICAL: 2.2 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1178&var-port=9104 [15:58:54] PROBLEM - MariaDB sustained replica lag on s8 on db1214 is CRITICAL: 2.2 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1214&var-port=9104 [16:00:32] PROBLEM - MariaDB sustained replica lag on s8 on db1178 is CRITICAL: 2.2 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1178&var-port=9104 [16:02:32] RECOVERY - MariaDB sustained replica lag on s8 on db1178 is OK: (C)2 ge (W)1 ge 0.8 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1178&var-port=9104 [16:04:51] PROBLEM - MariaDB sustained replica lag on s8 on db1192 is CRITICAL: 2 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1192&var-port=9104 [16:04:55] RECOVERY - MariaDB sustained replica lag on s8 on db1214 is OK: (C)2 ge (W)1 ge 0.4 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1214&var-port=9104 [16:05:51] RECOVERY - MariaDB sustained replica lag on s8 on db1192 is OK: (C)2 ge (W)1 ge 0.4 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1192&var-port=9104 [16:07:33] PROBLEM - MariaDB sustained replica lag on s8 on db1178 is CRITICAL: 3.2 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1178&var-port=9104 [16:13:51] PROBLEM - MariaDB sustained replica lag on s8 on db1192 is CRITICAL: 4.4 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1192&var-port=9104 [16:13:57] PROBLEM - MariaDB sustained replica lag on s8 on db1193 is CRITICAL: 2.2 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1193&var-port=9104 [16:14:51] RECOVERY - MariaDB sustained replica lag on s8 on db1192 is OK: (C)2 ge (W)1 ge 0.8 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1192&var-port=9104 [16:16:33] RECOVERY - MariaDB sustained replica lag on s8 on db1178 is OK: (C)2 ge (W)1 ge 0.6 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1178&var-port=9104 [16:16:57] PROBLEM - MariaDB sustained replica lag on s8 on db1193 is CRITICAL: 2 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1193&var-port=9104 [16:17:57] PROBLEM - MariaDB sustained replica lag on s8 on db1214 is CRITICAL: 2.4 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1214&var-port=9104 [16:21:57] PROBLEM - MariaDB sustained replica lag on s8 on db1193 is CRITICAL: 2.6 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1193&var-port=9104 [16:23:57] RECOVERY - MariaDB sustained replica lag on s8 on db1193 is OK: (C)2 ge (W)1 ge 1 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1193&var-port=9104 [16:27:59] RECOVERY - MariaDB sustained replica lag on s8 on db1214 is OK: (C)2 ge (W)1 ge 0.6 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1214&var-port=9104 [16:32:59] PROBLEM - MariaDB sustained replica lag on s8 on db1214 is CRITICAL: 2.4 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1214&var-port=9104 [16:38:59] PROBLEM - MariaDB sustained replica lag on s8 on db1214 is CRITICAL: 2.2 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1214&var-port=9104 [16:41:59] PROBLEM - MariaDB sustained replica lag on s8 on db1214 is CRITICAL: 3 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1214&var-port=9104 [16:42:52] this is getting out of hand [16:42:59] maybe bump it to five seconds for now? [16:51:59] PROBLEM - MariaDB sustained replica lag on s8 on db1214 is CRITICAL: 2.2 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1214&var-port=9104 [16:53:32] Amir1: +1 [16:53:59] PROBLEM - MariaDB sustained replica lag on s8 on db1214 is CRITICAL: 2.4 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1214&var-port=9104 [16:59:27] Amir1: I'm running maintain-views on pagelinks to fix the issue just reported on wikitech-l (cc dhinus) [16:59:42] taavi: the change hasn't landed on s5 yet [16:59:51] https://orchestrator.wikimedia.org/web/cluster/alias/s5 [16:59:58] you will need to run it again once it's done [17:00:00] PROBLEM - MariaDB sustained replica lag on s8 on db1214 is CRITICAL: 2.8 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1214&var-port=9104 [17:00:54] taavi: I've run it for other section. The reporter is from dewiki, they are reporting for s5 which just needs to actually get the schema change [17:01:00] *sections [17:01:00] RECOVERY - MariaDB sustained replica lag on s8 on db1214 is OK: (C)2 ge (W)1 ge 0.8 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1214&var-port=9104 [17:01:12] taavi thanks I'm not at my desk now, I will check later [17:05:22] Amir1: weirdly enough running maintain-views now seems to have fixed it already. or are you saying there's going to be an another schema change immediately afterwards that will break it again? [17:06:00] it's at the middle of the alert there, so it might work but still we should wait for the alter to finish [17:06:09] it might break again [17:09:59] PROBLEM - MariaDB sustained replica lag on s8 on db1203 is CRITICAL: 3.4 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1203&var-port=9104 [17:14:01] RECOVERY - MariaDB sustained replica lag on s8 on db1203 is OK: (C)2 ge (W)1 ge 0.8 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1203&var-port=9104 [17:28:09] PROBLEM - MariaDB sustained replica lag on s4 on db1248 is CRITICAL: 8 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1248&var-port=9104 [17:35:09] RECOVERY - MariaDB sustained replica lag on s4 on db1248 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1248&var-port=9104 [17:49:57] PROBLEM - MariaDB sustained replica lag on s8 on db1192 is CRITICAL: 3.2 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1192&var-port=9104 [17:54:57] RECOVERY - MariaDB sustained replica lag on s8 on db1192 is OK: (C)2 ge (W)1 ge 0.8 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1192&var-port=9104 [17:58:05] PROBLEM - MariaDB sustained replica lag on s8 on db1193 is CRITICAL: 3 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1193&var-port=9104 [18:00:08] RECOVERY - MariaDB sustained replica lag on s8 on db1193 is OK: (C)10 ge (W)5 ge 1.6 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1193&var-port=9104