[00:04:30] PROBLEM - MariaDB sustained replica lag on s1 on db1206 is CRITICAL: 31.6 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1206&var-port=9104 [00:14:30] RECOVERY - MariaDB sustained replica lag on s1 on db1206 is OK: (C)10 ge (W)5 ge 4.2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1206&var-port=9104 [00:33:30] PROBLEM - MariaDB sustained replica lag on s1 on db1206 is CRITICAL: 122.4 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1206&var-port=9104 [00:40:30] RECOVERY - MariaDB sustained replica lag on s1 on db1206 is OK: (C)10 ge (W)5 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1206&var-port=9104 [03:19:30] PROBLEM - MariaDB sustained replica lag on s1 on db1206 is CRITICAL: 11.8 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1206&var-port=9104 [03:20:30] RECOVERY - MariaDB sustained replica lag on s1 on db1206 is OK: (C)10 ge (W)5 ge 3.6 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1206&var-port=9104 [04:44:30] PROBLEM - MariaDB sustained replica lag on s1 on db1206 is CRITICAL: 10.6 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1206&var-port=9104 [04:47:30] RECOVERY - MariaDB sustained replica lag on s1 on db1206 is OK: (C)10 ge (W)5 ge 0.2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1206&var-port=9104 [05:17:30] PROBLEM - MariaDB sustained replica lag on s1 on db1206 is CRITICAL: 23.4 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1206&var-port=9104 [05:32:30] RECOVERY - MariaDB sustained replica lag on s1 on db1206 is OK: (C)10 ge (W)5 ge 3.2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1206&var-port=9104 [06:16:30] PROBLEM - MariaDB sustained replica lag on s1 on db1206 is CRITICAL: 19.6 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1206&var-port=9104 [06:22:09] Amir1: arnaudb you are not running anything on s7 codfw right? I want to switch its master either today or monday [06:23:37] nothing on my end nope [06:23:43] excellent thanks [06:29:30] RECOVERY - MariaDB sustained replica lag on s1 on db1206 is OK: (C)10 ge (W)5 ge 1.8 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1206&var-port=9104 [06:44:52] Amir1: it is a query the wikiexporter runs: SELECT /* WikiExporter::dumpPages */ /*! STRAIGHT_JOIN */ rev_id,rev_page,rev_actor,actor [07:13:31] PROBLEM - MariaDB sustained replica lag on s1 on db1206 is CRITICAL: 77.4 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1206&var-port=9104 [07:17:20] is that problem with s1 known? [07:17:44] Emperor: I literally was commenting on it yesterday and jus the line before [07:21:29] ah, OK, thanks :) [07:21:31] RECOVERY - MariaDB sustained replica lag on s1 on db1206 is OK: (C)10 ge (W)5 ge 1 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1206&var-port=9104 [07:59:36] Isn't that dumps? [07:59:56] They shouldn't be creating lag though [09:15:10] can I restart/upgrade db1245? I see marostegui set up a downtime there [09:16:28] It's a schema change [09:16:39] I see, let me see how it is going [09:17:10] is it for s4 or for s5? [09:27:30] I think it has been applied already, it is going by the db1248 [10:06:07] mmm, checking T364299 I don't think it has [10:06:08] T364299: Make rc_id a bigint - https://phabricator.wikimedia.org/T364299 [10:11:33] PROBLEM - MariaDB sustained replica lag on s1 on db1206 is CRITICAL: 28 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1206&var-port=9104 [10:13:33] RECOVERY - MariaDB sustained replica lag on s1 on db1206 is OK: (C)10 ge (W)5 ge 3.2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1206&var-port=9104 [10:19:33] PROBLEM - MariaDB sustained replica lag on s1 on db1206 is CRITICAL: 14 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1206&var-port=9104 [10:23:33] RECOVERY - MariaDB sustained replica lag on s1 on db1206 is OK: (C)10 ge (W)5 ge 2.6 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1206&var-port=9104 [10:28:02] marostegui: no I don't have anything on s7 for a bit [10:52:52] I may connect later to check something I have left ongoing, but otherwise I will soon leave until wednesday [10:52:54] Ok thanks! [12:23:24] I have pushed 10.6.18 to the repo [12:48:53] root@db1196:/srv/sqldata/enwiki# ls -Ssh | head --lines 30 [12:48:53] total 998G [12:49:13] finally enwiki is below 1TB. Even it's going to be like that for a couple of days only :D [12:50:13] Amir1: I am curious, why do you use that long command instead of: du -sh . [12:50:31] Although I like the Ssh [12:50:34] it's like ssssh [12:51:10] oh in this case, cuz I wanted to check how big pagelinks is too [12:51:59] it is so weird to see revision being so small [12:52:22] I know it is compressed, but still [12:52:47] a lot in slots, text and content [12:52:59] if you sum them up, that go quite high up [12:53:41] templatelinks is 90GB and pagelinks 73GB, this is them normalized. wow [12:54:02] at least several billions of rows