[00:07:32] (MysqlReplicationLag) firing: MySQL instance db2181:9104 has too large replication lag (17h 47m 36s) - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica - https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&refresh=1m&var-job=All&var-server=db2181&var-port=9104 - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLag [01:06:51] PROBLEM - MariaDB sustained replica lag on m1 on db1117 is CRITICAL: 12.6 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1117&var-port=13321 [01:07:25] PROBLEM - MariaDB sustained replica lag on m1 on db2160 is CRITICAL: 9.2 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2160&var-port=13321 [01:10:19] RECOVERY - MariaDB sustained replica lag on m1 on db1117 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1117&var-port=13321 [01:12:15] RECOVERY - MariaDB sustained replica lag on m1 on db2160 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2160&var-port=13321 [02:07:32] (MysqlReplicationLag) resolved: MySQL instance db2181:9104 has too large replication lag (9m 45s) - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica - https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&refresh=1m&var-job=All&var-server=db2181&var-port=9104 - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLag [07:22:00] jynus: let me know when I can stop m1 on db1117 [07:22:13] any time [07:22:20] ah cool [07:22:22] doing it now, thanks! [07:51:21] jynus: would monday or tuesday be a good day for m1 master switchover? [07:51:27] I needs to be one of those, as I am off for the rest of the week [07:52:33] any time after 8UTC should have backups finished - in any case, they don't affect db1117 [07:52:39] ? [07:52:44] no, but bacula [07:52:48] ah! [07:53:11] let me see [07:53:15] sure, no rush [07:53:18] Take your time [07:53:23] I didn't even create the task yet :) [07:56:06] what about 11 UTC, too late? [07:57:35] which day? [07:57:44] any of the 2 [07:58:02] let's do Monday then [12:56:22] I am going to call it a day, I have started early today and I am also not feeling great, see you all tomorrow [13:03:18] take care! [16:23:55] If I ask nicely (pretty please!) could someone look at wikipedia-commons-local-public.ad/a/ad/0_73aa1_2d9bafe_orig.jpg and tell me what (if anything) MW / commons thinks about that object? It's not present in swift even as a tombstone record [I want to delete it to check the deletion propagates to the variant container DB correctly] [16:25:08] From what Platonides said previously, a commons admin may be able to do so, though I suspect suitable database-fu can achieve the same result [17:28:51] this drops 30M rows from 700M rows of externallinks table in commons https://commons.wikimedia.org/w/index.php?title=Module:Coordinates&diff=prev&oldid=731255004 [17:30:29] make it 40M [17:56:51] I'm sure the job queue loves you [20:00:39] Just triggered a full reparse of English Wikipedia https://en.wikipedia.org/w/index.php?title=Module:Citation/CS1/Configuration&diff=prev&oldid=1138454880 [20:01:13] that will also remove millions of rows from enwiki [22:55:26] Emperor: looking [22:56:18] https://commons.wikimedia.org/w/index.php?title=File:0_73aa1_2d9bafe_orig.jpg&redirect=no was renamed to [22:56:21] https://commons.wikimedia.org/wiki/File:%D0%A6%D0%B5%D1%80%D0%BA%D0%BE%D0%B2%D1%8C_%D0%9F%D0%BE%D0%BA%D1%80%D0%BE%D0%B2%D0%B0_%D0%9F%D1%80%D0%B5%D1%81%D0%B2%D1%8F%D1%82%D0%BE%D0%B9_%D0%91%D0%BE%D0%B3%D0%BE%D1%80%D0%BE%D0%B4%D0%B8%D1%86%D1%8B_%D1%81_%D0%BA%D0%B5%D0%BB%D1%8C%D1%8F%D0%BC%D0%B8,_%D0%BA%D0%BE%D1%80%D0%BF%D1%83%D1%81_%E2%84%961,_%D0%A1%D0%B2%D1%8F%D1%82%D0%BE-%D0%9F%D0%BE% [22:56:27] D0%BA%D1%80%D0%BE%D0%B2%D1%81%D0%BA%D0%BE%D0%B3%D0%BE_%D0%BC%D0%BE%D0%BD%D0%B0%D1%81%D1%82%D1%8B%D1%80%D1%8F,_%D0%9A%D0%B8%D0%B5%D0%B2.jpg [22:56:30] lol [22:56:58] I didn't expect that Cyrillic filename to be so ugly :D [22:57:25] File:Церковь Покрова Пресвятой Богородицы с кельями, корпус №1, Свято-Покровского монастыря, Киев.jpg [22:57:26] shorturl [22:57:49] it was renamed on 26 August 2022 [22:58:20] there should not be a wikipedia-commons-local-public.ad/a/ad/0_73aa1_2d9bafe_orig.jpg, delete freely