[05:59:06] 171970768-171970768 180363398-180363398 [09:07:17] jynus: I am curious, why do we have to backup sources in eqiad for x1? [09:07:33] Ah I guess cause db1225 crashes you set up db1216 temporarily? [09:07:48] ah s/to/two/ ? [09:07:54] *two yeah [09:07:55] sorry [09:08:30] ideally we have two of everything, for fast switchover (e.g. so we unblock you soon), but we don't necesarilly have the capacity [09:08:42] current ones reflect those that crashed more frequently [09:08:56] cool thanks [09:09:07] plus x1 is easy to fit because it is smaller [09:10:08] so we have something like 2 hosts more than the minimum for handling hw issues in general [10:06:20] Any thoughts on what can be going on with this repo? https://gerrit.wikimedia.org/r/c/operations/software/+/920653 [10:06:23] This is so odd [10:11:03] stupid question: did you try rebasing manually? [10:11:14] or do you want me to try? [10:11:23] Yeah I did [10:12:00] let me try the same patch but with a different changest number, just in case [10:12:03] I had the same issue before but a recheck fixed it (and the repo was rebased before): https://gerrit.wikimedia.org/r/c/operations/software/+/920652 [10:15:00] hopefully I don't make the problem worse 0:-D [10:15:24] I sent https://gerrit.wikimedia.org/r/c/operations/software/+/920657 [10:15:34] which should be the same as yours, let's see [10:16:27] weird, because they have the same parrent [10:17:01] so strange [10:17:12] try merging mine, I guess there is no difference [10:17:28] and maybe resetting your local repor or something [10:18:05] repo, branch and parent is the same [10:18:09] I just rebased and it got your change fine [10:18:12] so I don't fully understand the difference [10:18:12] so strange [10:18:35] maybe jenkins hates you :-D [10:18:48] that's likely, as much as I hate them! [10:19:42] i don't know, try what I said, nuking the repo and cloning it again, maybe there is something that cannot be seen creating you problems [10:20:07] yeah I will do that [10:20:30] thanks for the help [11:32:18] marostegui: Thanks to Moritz now wmfdb and wmfmariadbpy are in cumin2002. Wanna test/check? [11:32:53] i am busy with something now but I will soon [11:33:02] Awesome [11:33:07] thanks for doing it [11:33:08] no rush on my side [11:45:51] I go eat lunch, will be back in an hour or so [12:47:04] PROBLEM - MariaDB sustained replica lag on m2 on db1217 is CRITICAL: 1.437e+04 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1217&var-port=13322 [12:47:16] PROBLEM - MariaDB sustained replica lag on m2 on db2133 is CRITICAL: 1.418e+04 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2133&var-port=9104 [12:52:03] Amir1: it seems to be working fine, I have done a switchover, also a table comparison and a stop-in-sync and they all worked fine [12:53:22] Awesome. I'll finish it once I'm back [12:53:42] excellent thanks [12:53:57] maybe we need to document those packaging processes at some point too [12:54:29] wow m2 is super delayed [12:55:42] This is why: https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&refresh=1m&var-job=All&var-server=db1195&var-port=9104&viewPanel=33 [12:56:00] https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&refresh=1m&var-job=All&var-server=db1195&var-port=9104&viewPanel=33&from=now-24h&to=now [12:56:50] https://grafana.wikimedia.org/goto/S1UQByUVz?orgId=1 [12:57:25] There are some pretty big INSERTS on OTRS [12:57:29] At that time [13:01:14] RECOVERY - MariaDB sustained replica lag on m2 on db2133 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2133&var-port=9104 [13:02:36] RECOVERY - MariaDB sustained replica lag on m2 on db1217 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1217&var-port=13322 [13:04:12] PROBLEM - MariaDB sustained replica lag on m2 on db2160 is CRITICAL: 1068 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2160&var-port=13322 [13:21:14] RECOVERY - MariaDB sustained replica lag on m2 on db2160 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2160&var-port=13322 [14:39:20] PROBLEM - MariaDB sustained replica lag on m2 on db2133 is CRITICAL: 2392 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2133&var-port=9104 [14:40:34] PROBLEM - MariaDB sustained replica lag on m2 on db1217 is CRITICAL: 1399 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1217&var-port=13322 [14:43:42] RECOVERY - MariaDB sustained replica lag on m2 on db1217 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1217&var-port=13322 [14:44:00] RECOVERY - MariaDB sustained replica lag on m2 on db2133 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2133&var-port=9104 [14:45:26] PROBLEM - MariaDB sustained replica lag on m2 on db2160 is CRITICAL: 287.4 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2160&var-port=13322 [14:50:10] RECOVERY - MariaDB sustained replica lag on m2 on db2160 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2160&var-port=13322