[05:46:11] <marostegui>	 https://phabricator.wikimedia.org/T311106#8058425 5000????
[06:31:16] <jinxer-wm>	 (PrometheusMysqldExporterFailed) firing: Prometheus-mysqld-exporter failed (db2161:9104)  - https://grafana.wikimedia.org/d/000000278/mysql-aggregated - https://alerts.wikimedia.org/?q=alertname%3DPrometheusMysqldExporterFailed
[06:44:18] <marostegui>	 ^ me
[06:45:11] <Amir1>	 marostegui: cumulative, it was still 600 
[06:45:16] <marostegui>	 ah right
[06:45:28] <marostegui>	 so we are still around those numbers
[06:47:23] <Amir1>	 yup
[06:47:45] <Amir1>	 the reason I put it is that the time it takes to die changes, I think it's some sort of a leak
[06:48:00] <marostegui>	 and 10.4 died around 700 right?
[06:48:07] <marostegui>	 Like when we disabled p_s entirely on 1.6
[06:48:09] <marostegui>	 10.6
[06:48:50] <Amir1>	 yeah
[08:21:16] <jinxer-wm>	 (PrometheusMysqldExporterFailed) resolved: Prometheus-mysqld-exporter failed (db2161:9104)  - https://grafana.wikimedia.org/d/000000278/mysql-aggregated - https://alerts.wikimedia.org/?q=alertname%3DPrometheusMysqldExporterFailed
[08:36:16] <marostegui>	 jynus: would you mind preparing a patch with the backup changes needed to replace db2078 with db2160 so I can merge it next week?
[08:36:27] <marostegui>	 jynus: No need to touch the proxies, I will change those later today 
[08:36:45] <jynus>	 ok
[08:36:48] <marostegui>	 thanks
[08:37:29] <jynus>	 are you aware of the grants?
[08:37:54] <marostegui>	 They should've not changed, the data has been copied over
[08:38:12] <jynus>	 ok
[08:38:24] <marostegui>	 so db2160 has been cloned from db2078 entirely
[08:42:35] <jynus>	 is there a ticket number?
[08:43:28] <marostegui>	 https://phabricator.wikimedia.org/T311493
[08:44:08] <jynus>	 https://gerrit.wikimedia.org/r/c/operations/puppet/+/811884
[08:45:18] <marostegui>	 thanks
[08:45:25] <marostegui>	 I might merge it later today or tomorrow morning if that's ok
[08:48:35] <jynus>	 ok
[08:49:08] <jynus>	 please think if there is something that could block you next week, I will be back on the 20 of july
[08:50:03] <marostegui>	 will do!
[09:07:09] <jynus>	 I am refreshing the mediabackup dumps, just in case (T312321) https://phab.wmfusercontent.org/file/data/kmsar4qy4ppqxwy73p4q/PHID-FILE-svkov6wzgv3unmqxdznk/Screenshot_20220707_110536.png
[09:07:09] <stashbot>	 T312321: Degraded RAID on db1176 - https://phabricator.wikimedia.org/T312321
[09:07:32] <marostegui>	 :)
[09:13:09] <jynus>	 I think -I may be wrong- that the plan is to return the borrowed dbs to you next quarter, when definitive hw arrives
[09:22:02] <marostegui>	 No rush :)
[10:41:00] <jynus>	 he he
[10:41:08] <jynus>	 we both said thank you at the same time
[11:04:58] <marostegui>	 great minds think alike
[11:08:45] <jynus>	 you mean chris and you, right? XD
[11:12:11] <marostegui>	 xdddd
[13:19:00] <Amir1>	 jynus: marostegui if you remember anything that's missing, please add them T312538
[13:19:00] <stashbot>	 T312538: Collect list of tickets done for fixing core drifts  - https://phabricator.wikimedia.org/T312538
[13:20:56] <marostegui>	 I think Unherreider has opened a lot too
[13:21:05] <marostegui>	 Like the ones opened yesterday
[13:21:41] <marostegui>	 Ah you also added https://phabricator.wikimedia.org/T132416 which already have a bunch of subtasks
[13:21:42] <marostegui>	 good
[13:22:10] <marostegui>	 jynus: I just merged https://gerrit.wikimedia.org/r/c/operations/puppet/+/811884
[14:13:19] <Amir1>	 marostegui: started the last schema change on db1160, it's the fun one on revision table. Probably will be done by tomorrow
[14:13:33] <marostegui>	 ok!
[14:13:49] <marostegui>	 I think I have some pending, so once you are fully done, let me know
[14:16:13] <Amir1>	 sure
[14:16:54] <marostegui>	 I am going to give weight to db1132
[14:32:03] <jynus>	 do you want me to do a quick dump test, for a small db as a test?
[14:32:57] <marostegui>	 to test db2078?
[14:33:04] <marostegui>	 well its replacement
[14:33:08] <jynus>	 yeah, I trust everyhing is good
[14:33:11] <jynus>	 but just in case
[14:33:13] <marostegui>	 yeah, please test it
[14:33:25] <jynus>	 unless you are still doing maintenance there
[14:33:32] <marostegui>	 no, go for it
[14:33:34] <jynus>	 something that doesn't involves otrs
[14:33:37] <jynus>	 :-D
[14:33:42] <marostegui>	 I need to change the proxies, but that shouldn't block you
[14:33:49] <jynus>	 yeah
[14:33:56] <jynus>	 as in, it wouldn't interact
[14:34:17] <jynus>	 let me see the smallest m db
[14:34:33] <jynus>	 if only we had a dashboard to check that quickly...
[14:35:30] <jynus>	 m5 is only 20 gb, should take just a few minutes to run
[15:33:13] <jynus>	 backup from db2160.codfw.wmnet:3325 finished correctly in 20 minutes BTW
[15:33:49] <marostegui>	 good!