[05:50:41] <marostegui>	 I am going to switch sanitarium masters in s4 codfw, so going to stop s4 codfw for a bit
[05:50:44] <marostegui>	 (the master that is)
[06:02:00] <marostegui>	 All done, I am going to check that the data is being sanitized correctly
[07:52:34] <Emperor>	 I see still some swift nodes need rebooting for T310483, le sigh
[08:01:16] <jinxer-wm>	 (PrometheusMysqldExporterFailed) firing: Prometheus-mysqld-exporter failed (db2156:9104)  - https://grafana.wikimedia.org/d/000000278/mysql-aggregated - https://alerts.wikimedia.org/?q=alertname%3DPrometheusMysqldExporterFailed
[08:06:16] <jinxer-wm>	 (PrometheusMysqldExporterFailed) firing: (2) Prometheus-mysqld-exporter failed (db2156:9104)  - https://grafana.wikimedia.org/d/000000278/mysql-aggregated - https://alerts.wikimedia.org/?q=alertname%3DPrometheusMysqldExporterFailed
[08:09:24] <marostegui>	 ^ that's me
[08:09:30] <marostegui>	 I am setting up new hosts
[08:16:16] <jinxer-wm>	 (PrometheusMysqldExporterFailed) firing: (2) Prometheus-mysqld-exporter failed (db2156:9104)  - https://grafana.wikimedia.org/d/000000278/mysql-aggregated - https://alerts.wikimedia.org/?q=alertname%3DPrometheusMysqldExporterFailed
[09:46:17] <jynus>	 I belive that is the case, because the 2 runners are not from the US
[09:47:11] <jynus>	 (I don't think it is a bad thing, given they are recorded and ones doesn't need to attend all of them)
[10:21:37] <Amir1>	 We don't have any public holidays here until October 😢
[10:40:14] <marostegui>	 jynus: I can join that
[10:40:21] <marostegui>	 It is at 18:00 CEST right?
[10:40:59] <jynus>	 you don't have to go if it is outside of your working hours, I am sure it will be recorded
[10:41:44] <jynus>	 e.g. maybe between you and Amir1 at least 1 can attend live
[10:41:55] <marostegui>	 jynus: I will attend live :)
[10:42:22] <jynus>	 and note you are not the only one to mention the schedule is not great
[10:42:30] <Amir1>	 I will attend as well
[10:46:16] <jinxer-wm>	 (PrometheusMysqldExporterFailed) resolved: Prometheus-mysqld-exporter failed (db2156:9104)  - https://grafana.wikimedia.org/d/000000278/mysql-aggregated - https://alerts.wikimedia.org/?q=alertname%3DPrometheusMysqldExporterFailed
[10:46:25] <Amir1>	 marostegui: backporting the rc optimizer hand-holding change. Stay tuned
[10:46:31] <marostegui>	 XD
[12:43:41] <jynus>	 so I though I was doing a trivial alter on my misc service
[12:44:02] <jynus>	 but I just realized that the table has 274 million rows
[12:44:19] <jynus>	 maybe time to cleanup old backup records?
[12:45:17] <marostegui>	 I remember we had a task to implement a clean up script
[12:45:42] <jynus>	 let me try to find it
[12:46:28] <jynus>	 T205627 I think
[12:46:29] <stashbot>	 T205627: Purge and monitor old metadata for the mariadb backups database - https://phabricator.wikimedia.org/T205627
[12:46:37] <marostegui>	 yeah that one
[12:48:30] <jynus>	 I think I will look into it, if not deletion, at least move historical records to a separate table
[12:50:08] <jynus>	 I think I may create lag on m1 replicas because of this big alter in some minutes
[12:50:20] <marostegui>	 that's fine, just downtime them
[12:50:34] <jynus>	 doing for 1 hour
[12:58:13] <RhinosF1>	 Amir1: we are 1.38 now
[12:58:24] <RhinosF1>	 also we are struggling with the actor migration
[12:58:48] <RhinosF1>	 it could take months on miraheze and are setup is not designed for that
[13:20:12] <jynus>	 Query OK, 0 rows affected (40 min 49.843 sec)
[13:20:28] <jynus>	 I think it was the fact that it was compressed + online that took so long
[13:26:03] <jynus>	 Should be finished in 1 hour, I have downtime'd the alerts for a couple: https://grafana.wikimedia.org/goto/9j3thSenz?orgId=1
[15:00:35] <Amir1>	 I'm fixing some misc drifts live, you might see some large writes on s7 and s5
[15:31:01] <jynus>	 marostegui: would you have 5 minutes of time for a feedback question tomorrow morning?
[15:32:27] <marostegui>	 sure thing
[15:33:50] <jynus>	 I would want Amir's opinion, too, as it is about presenting backup statistics, although probably not in (my) morning :-)
[15:44:05] <Amir1>	 jynus: let me know when you want it :)
[15:44:15] <Amir1>	 marostegui: I brought gifts for you T312028
[15:44:16] <stashbot>	 T312028: Random drifts on s3 part 2 - https://phabricator.wikimedia.org/T312028
[15:45:54] <marostegui>	 esus
[15:45:55] <marostegui>	 Jesus
[15:46:00] <marostegui>	 I will get them done before the switchover
[15:46:17] <marostegui>	 Or at least...try to :p
[15:46:40] <Amir1>	 :D
[15:50:19] <marostegui>	 Amir1: I assumed the first part   field-unsigned-mismatch means that   `bp_user` int(11) NOT NULL, needs to be unsigned right?
[15:50:38] <Amir1>	 yeah
[15:50:41] <marostegui>	 oki
[15:51:18] <Amir1>	 I think it was probably even requested but bot_passwords doesn't exist in every wiki (only non-SUL wikis)
[16:01:07] <marostegui>	 Amir1: where can I find the canonical definitions for the tables?
[16:01:14] <marostegui>	 So I don't have to bother you about the changes :p
[16:01:24] <Amir1>	 marostegui: https://github.com/wikimedia/mediawiki/blob/master/maintenance/tables-generated.sql
[16:01:27] <marostegui>	 I remember tables.sql was deprecated right
[16:01:28] <marostegui>	 There yo go
[16:01:30] <marostegui>	 Thanks <3
[16:03:26] <Amir1>	 Thanks for fixing them!