[08:14:47] <Amir1>	 started an alter table on revision table in s1 😭
[08:15:09] <Amir1>	 it has 24 hours for downtime, if it's not enough it might start paging,  but it's very very unlikely 
[08:15:31] <marostegui>	 Amir1: I would suggest to double check before leaving for the day and in case it hasn't finished, downtime it longer
[08:15:42] <Amir1>	 yeah
[08:15:52] <Amir1>	 will do. Thanks
[09:37:22] <Amir1>	 marostegui: btw, query site_stats table in wikidata (or commons or enwiki)
[09:45:31] <marostegui>	 oh nice
[09:45:40] <marostegui>	 <3
[09:45:56] <Amir1>	 if you can find a good way to measure improvements on wait time of the table, let me know :D
[09:46:42] <marostegui>	 maybe on performance data or something
[10:04:00] <milimetric>	 Amir1: does this break the views on the cloud replicas? "java.sql.SQLException: View 'etwiki_p.revision' references invalid table(s) or column(s) or function(s) or definer/invoker of view lack rights to use them"
[10:04:35] <marostegui>	 milimetric: probably
[10:04:44] <marostegui>	 Amir1: ^
[10:04:45] <milimetric>	 k... yeah, that'll block our monthly job
[10:05:14] <marostegui>	 milimetric: you use clouddb1021 right?
[10:05:18] <milimetric>	 yes
[10:05:24] <marostegui>	 let me fix it now
[10:05:44] <marostegui>	 milimetric: I would guess the report is for all the wikis?
[10:05:48] <milimetric>	 well, marostegui it'll be broken for dbs on s1 until the alter finishes, no?
[10:06:06] <marostegui>	 milimetric: but it might have not reached s1 yet
[10:06:09] <marostegui>	 that report is for etwiki
[10:06:13] <marostegui>	 which is on s3, and I am fixing it now
[10:06:15] <marostegui>	 let me check s1
[10:06:34] <marostegui>	 yeah it is broken there too
[10:06:38] <marostegui>	 let me fix it across all wikis
[10:07:01] <marostegui>	 milimetric: fixed for enwiki
[10:07:10] <Amir1>	 sorry I was afk for a second
[10:07:14] <milimetric>	 np, it's no rush
[10:07:23] <marostegui>	 milimetric: fixed also for s2
[10:07:44] <Amir1>	 what do we have for revision table? The only thing there was making rev_timestamp binary(14), it shouldn't break the view there
[10:08:10] <marostegui>	 milimetric: I have recreated all the views in clouddb1021 now, s3 is also done, so it should work everywhere
[10:09:07] <milimetric>	 I can confirm it works on etwiki, but should I start this job back up?  It'll take a few days.  Or will it break again?  And is it ok load-wise while the alter executes?
[10:09:29] <Amir1>	 I don't know how it broke :/ marostegui knows better
[10:09:31] <marostegui>	 Amir1: ^ is the alter done everywhere?
[10:09:45] <Amir1>	 which alter?
[10:09:59] <milimetric>	 I think he means "Amir1> started an alter table on revision table in s1 😭"
[10:10:03] <Amir1>	 the revision one is started on s1 today
[10:10:15] <milimetric>	 yeah, but you said that'd take 24h, it's only been 2
[10:10:26] <marostegui>	 milimetric: how long does the report take?
[10:10:28] <Amir1>	 https://drift-tracker.toolforge.org/report/core/
[10:10:47] <milimetric>	 it's not a report, it sqoops all rows from all big tables from all wikis :)  So like 3 days
[10:11:02] <marostegui>	 ah sorry, then yes, in 3 days the alter will arrive to clouddb1021 anytime
[10:11:10] <Amir1>	 milimetric: it takes a day per replica and even if it gets to the sanitarium master tomorrow (the worst case scenario), it'll take a while to reach clouddb 
[10:12:07] <milimetric>	 marostegui: ok, then we're delayed until you're done I guess.  Mind sending us a ping when things complete?
[10:12:22] <marostegui>	 Amir1:  ^ can you do that?
[10:12:37] <Amir1>	 sure thing
[10:13:47] <milimetric>	 ok, thank you both
[10:15:36] <Amir1>	 milimetric: the schema change is T298560
[10:15:36] <stashbot>	 T298560: Fix mismatching field type of revision.rev_timestamp on wmf wikis - https://phabricator.wikimedia.org/T298560
[10:18:06] <milimetric>	 thanks!
[11:16:43] <Amir1>	 marostegui: btw this will mostly go away (deploying its fix rn) https://logstash.wikimedia.org/goto/04366c1cdee8974356f82c96e50abdaa
[11:17:13] <marostegui>	 Ah good
[15:51:21] <RhinosF1>	 Is https://sqlconjuror.com/mariadb-setup-gtid-replication-using-mariabackup/ a sensible guide for setting up a new replica based on existing MariaDB install or is there anything I should be aware of?
[21:03:04] <milimetric>	 Amir1: I made https://phabricator.wikimedia.org/T309738 to follow up on our discussion
[21:11:48] <Amir1>	 Thank you so much milimetric !
[21:29:08] <RhinosF1>	 Amir1: do you know where bin logs are on a default mysql install?
[21:29:39] <Amir1>	 On /srv/sqldata?
[21:30:03] <Amir1>	 Usually/srv is the best place to start looking 
[21:30:09] <RhinosF1>	 Amir1: it has put normal tables in /var/lib/mysql
[21:37:02] <RhinosF1>	 Amir1: interesting question, we've granted Phabricator the replication client privilege but it never notices that it has it. do you know why it might?
[22:32:33] <RhinosF1>	 Amir1: i have it working, i haven't broken too much and cried yet about mariadb