[05:36:16] <marostegui>	 jynus: db2097 migrated to 10.6
[07:27:09] <jynus>	 ok, will run a backup to test it
[07:29:45] <marostegui>	 great
[09:54:04] <jynus>	 marostegui: any suggestion on https://phabricator.wikimedia.org/T334650#9050440 ? I can keep s6 backup sources eqiad on 10.4 (and so, its backups) or upgrade them already?
[10:01:20] <jynus>	 As I don't think I am a blocker, and it seems it may still take a few more days for a 100% migration, I will leave eqiad backups on 10.4 for now, but ready to migrate- as it seems the most conservative action
[10:02:24] <jynus>	 so doing https://phabricator.wikimedia.org/T334650#9047479 but just short of the actual upgrade
[10:45:12] <marostegui>	 yeah I'm going to migrate the candidate on Monday
[10:45:30] <marostegui>	 and give it 48h before doing the master switch 
[10:50:38] <jynus>	 thats for codfw, right?
[10:50:54] <jynus>	 or you mean eqiad?
[11:04:15] <marostegui>	 yeah codfw 
[11:05:05] <marostegui>	 just replied in phab for the equiad comment 
[11:06:10] <jynus>	 thank you, will be ready by then
[12:11:06] <jynus>	 another heads up for Amir1 that a new instance of s6 has appeared (db1225:s6) https://grafana.wikimedia.org/goto/ePgwqA34k?orgId=1
[12:12:03] <Amir1>	 new instances everywhere
[12:12:20] <jynus>	 technically, moved, but this is a 1 time thing because upgrade
[12:12:55] <marostegui>	 jynus: just to check, you are updating zarcillo with all these changes or you want me to?
[12:13:04] <jynus>	 I am doing it
[12:13:12] <marostegui>	 ah thanks <3
[12:13:16] <jynus>	 otherwise prometheus wouldn't work :-D
[12:13:20] <jynus>	 as it depends on that
[12:13:28] <marostegui>	 but if the host already exists it would
[12:13:32] <marostegui>	 Ah, but it is a different port, yeah
[12:13:44] <jynus>	 what I may not be doing is handling orch, I am not too familiar with that
[12:13:52] <jynus>	 altough I think I was told it was not a big deal?
[12:14:45] <marostegui>	 that's fine yeah
[12:14:51] <marostegui>	 It would delete itself after a few days
[12:14:54] <marostegui>	 If I don't do it first
[12:15:00] <marostegui>	 And it will show up by itself too, so that's fine
[12:15:18] <Amir1>	 do you remember what major db change happened in august/july/sep 2020?
[12:15:31] <marostegui>	 I don't even remember what I did 2 weeks ago
[12:15:49] <Amir1>	 https://usercontent.irccloud-cdn.com/file/YW7H3Ah6/grafik.png
[12:15:52] <Amir1>	 haha
[12:15:53] <jynus>	 Amir1: unsure but I would guess the first normalization
[12:16:21] <marostegui>	 MCR maybe?
[12:16:49] <Amir1>	 there is a small drop before that for MCR in archive table
[12:17:10] <jynus>	 Amir1: the stats contain per-file stats
[12:17:21] <jynus>	 so you may be able to get more info
[12:17:26] <jynus>	 there is an SQL query for that
[12:17:46] <Amir1>	 aah, I actually did that for s4
[12:17:53] <jynus>	 https://phabricator.wikimedia.org/P49455
[12:17:54] <Amir1>	 because it's growing wayyy tooo fast
[12:18:24] <jynus>	 just change the jobids there
[12:18:49] <Amir1>	 https://usercontent.irccloud-cdn.com/file/IaRrYeWv/table.png
[12:19:04] <Amir1>	 this is for commons
[12:19:48] <Amir1>	 I can make it run for enwiki
[12:20:04] <jynus>	 you can also check the differences between dump and the snapshots to avoid index or optimization differences
[12:20:36] <Amir1>	 yeah I have dumps too but they also don't take account structural changes that were important (dropping index, etc.)
[12:21:03] <jynus>	 yeah, plus they are compressed, which hides some of the redundancies
[12:21:45] <Amir1>	 after this graph for s4, I'll go talk to the community to cut off some of the templatelinks and categorylinks growth, I think there is a bit of an issue there. e.g. stuff like https://commons.wikimedia.org/wiki/Category:M%C3%A9rim%C3%A9e_ID_same_as_Wikidata
[12:22:54] <Amir1>	 jynus: funnily enough, I think backup_files needs normalizing of wiki id and file name, it's 32GB, queries take a looong time :P
[12:23:09] <jynus>	 ha ha ha
[12:23:19] <jynus>	 in theory it should be trimmed after 3 months
[12:23:26] <jynus>	 good that I didn't
[12:23:46] <Amir1>	 Yup, they are mountain of useful information
[12:23:59] <jynus>	 also the idea of backup_objects was to create some automation to aggregate per object (e.g. table, trigger, db) but was never done
[12:24:44] <jynus>	 I think adding indexes should be enough, but certainly the db hasn't been optimized for analytics (yet)
[12:25:48] <jynus>	 the other extension was to run the same scripts over a live db to compare and check backups more thoroughly
[12:26:50] * Emperor watches their laptop try and examine 2 billion thumbnail records
[12:27:10] <Amir1>	 xD
[12:27:14] <Amir1>	 big data
[12:29:50] <Amir1>	 https://usercontent.irccloud-cdn.com/file/lWpOUK4m/table.png
[12:29:52] <Amir1>	 for enwiki
[12:29:58] <Amir1>	 looks like contemporary art
[12:31:17] <Amir1>	 that looks like MCR on revision
[12:35:48] <Amir1>	 found it T238966
[12:35:49] <stashbot>	 T238966: Apply updates for MCR, actor migration, and content migration, to production wikis. - https://phabricator.wikimedia.org/T238966
[13:05:29] <jynus>	 MariaDB 10.6 backups working well, x1 and s6 worked well
[13:05:44] <Amir1>	 🎉
[13:21:51] <hashar>	 Amir1: +1 on those graphs being art :)
[15:43:46] <Amir1>	 who thought databases could become a source of art
[15:43:51] <Amir1>	 you learn a new thing every day
[16:02:02] <Emperor>	 :)