[04:57:50] <marostegui>	 Amir1: can you double check if db1143 (the s4 host that broke a week ago) has all templatelink schema changes you did or anything else you've ran un the last week?
[05:05:32] <Amir1>	 marostegui: morning. Sure
[05:06:36] <marostegui>	 Amir1: I have moved it from the old master to the new one
[05:06:45] <Amir1>	 thanks
[05:07:01] <Amir1>	 do you know why this s3 lag errors are showing up^?
[05:07:01] <marostegui>	 Amir1: I don't think I will be able to be present during the switchover. I will have my phone and my laptop next to me though
[05:07:29] <Amir1>	 marostegui: thanks. The major part is the topology change, the actual switchover is quite straightforward
[05:07:30] <Amir1>	 :D
[05:07:32] <marostegui>	 Amir1: I was just checking: https://logstash.wikimedia.org/goto/5834b2f509fbd30b3ed6a99b5274b8b8
[05:14:40] <Amir1>	 it has the templatelinks schema change
[05:14:50] <marostegui>	 cool!
[05:15:08] <Amir1>	 btw, templatelinks drop in s4 is started, 220GB dropped https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=dbstore1007&var-datasource=thanos&var-cluster=wmcs&from=now-24h&to=now&viewPanel=28
[05:15:20] <marostegui>	 oh excellent!
[05:15:29] <marostegui>	 jynus: ^ that might alert in backups' size
[06:10:37] <marostegui>	 All our proxies have been rebooted
[06:27:15] <Amir1>	 created next week's switchovers
[06:27:22] <Amir1>	 (tickets)
[07:49:30] <jynus>	 not sure why, but es backups are failing short after starting :-(
[07:51:13] <jynus>	 the hosts haven't been recently restarted or upgraded
[08:27:10] <jynus>	 I will be doing a rolling restart of dbprovs in aproximatelly 1 h (when m2 backup finishes), unless you have any ongoing or planned recovery
[08:39:18] <marostegui>	 Amir1: you can run the cx_corpora schema change on db2096 (old x1 master)
[08:39:36] <Amir1>	 awesome on it
[08:39:43] <marostegui>	 Amir1: and reboot it too
[08:40:52] <Amir1>	 s8ure
[09:59:26] <marostegui>	 I have switched over pc1 master, and now reverting as I have rebooted pc1011 already
[10:00:10] <marostegui>	 Once done I will move pc1013 to pc3
[10:46:42] <Amir1>	 marostegui: db2096 has got the cx schema change, rebooted and repooled
[10:49:24] <marostegui>	 :)
[14:14:51] <jynus>	 volans: to sumarize, there is nothing I want to touch about postgres dumps (yet), but fix the bacula side of things
[14:15:24] <jynus>	 although having a bit of better error handling on the one liner won't hurt
[14:16:03] <volans>	 ack
[14:16:26] <jynus>	 if you check modules/postgresql/files/dump_all.sh I think you would agree with me :-D
[14:19:22] <volans>	 can't right now, sorry, submerged by backlog emails
[14:21:05] <jynus>	 no problem, it almost just one line that says "/usr/bin/pg_dumpall | /bin/gzip > ${path}/psql-all-dbs-${date}.sql.gz" which is easy to improve
[14:21:31] <jynus>	 will add you as reviewer if you want
[15:52:34] <Amir1>	 https://www.irccloud.com/pastebin/ibjwdNog/wikimedia-relational-databases-2022.xml
[15:53:03] <Amir1>	 loading it in draw.io gives nice results
[15:53:23] <Amir1>	 https://usercontent.irccloud-cdn.com/file/NtImAuKw/image.png
[15:53:59] <Amir1>	 I haven't figured out what to do with this many arrows, ideas are welcome