[05:20:22] <marostegui>	 Amir1: I don't remeber if you said I can do s2 codfw switchover because your finished your stuff there?
[05:26:40] <arnaudb>	 will start https://phabricator.wikimedia.org/T366259
[05:52:05] <arnaudb>	 I'm a bit ahead of schedule, waiting for 6 UTC to disabnle puppet and move to the next steps
[05:52:14] <marostegui>	 ok :)
[05:52:52] <marostegui>	 arnaudb: wait
[05:53:01] <marostegui>	 arnaudb: the topogy isn't moved entirely
[05:53:03] <marostegui>	 did you see that?
[05:53:11] <arnaudb>	 oh yes I'm on orchestrator
[05:53:18] <arnaudb>	 my message was ahead of schedule as well x)
[05:53:24] <marostegui>	 ok
[05:53:30] <arnaudb>	 waiting for the cookbook to properly finish up
[05:53:46] <marostegui>	 ah ok
[05:54:06] <arnaudb>	 I was not clear, sorry about that haha
[05:54:22] <arnaudb>	 (now its over)
[05:54:38] <marostegui>	 yep
[05:54:57] <arnaudb>	 bbiab
[06:30:34] <marostegui>	 arnaudb: once you are done with the old master, leave it depooled (but start replication please)
[06:30:44] <marostegui>	 I have a bunch of stuff to run there - I think you do too?
[06:32:36] <arnaudb>	 I do indeed
[06:32:46] <marostegui>	 ok, I will run them once you've done yours
[06:32:47] <arnaudb>	 not a bunch but a quick schema change
[06:33:11] <marostegui>	 cool, let me know when you are done with it
[06:33:17] <marostegui>	 so I can run mine
[06:33:38] <arnaudb>	 will do
[06:35:03] <marostegui>	 thanks
[07:09:00] <arnaudb>	 marostegui: its all yours
[07:09:20] <marostegui>	 arnaudb: great thank you!
[07:22:35] <Emperor>	 Morning folks. I'm still looking for a +1 for  https://gerrit.wikimedia.org/r/c/operations/puppet/+/1038391 please :)
[07:54:15] <Emperor>	 thanks arnaud.b 
[08:14:31] <jynus>	 Last snapshot for s4 at eqiad (db1150) taken on 2024-06-03 07:46:21 is 1592 GiB, but the previous one was 1693 GiB, a change of -6.0 %
[08:31:08] <marostegui>	 Probably because of https://phabricator.wikimedia.org/T364069
[09:03:29] <Amir1>	 marostegui: I am done with s2 codfw
[09:04:28] <Amir1>	 Yeah. That rebuild dropped 100GB from templatelinks. I will explain why
[09:20:02] <Amir1>	 To explain more: The community cleaned up a lot of rows from templatelinks in commons via refactoring templates: https://phabricator.wikimedia.org/T343131#9467054 (and onwards)
[09:41:27] <Amir1>	 marostegui: when you're done please let me have some fun with the old s1 masters (codfw is not switched over yet I know but once you're done
[09:43:07] <marostegui>	 Amir1: will do, probably it won't be finished today
[09:43:12] <marostegui>	 s1 codfw I am going to switch it now
[09:43:43] <Amir1>	 thanks
[11:17:37] <btullis>	 Amir1: marostegui: I just spotted the lag on clouddb1021 s4/s8 which I see you marked with downtime - It seems to be dropping very slowly. Is there anything I should know about it, or have you got it in-hand?
[11:18:34] <btullis>	 I was about to start the bookworm upgrade and then move onto the transfer from clouddb1021 to an-redacteddb1001, but I'll hold off for now.
[11:21:25] <jinxer-wm>	 FIRING: [2x] SystemdUnitFailed: swift_dispersion_stats.service on thanos-fe1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[11:29:10] <Emperor>	 le sigh
[11:31:25] <jinxer-wm>	 FIRING: [2x] SystemdUnitFailed: swift_dispersion_stats.service on thanos-fe1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[11:32:00] <Emperor>	 I literally just fixed that
[11:34:30] <Amir1>	 btullis: meeting but that's expected. Give it a bit
[11:35:00] <btullis>	 Amir1: Ack, thanks. Happy meeting.
[11:36:25] <jinxer-wm>	 RESOLVED: [2x] SystemdUnitFailed: swift_dispersion_stats.service on thanos-fe1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[11:44:57] <marostegui>	 btullis: I was in a meeting but I see amir answered
[12:00:14] <btullis>	 Yup, thanks. I'll just monitor it and not do anything on it right now. Thanks.
[13:28:39] <Amir1>	 marostegui: okay if I bump the weight to 200 for db1194 (s7 host we discussed)? Maybe in batches of 50
[13:28:50] <marostegui>	 yeah, do it in batches
[13:52:55] <Emperor>	 I'm fixing the big bugs again... https://github.com/ceph/ceph/pull/57868
[13:55:42] <urandom>	 making the world a better place, one misplaced character at a time
[13:56:08] <urandom>	 🦾
[14:38:08] <Amir1>	 do we have a clone cookbook documentation?
[14:39:04] <Amir1>	 (like a wikitech page)
[14:41:53] <marostegui>	 Amir1: https://wikitech.wikimedia.org/wiki/MariaDB/Clone_a_host in the cloning 
[14:42:01] <marostegui>	 Clone via the sre.mysql.clone cookbook in a tmux:
[14:42:02] <marostegui>	 sudo cookbook sre.mysql.clone --source $source_server --target $destination_server --primary  $cluster_replication_source
[14:42:49] <Amir1>	 awesome. Thanks.
[14:43:42] <Amir1>	 I think it needs its own page since cloning can be done for multiple reasons, not just provisioning. It can get really confusing
[14:44:43] <marostegui>	 Yeah, that mixes a bit the provisioning part (like zarcillo database addtition)
[14:44:55] <marostegui>	 But that is the only part that maybe requires a note
[14:45:00] <marostegui>	 I will add it
[14:45:04] <Amir1>	 Thanks!
[14:47:12] <Amir1>	 something that bothers me about the clone cookbook (and it's sorta my fault) is that you have put fqdn otherwise it gives a cryptic error message
[14:47:28] <Amir1>	 I'd add an example in the clone section so people would know
[14:47:28] <marostegui>	 yeah
[14:47:38] <marostegui>	 ok I will do it now too
[14:48:27] <Amir1>	 thanks <3
[14:53:58] <marostegui>	 Amir1: db1184 old s1 master is ready for you. Can you take care of repooling it once you are done?
[14:54:05] <Amir1>	 marostegui: sorry if I'm misunderstanding this: https://phabricator.wikimedia.org/T366552#9860018 but if you're done with the old s1 codfw master, I have some stuff to discuss with that host
[14:54:15] <Amir1>	 oh sure thing
[14:54:23] <marostegui>	 Amir1: can you take db1184 first?
[14:54:33] <Amir1>	 definitely 
[14:54:36] <marostegui>	 let's do that
[14:54:58] <marostegui>	 db2203 is being repooled, once you are done with db1184, repool that one, and you can proceed with db2203 if that works?
[14:55:08] <Amir1>	 sounds good and fun
[14:55:17] <marostegui>	 enjoy db1184 then
[14:55:22] <Amir1>	 I think each one will take a day or more
[14:55:42] <Amir1>	 I have pagelinks and externallinks schema changes
[14:56:23] <marostegui>	 the pagelinks thing in s4 codfw took 2 weeks XD
[14:57:08] <Amir1>	 for s8 it'd take longer, that's why I'm doing codfw and eqiad in parallel 
[14:58:07] <Amir1>	 db1171:3318 is on 13th hour and counting 😭
[14:58:17] <Amir1>	 sorry, 14
[14:58:19] <marostegui>	 I will switch s2 codfw tomorrow
[14:58:29] <Amir1>	 awesome
[15:48:55] <dhinus>	 heads up I'm depooling, stopping and rebooting clouddb1013 (T366555)
[15:58:01] <btullis>	 clouddb1021.s8 lag is still going up, not down. Still expected? Is it running an alter table or something?
[16:12:47] <dhinus>	 clouddb1013 rebooted and repooled
[17:01:36] <Amir1>	 btullis: yes. Alter table