[07:07:40] <marostegui>	 Going to switch s2 codfw master
[07:16:45] <arnaudb>	 marostegui: can you let me know when you're done playing with it please
[07:16:49] <arnaudb>	 ?
[07:16:53] <marostegui>	 will do!
[07:16:58] <arnaudb>	 <3
[07:28:47] <marostegui>	 arnaudb: the switchover is done, I am running my schema change, will let you know when you can take the host
[07:28:54] <arnaudb>	 great thanks!
[08:38:05] <jynus>	 for some reason, yesterday's dump of s8 took 10 hours, so I am going to rerun it again (normally it takes 4)
[09:25:25] <jinxer-wm>	 FIRING: SystemdUnitFailed: ifup@eno5.service on ms-be2053:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[09:35:25] <jinxer-wm>	 RESOLVED: SystemdUnitFailed: ifup@eno5.service on ms-be2053:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[09:35:56] <marostegui>	 I am going to switch x2 master
[09:35:58] <marostegui>	 wish me luck
[09:39:58] <arnaudb>	 😱
[09:54:03] <Amir1>	 I abandoned lots of automatically generated gerrit patches for switchovers that were not merged (and probably were not needed anymore) if I abandoned anything by mistake, please restore
[09:55:08] <marostegui>	 Amir1: were tasks generated as well?
[09:55:31] <Amir1>	 yeah https://gerrit.wikimedia.org/r/q/topic:%22lsc%22
[09:55:49] <Amir1>	 e.g. https://gerrit.wikimedia.org/r/c/operations/puppet/+/1036601 which the task is now declined
[09:56:14] <marostegui>	 Amir1: Ah maybe, it got autothrottled by wikibugs? I didn't see them in our feed channel
[09:56:24] <marostegui>	 Ah but they are in my mail
[09:56:25] <marostegui>	 right
[10:01:28] <marostegui>	 arnaudb: the old s2 codfw master can be used. It is _pooled_ 
[10:02:52] <Amir1>	 I go to old codfw s1 master
[10:03:09] <arnaudb>	 thanks marostegui, will go for it 
[10:13:12] <marostegui>	 I have reimaged all the core bullseye hosts that were reported at https://phabricator.wikimedia.org/T366556 
[10:13:22] <arnaudb>	 I saw that, thanks!
[10:15:21] <arnaudb>	 db2204 was the one requiring the change, I guess I've missed a previous switchover 🤔
[10:16:26] <marostegui>	 Probably 
[10:16:46] <arnaudb>	 well, I'll have to swap it again 
[10:19:00] <Amir1>	 marostegui: I can't see T366684 I think the space should stay as S1
[10:19:13] <marostegui>	 I just changed it
[10:19:29] <Amir1>	 arnaudb: if you're not touching db2204, I have schema change there, can I? :D
[10:19:42] <Amir1>	 Thanks marostegui, do you want me to run my script?
[10:19:53] <marostegui>	 Amir1: which script?
[10:20:08] <Amir1>	 oh, it's only bullseye, nvm then
[10:20:17] <arnaudb>	 Amir1: please go ahead, I did not wanted to perform a switchmaster right now as I'm gonna eat lunch :D
[10:23:44] <Amir1>	 started, it's going to take a couple of hours :D
[12:50:27] <arnaudb>	 fyi jclarck-ctr will go for T363119 → host has been depooled and downtimed
[12:50:27] <stashbot>	 T363119: db1246 crashed - https://phabricator.wikimedia.org/T363119
[17:30:11] <swfrench-wmf>	 FYI, some time this or next week, I'll deploy a new conftool for T365123. flagging that here, since the main change in behavior is that dbctl will now validate changes to "external" sections just like it does for "regular" (i.e., each section has exactly one master and replica count satisfies min_replicas).
[17:30:11] <swfrench-wmf>	 I've verified that all external-flavored sections should validate.
[17:30:11] <swfrench-wmf>	 in any case, let me know if you have any questions / concerns :)
[17:30:11] <stashbot>	 T365123: Make dbctl check for depooled future masters  - https://phabricator.wikimedia.org/T365123
[23:36:33] <Amir1>	 > s8 	eqiad 	snapshot 	wrong_size 	19 hours ago 	1.4 TB 	-9.0 % 	The previous backup had a size of 1.6 TB, a change larger than 5.0%. 
[23:36:51] <Amir1>	 pagelinks drop