[08:53:20] <federico3>	 db2157 errored during a schema change (without making changes), I'm repooling the host for now and wait for Amir1 to look into it
[10:05:36] <marostegui>	 federico3: you may want to check moritzm's comment from yesterday about cumin1003 and check if it will affect you
[10:08:16] <federico3>	 @moritzm on my side I can move to a different cumin host, just let us know when you plan to reboot it 
[10:21:36] <moritzm>	 ok! we're aiming for next Tues at 9 UTC; I'll send a mail to sre-at-large later
[12:06:36] <marostegui>	 clouddb1019 is down, I am checking with dhinus 
[12:08:25] <dhinus>	 I didn't touch it...
[12:08:34] <marostegui>	 so it went down on its own
[12:08:46] <marostegui>	 creating a task
[12:08:59] <dhinus>	 thx
[12:09:17] * marostegui https://phabricator.wikimedia.org/T422813
[12:11:18] <marostegui>	 HW related it seems: https://phabricator.wikimedia.org/T422813#11804002
[12:11:40] <dhinus>	 I'll make sure it's depooled
[12:11:56] <dhinus>	 though haproxy should have detected it as down
[12:13:49] <marostegui>	 Thanks
[12:13:50] <dhinus>	 manually set as depooled
[12:14:41] <marostegui>	 Thank you!
[12:16:09] <dhinus>	 I'm gonna raise the query timeout on clouddb1015, as per https://wikitech.wikimedia.org/wiki/Portal:Data_Services/Admin/Runbooks/Depool_wikireplicas#web_vs_analytics_considerations
[12:16:28] <marostegui>	 Cool thank you
[12:16:33] <marostegui>	 I pinged ops-eqiad to double check
[12:17:13] <dhinus>	 thanks, let's see what they say. I'll keep an eye on clouddb1015 to make sure it can handle all the traffic to s4/s6
[12:17:25] <dhinus>	 it did handle it in the past, so we should be fine
[12:17:31] <marostegui>	 It should be okay
[12:18:43] <dhinus>	 side note: this will block my task of checking the grants on ALL clouddbs :P
[12:18:54] <marostegui>	 haha
[12:19:30] <marostegui>	 I hope the new hw arrives soon once they start having these issues, they could be frequent :(
[12:27:26] <federico3>	 @marostegui meanwhile can I start the rolling restarts for PC and MS with the new rolling_restart_pc_ms.py?
[12:27:38] <marostegui>	 yep, works for me
[13:41:32] <marostegui>	 dhinus: https://phabricator.wikimedia.org/T422813 let's reimage clouddb1019 to debian trixie instead of 1015, they both belong to the same sections, so that's fine
[13:41:56] <dhinus>	 marostegui: ok!
[13:42:32] <marostegui>	 dhinus: I will take care of that
[13:43:08] <dhinus>	 dhinus: ack
[13:43:13] <dhinus>	 marostegui: ack ;)
[14:33:11] <Amir1>	 any maint happening on databases? I want to deploy this https://gerrit.wikimedia.org/r/c/operations/puppet/+/1265374 which should be noop but need to disable puppet and co just in case
[14:53:35] <Amir1>	 I have disabled puppet on all dbs
[14:53:40] <Amir1>	 ladsgroup@cumin1003:~$ sudo cumin 'A:db-all' 'disable-puppet "merging gerrit:1265374"'
[14:56:05] <Amir1>	 testing on db2156
[15:01:40] <Amir1>	 rolling to everywhere. Please let me know if there is issues
[15:02:26] <Amir1>	 moritzm: marostegui ^
[15:02:41] <marostegui>	 Thanks!
[15:06:54] <moritzm>	 great!
[15:07:28] <moritzm>	 there's a second patch still needed, then we can aim at migrating individual DB hosts
[15:07:40] <moritzm>	 I'll look into it next week, I'm off tomorrow for on-call comp
[15:09:03] <jynus>	 moritzm: is that for ferm -> nftables ?