[07:18:58] <federico3>	 marostegui: can I start the schema change in s8 codfw?
[07:19:14] <marostegui>	 federico3: yes
[07:20:00] <federico3>	 also s2 in eqiad?
[07:21:00] <marostegui>	 yes
[07:21:59] <federico3>	 thanks, running on s2 eqiad
[10:34:48] <icinga-wm>	 PROBLEM - MariaDB sustained replica lag on s7 on db2220 is CRITICAL: 15 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2220&var-port=9104
[10:35:50] <icinga-wm>	 RECOVERY - MariaDB sustained replica lag on s7 on db2220 is OK: (C)10 ge (W)5 ge 2.4 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2220&var-port=9104
[10:36:01] <jynus>	 ?
[10:36:04] <icinga-wm>	 PROBLEM - MariaDB sustained replica lag on s7 on db2222 is CRITICAL: 19 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2222&var-port=9104
[10:37:04] <icinga-wm>	 RECOVERY - MariaDB sustained replica lag on s7 on db2222 is OK: (C)10 ge (W)5 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2222&var-port=9104
[12:04:09] <federico3>	 Amir1: FYI I have a WIP implementation of https://phabricator.wikimedia.org/T299441 "Avoid depooling hosts if the schema change has been applied before"  https://gitlab.wikimedia.org/repos/data_persistence/dbtools/auto_schema/-/merge_requests/14
[12:06:27] <Amir1>	 thanks. I take a look soon.
[12:07:41] <federico3>	 Amir1: there's a bunch of MRs stacked on each other. Most are only adding testing and type hints without any logic change. I suggest we review them in our 1:1 today
[12:07:59] <Amir1>	 sounds good
[15:41:01] <dhinus>	 o/ I'm trying to figure out which partman recipe we need for the new clouddb hosts T393733
[15:41:01] <stashbot>	 T393733: Q4:rack/setup/install clouddb102[2-5] - https://phabricator.wikimedia.org/T393733
[15:41:57] <dhinus>	 they're supermicro servers similar (but not identical) to db2243 and db1257
[15:51:00] <dhinus>	 it looks like partman/custom/db.cfg worked for db2243 https://gerrit.wikimedia.org/r/c/operations/puppet/+/1105832
[15:51:07] <dhinus>	 I'll try using the same
[16:22:42] <dhinus>	 https://gerrit.wikimedia.org/r/c/operations/puppet/+/1173974