[01:09:18] <icinga-wm_>	 PROBLEM - MariaDB sustained replica lag on m1 on db1217 is CRITICAL: 4.2 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1217&var-port=13321
[01:09:30] <icinga-wm_>	 PROBLEM - MariaDB sustained replica lag on m1 on db2160 is CRITICAL: 11.4 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2160&var-port=13321
[01:10:52] <icinga-wm_>	 RECOVERY - MariaDB sustained replica lag on m1 on db1217 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1217&var-port=13321
[01:12:38] <icinga-wm_>	 RECOVERY - MariaDB sustained replica lag on m1 on db2160 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2160&var-port=13321
[05:07:43] <marostegui>	 Reminder: NO MORE DB maintenance after 9 AM UTC 
[05:07:46] <marostegui>	 Amir1: ^
[08:50:57] <Amir1>	 good morning, yeah, I finished abuse filter exactly because of that
[08:55:21] <marostegui>	 great
[09:24:07] <Amir1>	 marostegui: https://gerrit.wikimedia.org/r/c/operations/software/schema-changes/+/910428 what do you think of this?
[09:24:27] <marostegui>	 ooooh
[09:24:43] <marostegui>	 that is going to be helpful indeed
[09:25:06] <marostegui>	 I like this https://gerrit.wikimedia.org/r/c/operations/software/schema-changes/+/910428/1/2023/change_cul_user_T326011.py
[09:26:15] <marostegui>	 It would be nice to add examples on code for more complex things like: https://gerrit.wikimedia.org/r/c/operations/software/schema-changes/+/910428/1/2023/fix_creditssource_drifts.py
[09:26:33] <marostegui>	 Or at least with the most common cases (column data type change, add/remove defaults)
[09:26:34] <marostegui>	 etc
[09:26:36] <marostegui>	 what do you think?
[09:27:00] <Amir1>	 sure, I'm planning to add them in the wikitech page, would that be oaky?
[09:27:06] <marostegui>	 ah yeah
[09:27:12] <marostegui>	 maybe add a link to that and that's it
[09:27:34] <Amir1>	 sure
[09:47:21] <marostegui>	 while reviewing all the stuff for the switchover I just found this heh: https://gerrit.wikimedia.org/r/c/operations/puppet/+/910435/
[09:49:11] <Amir1>	 ouch 
[09:49:24] <marostegui>	 it is not a big deal really, but just fixed it
[09:50:05] <marostegui>	 pt-heartbeat runs anyways
[10:00:57] <jynus>	 that is a good example of why I fought to leave all hosts rw!
[10:01:07] <jynus>	 back in the day
[10:01:31] <Amir1>	 https://wikitech.wikimedia.org/wiki/Auto_schema#Check_function
[10:01:37] <Amir1>	 Added some info
[10:01:53] <Amir1>	 will add more soon
[10:44:28] <Amir1>	 https://wikitech.wikimedia.org/wiki/Auto_schema/examples
[10:55:25] <marostegui>	 excellent
[10:55:26] <marostegui>	 thank you
[10:55:28] <marostegui>	 that's useful