[00:04:32] <jinxer-wm>	 (MysqlReplicationLag) firing: (3) MySQL instance db1154:13311 has too large replication lag (20h 1m 56s) - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica  - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLag
[00:09:32] <jinxer-wm>	 (MysqlReplicationLag) firing: (3) MySQL instance db1154:13311 has too large replication lag (18h 38m 55s) - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica  - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLag
[00:10:55] <icinga-wm>	 RECOVERY - MariaDB sustained replica lag on s8 on db1154 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1154&var-port=13318
[00:13:35] <icinga-wm>	 RECOVERY - MariaDB sustained replica lag on s3 on db1154 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1154&var-port=13313
[01:39:32] <jinxer-wm>	 (MysqlReplicationLag) resolved: MySQL instance db1154:13311 has too large replication lag (6m 35s) - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica - https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&refresh=1m&var-job=All&var-server=db1154&var-port=13311 - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLag
[01:41:42] <icinga-wm>	 RECOVERY - MariaDB sustained replica lag on s1 on db1154 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1154&var-port=13311
[08:22:43] <Amir1>	 marostegui: btw, the schema change (T318605) I'm about to start on externallinks table adds some new columns and such (nbd) but it also removes an extra field from one of the indexes. This shouldn't change the read patterns but mariadb is full of surprises so if you see slow queries show up, let me know
[08:22:44] <stashbot>	 T318605: Deploy new externallinks fields to production - https://phabricator.wikimedia.org/T318605
[08:23:14] <Amir1>	 ./2022/add_cuc_user_ip_time_index_T321123.py:6:19: W291 trailing whitespace :P
[09:13:30] <Emperor>	 ugh, ms-be1059's permissions are wrong - part of it is 902:902 and part 130:130
[10:17:11] <marostegui>	 I am going to failover es1, es2 and es3 masters (should be a noop)
[10:17:55] <marostegui>	 ^ for codfw
[10:25:38] <jynus>	 no blocker on my side