[05:51:34] let me know when you are done with switchover tasks so I can show you something [05:52:34] wildo [05:54:32] Amir1: https://phabricator.wikimedia.org/T312028#8050255 [05:54:49] marostegui: thanks. I will [06:35:35] Amir1: I am repooling db1132 with ps enabled [06:35:40] I will let you know when done [06:36:07] Thanks [07:03:17] Going to switchover s3 codfw sanitarium, so I will stop s3 codfw master [08:17:14] jynus: what's up! [08:17:33] let me pm you [08:18:12] sure [09:47:29] Amir1: you can go for db1132, it is fully repooled [09:48:16] Thanks. I'm afk rn will do it soon [09:48:21] no problems [09:48:26] once you are done, please leave it depooled [09:48:39] Sure [12:01:16] (PrometheusMysqldExporterFailed) firing: Prometheus-mysqld-exporter failed (db2159:9104) - https://grafana.wikimedia.org/d/000000278/mysql-aggregated - https://alerts.wikimedia.org/?q=alertname%3DPrometheusMysqldExporterFailed [12:01:26] ^me [12:51:16] (PrometheusMysqldExporterFailed) resolved: Prometheus-mysqld-exporter failed (db2159:9104) - https://grafana.wikimedia.org/d/000000278/mysql-aggregated - https://alerts.wikimedia.org/?q=alertname%3DPrometheusMysqldExporterFailed [14:39:10] PROBLEM - MariaDB sustained replica lag on m1 on db2078 is CRITICAL: 9.8 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2078&var-port=13321 [14:39:23] this time is not me [14:39:52] probably etherpad based on traffic [14:40:38] or bacula, that gets naughty sometimes on the first week of the month [14:41:56] RECOVERY - MariaDB sustained replica lag on m1 on db2078 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2078&var-port=13321