[05:54:25] <jinxer-wm>	 FIRING: [2x] SystemdUnitFailed: mariadb.service on db2250:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[06:04:13] <jinxer-wm>	 RESOLVED: [2x] SystemdUnitFailed: mariadb.service on db2250:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[06:18:25] <marostegui>	 federico3: so we got the alert again:
[06:18:26] <marostegui>	 [06:06:54]  <jinxer-wm> FIRING: ZarcilloDataIngestionMetricAbsent: Zarcillo data ingestion metric is missing - https://wikitech.wikimedia.org/wiki/MariaDB/Zarcillo - https://grafana.wikimedia.org/d/celrzpf6av8qob/zarcillo - https://alerts.wikimedia.org/?q=alertname%3DZarcilloDataIngestionMetricAbsent
[06:18:38] <marostegui>	 We should probably either revert or try to fix it to avoid the noise
[07:01:04] <jinxer-wm>	 FIRING: MysqlPredictiveFreeDiskSpace: Host db1265:9100 predictive low disk space on /srv - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting - https://grafana.wikimedia.org/goto/Jdz2PnLNg?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DMysqlPredictiveFreeDiskSpace
[07:03:10] <marostegui>	 ^expected
[08:54:52] <marostegui>	 Amir1: https://phabricator.wikimedia.org/T433990 you can proceed with the old x1 master (db2191) which is being pooled, so if you want me to stop the pooling I can do that and then you take care of repooling it? or you can wait around 1h so the pooling finishes and then you depool+run your stuff+repool
[08:56:38] <Amir1>	 it can wait, no worries
[08:56:44] <Amir1>	 thank you for the switchover
[08:57:05] <marostegui>	 thanks!
[10:14:35] <federico3>	 @marostegui can we try deploy https://gerrit.wikimedia.org/r/c/operations/alerts/+/1352562 ? It shouldn't harm
[10:55:43] <marostegui>	 federico3: +1ed
[11:46:28] <marostegui>	 federico3: what's the status of this https://phabricator.wikimedia.org/T440260
[11:47:29] <federico3>	 the pool/depool was fixed, the clone one should be fixed but it's not tested with a real clone yet. Do we have a clone to do?
[11:47:49] <marostegui>	 as i mentioned on the ticket, can you maybe test with the testing hosts?
[11:50:49] <federico3>	 I did a test run for pool depool already, I'll reply to the task. I'll start a clone using only 2 replicas
[11:51:47] <marostegui>	 thanks
[11:52:17] <marostegui>	 Let's also try to address this before the end of the week so we can close that task too: https://gerrit.wikimedia.org/r/c/operations/cookbooks/+/1351376