[16:00:51] <volans>	 Emperor: there are some alerts on icinga for moss-be200[1-2], if known/WIP should those be downtimed/silenced for the time being?
[16:06:40] <volans>	 is dbproxy2004 active/in production? It has notifications disabled on icinga
[16:06:48] <volans>	 via hiera
[16:08:13] <volans>	 (also an outstanding alert: check_failover servers up 2 down 1: mariadb,db2135 )
[16:10:23] <Emperor>	 volans: yeah, they're still in the "playground for testing the convert_disks cookbook" which I need to get back to.
[16:10:55] <Emperor>	 date
[16:10:58] <Emperor>	 oops
[16:11:08] <volans>	 ack, thx
[16:11:19] <Emperor>	 I'll downtime them 'til next year
[16:11:41] <volans>	 SGTM, thx
[16:12:53] <Emperor>	 {done}
[16:19:29] <marostegui>	 volans: those are in misc, so not used
[16:21:32] <volans>	 marostegui: ack, I was unsure as it's in the DNS: m5-master       5M  IN CNAME    dbproxy2004
[16:21:43] <marostegui>	 volans: yeah, but misc in codfw isn't used
[16:21:50] <volans>	 got it :D
[16:21:53] <marostegui>	 we'll take care of the alert anyways, thanks for letting us know
[16:22:13] <volans>	 thanks <3
[23:33:15] <icinga-wm>	 PROBLEM - MariaDB sustained replica lag on s4 on db1221 is CRITICAL: 25 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1221&var-port=9104
[23:34:21] <icinga-wm>	 PROBLEM - MariaDB sustained replica lag on s4 on db1248 is CRITICAL: 2.6 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1248&var-port=9104
[23:36:07] <icinga-wm>	 PROBLEM - MariaDB sustained replica lag on s4 on db1155 is CRITICAL: 17.6 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1155&var-port=13314
[23:50:35] <icinga-wm>	 RECOVERY - MariaDB sustained replica lag on s4 on db1248 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1248&var-port=9104
[23:52:27] <icinga-wm>	 RECOVERY - MariaDB sustained replica lag on s4 on db1221 is OK: (C)2 ge (W)1 ge 0.2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1221&var-port=9104
[23:53:51] <icinga-wm>	 RECOVERY - MariaDB sustained replica lag on s4 on db1155 is OK: (C)2 ge (W)1 ge 0.2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1155&var-port=13314