[09:16:11] Can I get a review https://gerrit.wikimedia.org/r/c/operations/puppet/+/812828? [09:16:24] checking [09:16:28] <3 [09:19:48] checked all dns and ip [09:19:55] thank you! [09:19:59] not sure if there will be more keys that need update on hiera? [09:20:08] maybe on a separate patch I guess [09:20:15] Yeah, I need to do some more stuff [09:20:20] But I will reload the proxies first [09:20:20] ok then [09:23:35] careful, see puppet alerting issue, you may want to delay deployment [09:23:45] I guess too late [09:24:09] It went fine on my changes [09:24:57] yeah, but icinga won't probably show the changes until that is fixed [10:21:01] PROBLEM - Check unit status of swift_ring_manager on ms-fe1009 is CRITICAL: CRITICAL: Status of the systemd unit swift_ring_manager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [10:22:40] (I am mostly doing inspiration week projecty stuff this week - may be best to ping me explicitly if you want me to look at something (which I am very happy to do! just not reading IRC as much as normal)) [11:13:07] RECOVERY - Check unit status of swift_ring_manager on ms-fe1009 is OK: OK: Status of the systemd unit swift_ring_manager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [12:01:16] (PrometheusMysqldExporterFailed) firing: Prometheus-mysqld-exporter failed (db2163:9104) - https://grafana.wikimedia.org/d/000000278/mysql-aggregated - https://alerts.wikimedia.org/?q=alertname%3DPrometheusMysqldExporterFailed [12:01:21] ^ me [12:51:16] (PrometheusMysqldExporterFailed) resolved: Prometheus-mysqld-exporter failed (db2163:9104) - https://grafana.wikimedia.org/d/000000278/mysql-aggregated - https://alerts.wikimedia.org/?q=alertname%3DPrometheusMysqldExporterFailed [20:25:26] PROBLEM - MariaDB sustained replica lag on x2 on db2142 is CRITICAL: 19.8 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2142&var-port=9104 [20:26:26] PROBLEM - MariaDB sustained replica lag on x2 on db2143 is CRITICAL: 3.6 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2143&var-port=9104 [20:29:04] PROBLEM - MariaDB sustained replica lag on x2 on db2144 is CRITICAL: 19.6 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2144&var-port=9104 [20:34:14] RECOVERY - MariaDB sustained replica lag on x2 on db2144 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2144&var-port=9104 [20:36:44] RECOVERY - MariaDB sustained replica lag on x2 on db2143 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2143&var-port=9104 [20:47:00] PROBLEM - MariaDB sustained replica lag on x2 on db2144 is CRITICAL: 21.6 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2144&var-port=9104 [20:49:30] PROBLEM - MariaDB sustained replica lag on x2 on db2143 is CRITICAL: 26.6 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2143&var-port=9104 [20:52:08] RECOVERY - MariaDB sustained replica lag on x2 on db2144 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2144&var-port=9104 [21:02:16] RECOVERY - MariaDB sustained replica lag on x2 on db2143 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2143&var-port=9104 [21:07:28] PROBLEM - MariaDB sustained replica lag on x2 on db2144 is CRITICAL: 94 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2144&var-port=9104 [21:27:54] PROBLEM - MariaDB sustained replica lag on x2 on db2143 is CRITICAL: 74.2 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2143&var-port=9104 [21:53:28] RECOVERY - MariaDB sustained replica lag on x2 on db2143 is OK: (C)2 ge (W)1 ge 0.2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2143&var-port=9104 [21:56:08] RECOVERY - MariaDB sustained replica lag on x2 on db2144 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2144&var-port=9104 [21:57:32] RECOVERY - MariaDB sustained replica lag on x2 on db2142 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2142&var-port=9104