[03:11:16] <jinxer-wm>	 (PrometheusMysqldExporterFailed) firing: Prometheus-mysqld-exporter failed (es1022:9104)  - https://grafana.wikimedia.org/d/000000278/mysql-aggregated - https://alerts.wikimedia.org
[07:11:31] <jinxer-wm>	 (PrometheusMysqldExporterFailed) firing: Prometheus-mysqld-exporter failed (es1022:9104)  - https://grafana.wikimedia.org/d/000000278/mysql-aggregated - https://alerts.wikimedia.org
[07:18:03] <marostegui>	 let's see
[07:21:16] <jinxer-wm>	 (PrometheusMysqldExporterFailed) resolved: Prometheus-mysqld-exporter failed (es1022:9104)  - https://grafana.wikimedia.org/d/000000278/mysql-aggregated - https://alerts.wikimedia.org
[08:17:20] <Emperor>	 Happy 2022 folks :) What a lot of email to catch up on...
[09:11:51] <Amir1>	 Emperor: welcome back and happy new year
[10:05:16] <Amir1>	 how does this look? https://dbtree.wikimedia.org/
[10:06:12] <taavi>	 the tendril-legacy link does not work
[10:07:01] <Amir1>	 taavi: yup, that will go live soon-ish
[10:07:39] <taavi>	 I mean literally that the link does not work, not that its destination does not work
[10:09:13] <Amir1>	 aaah, okay, I make a patch for that
[10:09:45] <Emperor>	 rebuilding the replacement drive per T297933
[10:09:46] <stashbot>	 T297933: ms-be2065 failed drive sdq - https://phabricator.wikimedia.org/T297933
[10:17:44] <Amir1>	 fixed and merged
[11:41:37] <RhinosF1>	 Amir1: https://noc.wikimedia.org/db.php still talks about entrée
[11:42:19] <Amir1>	 I'll make a patch 
[11:42:29] <Amir1>	 https://usercontent.irccloud-cdn.com/file/8CTjl260/1641296541509.jpeg
[11:42:53] <Amir1>	 It's responsive 🤪
[11:45:00] <RhinosF1>	 It is
[11:45:21] * RhinosF1 is mobile because too lazy to switch to personal MacBook login and should be doing work
[12:07:10] <godog>	 Emperor: thank you re: T297933
[12:07:11] <stashbot>	 T297933: ms-be2065 failed drive sdq - https://phabricator.wikimedia.org/T297933
[12:12:20] <Emperor>	 YW, it was more diverting that catching up on email :)
[15:59:48] <icinga-wm>	 PROBLEM - MariaDB sustained replica lag on m1 on db1117 is CRITICAL: 42.2 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1117&var-port=13321
[16:02:04] <icinga-wm>	 RECOVERY - MariaDB sustained replica lag on m1 on db1117 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1117&var-port=13321