[00:14:55] <jinxer-wm>	 (MysqlReplicationLag) firing: MySQL instance db1202:9104 has too large replication lag (1h 15m 50s) - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica - https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&refresh=1m&var-job=All&var-server=db1202&var-port=9104 - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLag
[00:23:49] <jinxer-wm>	 (MysqlReplicationLag) resolved: MySQL instance db1202:9104 has too large replication lag (7m 39s) - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica - https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&refresh=1m&var-job=All&var-server=db1202&var-port=9104 - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLag
[05:20:35] <Amir1>	 https://www.irccloud.com/pastebin/h44bCDVE/
[05:20:44] <Amir1>	 the diff, is this okay?
[05:51:21] <marostegui>	 should be fine
[05:53:53] <Amir1>	 cool
[09:03:47] <godog>	 Emperor: what do you think re: https://gerrit.wikimedia.org/r/c/operations/puppet/+/848349 ?
[09:07:09] * Emperor looks up from endless rebooting
[09:14:25] <Emperor>	 [commented on the CR]
[09:15:20] <godog>	 thanks! will look shortly
[09:30:16] <jynus>	 es backups are running, if you don't mind waiting 24 h or so, they should be finished by then before rebooting them
[09:44:20] <marostegui>	 Amir1:  I am going to start rebooting dbproxy hosts
[10:17:28] <Amir1>	 marostegui: thanks <3
[11:42:27] <marostegui>	 Amir1:  so you'll take care of restarting mariadb across all sanitarium hosts for https://gerrit.wikimedia.org/r/c/operations/puppet/+/831542 ?
[11:42:45] <marostegui>	 remember it also needs restarting on codfw sanitarium hosts (db2094 and db2095)
[11:43:05] <Amir1>	 I think so because we need to reboot them for kernel anyway
[11:43:12] <marostegui>	 excellent
[11:43:38] <marostegui>	 But we shouldn't allow table creations until the restart is done
[11:43:44] <marostegui>	 (and we have checked the replication filter is in place)
[13:02:16] <godog>	 Emperor: unless you have (any?) objections I'll go ahead with https://gerrit.wikimedia.org/r/c/operations/puppet/+/848349
[13:19:09] <Emperor>	 godog: placet (and I stuck a +1 there for the record)
[13:19:43] <godog>	 Emperor: cheers!
[13:55:28] <Emperor>	 https://aastatus.net/42480 was where I went
[15:50:04] <Reedy>	 ooh, another a&a user? :)
[15:50:22] <Emperor>	 indeed so