[00:12:31] <icinga-wm>	 PROBLEM - MariaDB sustained replica lag on s1 on db1234 is CRITICAL: 2.2 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1234&var-port=9104
[00:13:31] <icinga-wm>	 RECOVERY - MariaDB sustained replica lag on s1 on db1234 is OK: (C)2 ge (W)1 ge 0.4 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1234&var-port=9104
[10:07:10] <Amir1>	 root@db1163:/srv/sqldata/enwiki# ls -l | grep -i text
[10:07:10] <Amir1>	 -rw-r----- 1 mysql mysql          998 Nov  3  2021 text.frm
[10:07:11] <Amir1>	 -rw-r----- 1 mysql mysql  57847840768 May  9 20:24 text.ibd
[10:07:23] <Amir1>	 i.e. text is not being written into anymore
[10:13:08] <Emperor>	 \o/
[10:18:29] <Emperor>	 Amir1: do you have any thoughts on T362749 ? My thought is roughly that this needs Doing Properly (i.e. lift wing needs suitable credentials), and that those are going to need to be quite substantial credentials if it's to access other users' stashes
[10:18:32] <stashbot>	 T362749: Deploy logo-detection model-server to LiftWing staging - https://phabricator.wikimedia.org/T362749
[10:18:59] <Amir1>	 I check
[10:19:01] <Amir1>	 Thanks@
[10:23:54] <Emperor>	 <3
[12:26:53] <elukey>	 urandom: o/
[12:27:15] <elukey>	 if/when you have a moment today, shall we drop the old cassandra ca/tls dir on puppet private?
[12:27:42] <elukey>	 I'd need your supervision before committing to puppet private, a review would be nice :D
[12:28:07] <elukey>	 we can disable puppet on all cassandra nodes to be sure, then we slowly reenable to catch issues
[12:38:06] * Emperor wonders if that's the sort of production change we would typically avoid on a Friday?
[12:54:57] <dhinus>	 o/ I would like to deploy this wikireplica view change: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1029709
[12:55:29] <dhinus>	 is it fine to do it today, or should I avoid doing it on a Friday?
[12:56:33] <Amir1>	 dhinus: let's do it on Monday
[12:56:35] <Amir1>	 Manuel is out
[12:59:25] <dhinus>	 ack
[13:16:14] <urandom>	 elukey: maybe we should wait until Monday :)
[13:18:43] <elukey>	 Emperor, urandom - okok, this is basically a no-op and with puppet disabled we just make sure it really is, so I considered it very safe. But okok let's do it on monday :)