[07:24:13] <moritzm>	 hi DBAs, when checking the decom for the old IDP nodes I realised we probably have a stale mysql grant: https://github.com/wikimedia/operations-puppet/blob/production/modules/role/templates/mariadb/grants/clouddb.sql.erb#L36
[07:24:24] <moritzm>	 that IP is currently held by idp2002
[07:24:41] <moritzm>	 I'd just make a patch to drop it from the grants file, or am I missing something here?
[07:26:01] <taavi>	 that looks outdated in general
[07:36:28] <arnaudb>	 we don't have grant automation yet so you'll have to drop those yourself if they are still in place. but the patch is more than welcome!
[07:40:33] <arnaudb>	 erratum: we* will drop those :D sorry for this
[07:45:08] <moritzm>	 after some discussion with Simon this is in fact clarified, this was for the IDPs to be able to retrieve U2F records (which are now obsoleted by webauthn), so these can in facat legitimate go away
[07:46:35] <moritzm>	 so this was probably juts a stale comment which still referrred to labspuppet via copy&pasta
[09:28:54] <arnaudb>	 I've accidentally depooled db1165, it's repooling 
[09:34:52] <jynus>	 db2101 crashed?
[09:36:15] <arnaudb>	 looks like it rebooted
[09:36:31] <arnaudb>	 has*
[09:37:04] <jinxer-wm>	 (SystemdUnitFailed) firing: wmf_auto_restart_prometheus-mysqld-exporter.service on db2202:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[09:42:11] <jynus>	 arnaudb: "The server could not be powered on or a server critical error occurred"
[09:43:41] <jynus>	 Uncorrectable Machine Check Exception + DIMM Failure - Uncorrectable Memory Error (Processor 2, DIMM 9)
[09:43:55] <arnaudb>	 oof indeed, seems like a good reason to crash
[09:44:49] <jynus>	 we will decomission it
[09:45:22] <jynus>	 I just set up db2201
[09:46:22] <jynus>	 https://phabricator.wikimedia.org/T362311
[10:00:29] <jynus>	 I forgot to upgrade xtrabackup (wmf-mariadb106) on dbprov2004 to 10.6.17, so the prepare failed
[11:51:04] <arnaudb>	 moritzm: I've just discussed with taavi and they're telling me that this can be probably dropped on galera, you confirm it's obsolete and OK to be discarded?
[11:55:35] <moritzm>	 no objections!
[11:56:00] <arnaudb>	 thanks :)
[12:45:45] <icinga-wm>	 PROBLEM - MariaDB sustained replica lag on s6 on db2129 is CRITICAL: 51.2 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2129&var-port=9104
[12:46:45] <icinga-wm>	 RECOVERY - MariaDB sustained replica lag on s6 on db2129 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2129&var-port=9104
[13:20:48] <icinga-wm>	 PROBLEM - MariaDB sustained replica lag on m3 on db2134 is CRITICAL: 13.5 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2134&var-port=9104
[13:22:48] <icinga-wm>	 RECOVERY - MariaDB sustained replica lag on m3 on db2134 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2134&var-port=9104
[15:12:29] <elukey>	 urandom: o/
[15:12:37] <elukey>	 aqs1010's instances are running with PKI!
[15:13:21] <urandom>	 \o/
[15:14:16] <urandom>	 So... by "running", we mean that they are using the new bundle, which also contains the old self-signed rootCa, yes?
[15:16:39] <elukey>	 running the new truststore with the bundle (self-signed + root PKI) and TLS cert from PKI (keystore)
[15:17:00] <urandom>	 right, sorry, that's what I meant
[15:19:13] <urandom>	 so yes, \o/
[15:23:29] <elukey>	 nodetool statuses look good, now the only issue could come from clients
[15:23:43] <elukey>	 but there shouldn't be any, hopefully :D
[15:23:55] <Emperor>	 ah, who cares about the clients? :)
[15:23:55] <elukey>	 next week if nothing goes on fire we can proceed with the rest, gradually of course
[15:23:59] <elukey>	 ahahhaahh yes yes
[15:27:35] <elukey>	 (I mean we don't do TLS verification in any client up to now so easy win)
[17:04:11] <jinxer-wm>	 (SystemdUnitFailed) firing: wmf_auto_restart_prometheus-mysqld-exporter@x1.service on db2101:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[17:09:11] <jinxer-wm>	 (SystemdUnitFailed) resolved: wmf_auto_restart_prometheus-mysqld-exporter@x1.service on db2101:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[23:30:33] <swfrench-wmf>	 FYI, the upcoming etcd maintenance I mentioned on 4/9 is unlikely to be scheduled for next week. feel free to plan schema changes touching dbctl. I'll follow up here when we have a more concrete window.