[07:12:08] <Amir1>	 marostegui: https://gerrit.wikimedia.org/r/c/operations/software/+/759645 :D
[07:14:14] <marostegui>	 oh nice!
[11:26:02] <icinga-wm>	 PROBLEM - MariaDB sustained replica lag on x1 on db2096 is CRITICAL: 143 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2096&var-port=9104
[11:27:11] <marostegui>	 I have created https://phabricator.wikimedia.org/T300965, can someone pick it up on Monday?
[11:28:12] <marostegui>	 It will need coordinaton with DCOps
[11:32:31] <marostegui>	 hehe, looks like friday is about memory <+icinga-wm> PROBLEM - MariaDB memory on db1115 is CRITICAL: CRIT Memory 95% used. Largest process: mysqld (30163) = 92.4% https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting
[11:32:46] <marostegui>	 Tendril is being shutdown next week, so I am going to ignore that alert
[11:32:57] <marostegui>	 But someone would need to work on db2096 next week with papaul
[11:56:51] <icinga-wm>	 RECOVERY - MariaDB sustained replica lag on x1 on db2096 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2096&var-port=9104
[13:50:05] <Amir1>	 marostegui: how we should decide one a time for T300976?
[13:50:05] <stashbot>	 T300976: Switchover es5 master - https://phabricator.wikimedia.org/T300976
[13:51:54] <marostegui>	 Amir1: it is up to you, but I normally avoid Mondays or Fridays. I tend to them early in the morning so if something were to happen to the new master you have the whole day to monitor/replace/rollback etc
[13:52:13] <Amir1>	 noted, thanks
[13:52:53] <Amir1>	 marostegui: 10AM UTC Wed?
[13:52:58] <Amir1>	 9 Feb
[13:53:13] <marostegui>	 sure, up to you
[17:42:38] <andrewbogott>	 If anyone has time to answer some naive mariadb questions, I've got 'em!
[17:43:05] <andrewbogott>	 Specifically, I'm trying to migrate a service from buster (10.3) to bullseye (10.5) upstream packages
[17:43:40] <andrewbogott>	 And I have schema that work fine on 10.3 but on 10.5 I'm getting lots of size limits, things like "Row size too large. The maximum row size for the used table type, not counting BLOBs, is 65535."
[17:44:13] <andrewbogott>	 Wondering if I should blame the schema or if there's just some default that changed between versions (e.g. innodb_large_prefix')
[17:46:28] <kormat>	 andrewbogott: `pt-config-diff` may be able to help you pin that down, if you still have one of each running
[17:46:44] <andrewbogott>	 I do!  will try...
[17:47:22] <andrewbogott>	 oh wait, will that diff my.cnf?  Because that should be the same, it's puppetized
[17:47:40] <kormat>	 no, it connects to the servers
[17:47:53] <kormat>	 example usage:
[17:48:00] <kormat>	 `sudo pt-config-diff --defaults-file /root/.my.cnf h=db1136.eqiad.wmnet h=db2121.codfw.wmnet`
[17:49:02] <andrewbogott>	 oh cool
[17:49:14] * andrewbogott will probably have to do some firewall things
[18:13:38] <andrewbogott>	 kormat: I think I may be unblocked now! thanks for the pointer