[01:10:38] <icinga-wm>	 PROBLEM - MariaDB sustained replica lag on m1 on db1217 is CRITICAL: 8.8 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1217&var-port=13321
[01:12:06] <icinga-wm>	 RECOVERY - MariaDB sustained replica lag on m1 on db1217 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1217&var-port=13321
[01:13:24] <icinga-wm>	 PROBLEM - MariaDB sustained replica lag on m1 on db2160 is CRITICAL: 17.2 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2160&var-port=13321
[01:14:52] <icinga-wm>	 RECOVERY - MariaDB sustained replica lag on m1 on db2160 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2160&var-port=13321
[05:40:27] <marostegui>	 I am going to failover m5 proxy
[06:37:38] <arnaudb>	 good morning y'all
[06:41:44] <Amir1>	 arnaudb: Good morning and welcome o/
[06:42:41] <arnaudb>	 thank you! 
[06:51:17] <jynus>	 good morning arnaudb
[06:52:08] <arnaudb>	 hello jynus
[06:56:24] <marostegui>	 arnaudb: welcome! (Manuel here)
[06:58:22] <arnaudb>	 thank you!
[09:23:46] <elukey>	 Emperor: o/
[09:24:05] <marostegui>	 elukey: he's out today
[09:24:09] <elukey>	 when you have a moment do you mind to let me know if https://phabricator.wikimedia.org/T345058 is crazy?
[09:24:12] <elukey>	 marostegui: ahhh okok
[09:24:22] <elukey>	 I have a cassandra procedure to do
[09:24:41] * marostegui runs
[09:24:52] <marostegui>	 elukey: you might want to ping urando.m once's he's online
[09:25:11] <elukey>	 <3
[09:25:32] <elukey>	 people don't trust me in #serviceops, I thought to get some love in here, but same thing
[09:25:43] <elukey>	 :D :D :D
[09:25:44] <marostegui>	 elukey: maybe it is time to get the hint
[09:26:11] <elukey>	 marostegui: exactly yes, thanks for the direct feedback :D
[09:26:29] <marostegui>	 my pleasure, you know you can always come here for some more
[09:28:28] <elukey>	 I will
[09:29:22] <marostegui>	 <3
[09:31:50] <elukey>	 <3
[10:20:37] <Amir1>	 marostegui: I haven't touched s6 for reboots nor schema changes. But that's basically the only section left in some cases. Can I do stuff on it? :D
[10:20:50] <marostegui>	 yeah
[10:21:00] <marostegui>	 I need to fist test RBR on sanitarium on codfw
[10:21:11] <marostegui>	 Amir1: However, be _very_ careful with sanitarium hosts
[10:21:18] <marostegui>	 How does the script stop mariadb?
[10:21:36] <Amir1>	 https://gerrit.wikimedia.org/r/c/operations/software/+/830659/1/dbtools/auto_schema/rolling_restart.py
[10:21:42] <Amir1>	 stop slave; 
[10:21:46] <Amir1>	 then stop mariadb
[10:22:01] <marostegui>	 Actually, you've rebooted them before already for all the other sections right?
[10:22:09] <Amir1>	 I'm not planning to touch sanitarium yet. Its master will do that though
[10:22:18] <marostegui>	 Fine yeah
[10:22:20] <marostegui>	 The master is ok
[10:22:31] <Amir1>	 cool
[10:22:45] <Amir1>	 I leave reboot santariums to you, you seem to like it :D
[10:22:52] <marostegui>	 :-(
[10:23:14] <Amir1>	 if you do the one that has s6 on it, I can take care of the rest
[10:23:45] <Amir1>	 https://grafana.wikimedia.org/d/000000377/host-overview?viewPanel=28&orgId=1&var-server=db1106&var-datasource=thanos&var-cluster=wmcs&from=1693202599462&to=1693212280891
[10:23:47] <marostegui>	 yeah I need to install 10.6 there
[10:23:53] <Amir1>	 btw, extlinks drop of s1: 60GB
[10:23:57] <marostegui>	 It is going to take me a few days, I want to test stuff on codfw
[10:24:15] <Amir1>	 sure, I'm on clinic duty this week
[10:24:16] <Amir1>	 no rush
[13:12:22] <marostegui>	 Amir1: Going to restart db1155 RIP
[13:12:33] <marostegui>	 Let's see what breaks this time :-/
[13:12:46] <Amir1>	 🤞
[13:13:00] <Amir1>	 I should make it a cron job
[13:13:18] <Amir1>	 like chaos monkey but instead of monkey, a godzilla 
[13:13:47] <Amir1>	 bonus point: Doing it on 2am sundays
[13:17:54] <marostegui>	 ok, host back and replication started
[13:17:55] <marostegui>	 we'll see
[13:49:29] <marostegui>	 jynus: can you remind me when m2 backups are taken? 
[13:49:46] <marostegui>	 basically when I can stop m2 backup source for a couple of hours
[13:50:12] <jynus>	 right now until 0 utc
[13:50:20] <marostegui>	 ok! thanks!
[13:50:39] <jynus>	 or starting I think around 14 or so (they take while long because of OTRS, precisely)
[13:50:57] <marostegui>	 So, you mean they are starting nowish or that I can do it nowish :)
[13:50:57] <jynus>	 tomorrow
[13:51:05] <jynus>	 now it is free
[13:51:12] <marostegui>	 great thanks
[14:11:09] <jynus>	 marostegui: how did you learn about DC switchover! :-D
[14:11:11] <jynus>	 ?
[14:11:38] <marostegui>	 jynus: The switchover ghost is always around us
[14:11:48] <jynus>	 uuuuuuuuuh
[14:12:22] <jynus>	 do you know who is leading it?
[14:12:51] <marostegui>	 I don't know who is specifically doing it this year from serviceops, nop
[14:12:54] <marostegui>	 claime ^
[14:13:24] <claime>	 kamila will handle it, but we have a kickoff meeting this week to determine exactly when it happens
[14:13:37] <jynus>	 nice
[14:13:49] <claime>	 It should have been sept 20th, but it probably won't be
[14:13:56] <marostegui>	 claime: could you tell her to talk to me once that's is decided?
[14:13:59] <claime>	 ofc
[14:14:04] <marostegui>	 thanks <3
[15:05:08] <Amir1>	 extlinks in s4 drops 230GB in some hosts and 140GB in some other https://grafana.wikimedia.org/d/000000377/host-overview?viewPanel=28&orgId=1&var-server=db2099&var-datasource=thanos&var-cluster=wmcs&from=1693216211027&to=1693235021027
[15:44:38] <marostegui>	 sobanski: db1118 is ready, I will post on the task
[18:51:25] <marostegui>	 jynus: is there an onboarding task/page already for arnaudb ?
[18:51:57] <jynus>	 yes
[18:52:02] <jynus>	 I will pm it to you