[00:15:29] <legoktm>	 awesome job Amir1 and jynus!
[02:50:29] <Amir1>	 ^^
[06:25:23] <icinga-wm>	 PROBLEM - MariaDB sustained replica lag on m1 on db1117 is CRITICAL: 48 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1117&var-port=13321
[06:27:59] <icinga-wm>	 RECOVERY - MariaDB sustained replica lag on m1 on db1117 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1117&var-port=13321
[10:46:40] <kormat>	 Amir1: i'm just noticing that db1126 is still at weight 100. is that intended?
[10:47:14] <Amir1>	 kormat: I check, It shouldn't be
[10:47:28] <kormat>	 https://phabricator.wikimedia.org/P21596 says it was at 275 previously
[10:48:40] <Amir1>	 kormat: aah, I remember, that's the incident 
[10:48:46] <kormat>	 yeah :)
[10:48:49] <Amir1>	 it caused an incident by being slow
[10:49:03] <Amir1>	 I think it's fine to increase it gradaully 
[10:50:42] <kormat>	 want me to handle it?
[11:00:48] <Amir1>	 kormat: sure. Thanks
[11:03:01] <kormat>	 Amir1: repool running.
[11:03:38] <Amir1>	 awesome, let me know once done 
[11:03:47] <Amir1>	 because I want to run a schema change on s8 
[11:05:47] <Amir1>	 kormat: oh and when you have time, I have a patch in auto_schema for you <3
[11:05:58] <Amir1>	 tested it and it works fine
[11:09:03] <kormat>	 ack, have seen :)
[11:11:57] <hoo>	 hey Amir1 :) Regarding https://phabricator.wikimedia.org/T300255: How long could these connections be reasonable? 30m?
[11:12:20] <Amir1>	 hoo: hi, long time no see. Yeah, half an hour is good
[11:12:50] <Amir1>	 below an hour is important because otherwise the automation tools consider it failed and repool it back
[11:13:19] <Amir1>	 my scripts are ten minutes though but they are not dumpers 
[11:13:23] <hoo>	 Sounds good… I'll do some benchmarking with the low-entity-id large numbers and that should be fine for everything else
[11:13:41] <Amir1>	 awesome
[14:45:19] <kormat>	 Amir1: repooling of db1126 finished (a while ago)
[14:45:30] <Amir1>	 awesome, thanks
[17:28:21] <topranks>	 Emperor: My apologies about the server install in the new racks, for the most part good to go on that
[17:28:26] <topranks>	 should have updated you sooner
[17:28:59] <topranks>	 I responded to the task there, given it's one of the first in the new racks it'd help me to follow the process make sure there are no glitches and double-check the network elements get set up the way we need