[01:06:19] <andrewbogott>	 !log tools truncating logfiles with find . -name '*.log'  -size +1G -exec truncate --size=100M {} \;
[01:06:21] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[02:17:01] <andrewbogott>	 !log stopping neutron-l3-agent on cloudnet1005 because it's logging at a furious rate and about to fill the drive
[02:17:02] <stashbot>	 andrewbogott: Unknown project "stopping"
[02:58:48] <AmandaNP>	 !log utrs add WM Header debug manually into vendor files on server T325692
[02:58:51] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Utrs/SAL
[02:58:51] <stashbot>	 T325692: API errors in relation to UTRS - https://phabricator.wikimedia.org/T325692
[10:12:26] <arturo>	 !log admin [codfw1dev] failover neutron-l3-agent between cloudnet2005-dev/cloudnet2006-dev a couple of times T327463
[10:12:30] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[10:12:30] <stashbot>	 T327463: neutron-keepalived-state-change debug turned on, filling logs - https://phabricator.wikimedia.org/T327463
[13:15:51] <arturo>	 !log admin reinstall python3-neutron (to reset manual patching) on all cloudnet nodes and patch it via puppet, then restart neutron-l3-agent by hand (T327463)
[13:15:55] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[13:15:55] <stashbot>	 T327463: neutron-keepalived-state-change debug turned on, filling logs - https://phabricator.wikimedia.org/T327463
[13:45:23] <andrewbogott>	 arturo: you found where 'debug' was turned on! Thank you for researching that bug -- I noticed the problem after I was mostly out of brainpower for the day.
[13:45:58] <arturo>	 andrewbogott: that's OK :-) there's still the mystery of why this never happened before. That code has been unchanged for years
[13:46:29] <andrewbogott>	 Oh, it's not from the Z upgrade?
[13:47:39] <arturo>	 git blame on https://opendev.org/openstack/neutron/src/branch/master/neutron/agent/l3/keepalived_state_change.py#L172 shows it has been like that for years
[13:48:03] <andrewbogott>	 huh
[13:48:36] <andrewbogott>	 Well we avoided having our network crash due to a full disk, so I'm calling this a win :)
[13:48:50] <arturo>	 =)
[14:09:20] <andrewbogott>	 btw arturo, it is extremely good to have you back from leave!
[15:20:15] <arturo>	 🎉
[15:20:22] <arturo>	 thanks for the kind words
[21:24:09] <andrewbogott>	 !log tools truncating logfiles with find . -name '*.out'  -size +1G -exec truncate --size=100M {} \;
[21:24:12] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[22:21:59] <NicoleMarieCzeck>	 612/651
[23:24:14] <andrewbogott>	 !log tools truncating logfiles with find . -name '*.err'  -size +1G -exec truncate --size=100M {} \;
[23:24:16] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[23:27:12] <legoktm>	 andrewbogott: any reason that's not a daily cron?
[23:28:59] <andrewbogott>	 Not really!
[23:45:09] <andrewbogott>	 I guess I think of it as a drastic measure... really we should provide a proper log service so the issue doesn't come up as much.