[06:28:19] <elukey>	 hello folks
[06:28:36] <elukey>	 again root partition filling up for an-airflow
[06:29:08] <elukey>	 I can run the log retention drop to say 20d (instead of 30d) to free space
[07:18:18] <dcausse>	 elukey: I know there's T283856 but not sure we're hit by that now
[07:18:19] <stashbot>	 T283856: Airflow filled disks after losing connection to sql server - https://phabricator.wikimedia.org/T283856
[07:19:54] <elukey>	 dcausse: the last time I just reduced the rention of logs temporarily, it worked fine, not sure what else we can do
[07:20:08] <elukey>	 (manual/selective log clean up is an option but very tedious)
[07:20:13] <dcausse>	 what I see is a lot in /var/log/airflow/scheduler/ (250M/day) but they don't seem particularly useful (just info saying "loaded DAG XYZ"
[07:20:15] <dcausse>	 )
[07:20:36] <dcausse>	 I wonder if we could not simply reduce retention on these 
[07:21:18] <elukey>	 definitely
[07:22:07] <elukey>	 find /var/log/airflow/scheduler -type f -mtime +21 -delete
[07:22:14] <elukey>	 dcausse: --^
[07:22:20] <elukey>	 and then
[07:22:29] <elukey>	 the same with -type d -mtime +21 -empty -delete
[07:23:00] <elukey>	 it may need more than +21
[07:24:28] <dcausse>	 elukey: I can run this indeed
[07:26:00] <dcausse>	 back to 89%
[07:26:15] <dcausse>	 I'm sure we can delete more but we might wait for Erik
[07:46:30] <elukey>	 dcausse: <3
[10:47:32] <dcausse>	 lunch
[14:18:09] <mpham>	 cool mouth meat talk trey!
[14:41:59] <gehel>	 yeah! I liked that talk! There are definitely a few where I can't understand or cant speak the differences
[14:57:17] <dcausse>	 likewise, congrats Trey!
[15:07:38] <ebernhardson>	 lol, i'm in that picture but noone will recognize me :)
[15:16:27] <dcausse>	 ebernhardson: which year?
[15:21:16] <ebernhardson>	 dcausse: 2015 with short hair, in the next one for 2017 i have long hair
[15:21:33] <ebernhardson>	 well, "short"
[15:21:55] <ebernhardson>	 trey was easy to find in the pictures as well, being tall helps :)
[15:37:55] <dcausse>	 in 2015 you're easy to spot, now I wonder if you are in the 2013 one, I seem to recognize you but I could totally be mistaken :)
[15:39:03] <ebernhardson>	 hmm, i should be. started earlier in 2013
[15:39:14] <ebernhardson>	 maybe i was thinking of the 2013 image, now i have to find them. on officewiki?
[15:39:37] <dcausse>	 it's public https://upload.wikimedia.org/wikipedia/commons/c/c5/WMF_2013_Staff_Photo.jpg :)
[15:39:54] <ebernhardson>	 ahh, yes thats the one i mean. I'm in a suit jacket bottom right
[15:40:08] <dcausse>	 yes so that was you :)
[15:40:25] <ebernhardson>	 i don't think i'll revisit that particular haircut :)
[15:40:57] <dcausse>	 I guess it was a transition phase :P
[15:41:01] <ebernhardson>	 lol
[15:41:58] <ebernhardson>	 that year head of hr asked people to dress up for that day for the photo. You acan see how well it worked :) Thomasz there with pinstripes at least
[15:42:42] <dcausse>	 :)
[16:08:51] <Trey314159>	 Thanks, guys! I enjoyed making the flapping mouth meat video.
[16:09:34] <Trey314159>	 I was able to find Erik in the 2013 picture, but it helped knowing he had to be there *somewhere*. Not sure I would have recognized him without that extra nudge.