[12:55:08] <godog>	 oncallers: I'm testing prometheus2007 as the only pybal-pooled host as part of https://phabricator.wikimedia.org/T383232 . note that this change does not affect queries going through thanos
[12:55:48] <godog>	 i.e. only the read path for prometheus-codfw.w.o and clients using prometheus.svc.codfw
[12:57:03] <volans>	 k
[12:57:04] <volans>	 thx
[14:40:57] <vgutierrez>	 `check_conntrack_table_size` alert is expected?
[14:42:25] <claime>	 vgutierrez: slyngs is working on removing the check from icinga afaict
[14:42:53] <vgutierrez>	 ack
[14:43:13] <slyngs>	 vgutierrez: Yeah, sorry, I got the patches submitted in the wrong order. It is fixed, but needs to rollout.
[14:58:47] <swfrench-wmf>	 slyngs: ok to merge your puppet patch?
[14:58:55] <slyngs>	 Yes, please do
[14:59:31] <swfrench-wmf>	 slyngs: ack, done
[14:59:39] <slyngs>	 Thanks
[15:03:51] <slyngs>	 vgutierrez: Again sorry, should be fixed now. I'll remove the monitoring service a little more carefully sometime tomorrow. 
[16:02:34] <fabfur>	 moritzm: I have your change on puppet along with mine, going to merge, ok? 
[16:03:03] <moritzm>	 fabfur: ack
[22:14:27] <urandom>	 isaranto: any eta on the liftwing error rate?
[22:14:46] <urandom>	 isaranto: wondering how long to set a silence for
[22:15:07] <urandom>	 (also, is there a phab to reference?)
[22:27:30] <isaranto>	 urandom o/ let's set it for a day. the traffic is just too high to handle at the moment. there is this task that we can use for reference https://phabricator.wikimedia.org/T387019
[22:27:49] <urandom>	 isaranto: is 12 hours ok?
[22:27:51] <isaranto>	 there was an issue already and it exploded when traffic increased
[22:28:03] <urandom>	 or do we need longer?
[22:34:14] <isaranto>	 a little longer for sure (18h)
[22:35:57] <isaranto>	 thank you!
[22:42:02] <urandom>	 ok