[01:57:23] <nihal>	 hay
[10:19:12] <vgutierrez>	 hmm Emperor dunno if something has been done regarding what I've mentioned yesterday but https://grafana.wikimedia.org/d/1T_4O08Wk/ats-backends-origin-servers-overview?orgId=1&var-datasource=eqsin%20prometheus%2Fops&var-cluster=upload&var-origin=swift.discovery.wmnet&viewPanel=12 shows a short spike today, though the situation has improved
[10:19:33] <Emperor>	 vgutierrez: I've made no changes to prod swift this year 
[10:20:02] <Emperor>	 codfw frontends got a rolling-restart yesterday UTC-evening
[10:20:34] <vgutierrez>	 yep... I'm seeing that on SAL
[10:20:41] <vgutierrez>	 so probably that addressed it
[10:20:53] <Emperor>	 https://grafana.wikimedia.org/d/OPgmB1Eiz/swift?orgId=1&var-DC=codfw&var-prometheus=codfw%20prometheus%2Fops&from=now-7d&to=now-1m doesn't show either oddly
[10:21:31] <Emperor>	 https://grafana.wikimedia.org/d/000000584/swift-4gs?orgId=1&var-DC=codfw&var-prometheus=thanos&from=now-24h&to=now-1m does have a latency-spike but suggests it's already passed
[10:22:51] <vgutierrez>	 yikes
[10:23:47] <vgutierrez>	 maybe we should tighten a little bit the monitoring/alerting considering those 502s are user facing
[11:18:36] <Emperor>	 annoyingly those 50xs aren't visible in the proxy-server logs
[11:24:48] <Emperor>	 Though I do find 'proxy-server: STDERR: swift.common.exceptions.ChunkWriteTimeout: 60.0 seconds' appearing across the frontends between about 09:35 and 09:51
[11:31:05] <Emperor>	 https://phabricator.wikimedia.org/P42760 is a typical backtrace, which isn't vastly enlightening
[12:30:22] <vgutierrez>	 Emperor: a 502 is a gateway timeout.. it could be triggered by the nginx terminating the TLS in front of swift
[14:03:48] <moritzm>	 I need to disable puppet fleet-wide for approx 15 mins for a puppetdb, will start in about five minutes
[14:04:02] <moritzm>	 "puppetdb-related change" I meant
[14:23:24] <moritzm>	 puppet is back on
[16:11:43] <jbond>	 Emperor: re 859592 ms-be2050 yes i think i did expect you to merge im not really sure of the impact (if any) sorry should have been abit more explicit about that
[16:12:41] <Emperor>	 jbond: cool, sorry to have dropped the ball a bit there. I'll get that merged (probably tomorrow now).
[16:15:54] <jbond>	 Emperor: cool and let me know how it gose