[01:47:57] <jinxer-wm>	 (HAProxyEdgeTrafficDrop) firing: 66% request drop in text@drmrs during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=drmrs&var-cache_type=text - https://alerts.wikimedia.org/?q=alertname%3DHAProxyEdgeTrafficDrop
[01:52:56] <jinxer-wm>	 (HAProxyEdgeTrafficDrop) resolved: 66% request drop in text@drmrs during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=drmrs&var-cache_type=text - https://alerts.wikimedia.org/?q=alertname%3DHAProxyEdgeTrafficDrop
[03:24:51] <wikibugs>	 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-eqiad: eqiad: Move links to new MPC7E linecard - https://phabricator.wikimedia.org/T304712 (10Papaul)
[04:53:16] <jinxer-wm>	 (VarnishTrafficDrop) firing: Varnish traffic in eqiad has dropped 69.59598083042479% - https://wikitech.wikimedia.org/wiki/Varnish - https://grafana.wikimedia.org/d/000000180/varnish-http-requests?viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DVarnishTrafficDrop
[04:58:16] <jinxer-wm>	 (VarnishTrafficDrop) resolved: Varnish traffic in eqiad has dropped 69.11767940777685% - https://wikitech.wikimedia.org/wiki/Varnish - https://grafana.wikimedia.org/d/000000180/varnish-http-requests?viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DVarnishTrafficDrop
[05:14:56] <jinxer-wm>	 (HAProxyEdgeTrafficDrop) firing: (2) 62% request drop in text@eqiad during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop  - https://alerts.wikimedia.org/?q=alertname%3DHAProxyEdgeTrafficDrop
[05:19:56] <jinxer-wm>	 (HAProxyEdgeTrafficDrop) resolved: (2) 68% request drop in text@eqiad during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop  - https://alerts.wikimedia.org/?q=alertname%3DHAProxyEdgeTrafficDrop
[05:54:56] <jinxer-wm>	 (HAProxyEdgeTrafficDrop) firing: 57% request drop in text@codfw during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=codfw&var-cache_type=text - https://alerts.wikimedia.org/?q=alertname%3DHAProxyEdgeTrafficDrop
[05:59:56] <jinxer-wm>	 (HAProxyEdgeTrafficDrop) firing: (5) 46% request drop in text@codfw during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop  - https://alerts.wikimedia.org/?q=alertname%3DHAProxyEdgeTrafficDrop
[06:04:56] <jinxer-wm>	 (HAProxyEdgeTrafficDrop) resolved: (5) 68% request drop in text@codfw during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop  - https://alerts.wikimedia.org/?q=alertname%3DHAProxyEdgeTrafficDrop
[08:28:22] <wikibugs>	 10Traffic, 10SRE, 10Patch-For-Review, 10Performance-Team (Radar): Test haproxy as a WMF's CDN TLS terminator with real traffic - https://phabricator.wikimedia.org/T290005 (10Vgutierrez) 05In progress→03Resolved a:03Vgutierrez
[08:31:53] <wikibugs>	 10Traffic, 10Performance-Team, 10SRE, 10SRE-swift-storage, 10Patch-For-Review: Automatically clean up unused thumbnails in Swift - https://phabricator.wikimedia.org/T211661 (10Ladsgroup) >>! In T211661#8139653, @Joe wrote: > I would love to see some numbers on how many thumbnails get a response from swif...
[08:37:34] <wikibugs>	 10Traffic, 10Performance-Team, 10SRE, 10SRE-swift-storage, 10Patch-For-Review: Automatically clean up unused thumbnails in Swift - https://phabricator.wikimedia.org/T211661 (10Ladsgroup) Another way that might be much much easier. Delete old thumbnails in small portion of swift and check how much request...
[12:27:30] <wikibugs>	 10Traffic, 10Performance-Team (Radar): Track TTFB per Cache Status Code in ATS - https://phabricator.wikimedia.org/T321484 (10Vgutierrez)
[12:27:51] <wikibugs>	 10Traffic, 10Performance-Team (Radar): Track TTFB per Cache Status Code in ATS - https://phabricator.wikimedia.org/T321484 (10Vgutierrez) p:05Triage→03Medium
[14:26:15] <wikibugs>	 10Traffic, 10DC-Ops, 10SRE, 10ops-ulsfo, 10Patch-For-Review: Q1:rack/setup/install cp40[37-52] - https://phabricator.wikimedia.org/T317244 (10ssingh)
[14:34:16] <jinxer-wm>	 (VarnishTrafficDrop) firing: (2) Varnish traffic in eqiad has dropped 64.08753063623759% - https://wikitech.wikimedia.org/wiki/Varnish - https://grafana.wikimedia.org/d/000000180/varnish-http-requests?viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DVarnishTrafficDrop
[14:34:56] <jinxer-wm>	 (HAProxyEdgeTrafficDrop) firing: 51% request drop in text@eqsin during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=eqsin&var-cache_type=text - https://alerts.wikimedia.org/?q=alertname%3DHAProxyEdgeTrafficDrop
[14:39:16] <jinxer-wm>	 (VarnishTrafficDrop) resolved: (6) Varnish traffic in eqiad has dropped 52.46831015925706% - https://wikitech.wikimedia.org/wiki/Varnish - https://grafana.wikimedia.org/d/000000180/varnish-http-requests?viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DVarnishTrafficDrop
[14:39:56] <jinxer-wm>	 (HAProxyEdgeTrafficDrop) resolved: (3) 61% request drop in text@eqiad during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop  - https://alerts.wikimedia.org/?q=alertname%3DHAProxyEdgeTrafficDrop
[14:44:41] <wikibugs>	 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-eqiad: eqiad: Move links to new MPC7E linecard - https://phabricator.wikimedia.org/T304712 (10Papaul)
[15:25:06] <wikibugs>	 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-eqiad: eqiad: Move links to new MPC7E linecard - https://phabricator.wikimedia.org/T304712 (10Papaul)
[15:38:52] <wikibugs>	 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-eqiad: eqiad: Move links to new MPC7E linecard - https://phabricator.wikimedia.org/T304712 (10Papaul)
[15:49:48] <wikibugs>	 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-eqiad: eqiad: Move links to new MPC7E linecard - https://phabricator.wikimedia.org/T304712 (10Papaul)
[15:50:35] <wikibugs>	 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-eqiad: eqiad: Move links to new MPC7E linecard - https://phabricator.wikimedia.org/T304712 (10Papaul) a:05Papaul→03ayounsi
[15:56:27] <wikibugs>	 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-eqiad: eqiad: Move links to new MPC7E linecard - https://phabricator.wikimedia.org/T304712 (10Papaul) a:05ayounsi→03Papaul
[17:47:56] <jinxer-wm>	 (HAProxyEdgeTrafficDrop) firing: 29% request drop in text@eqsin during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=eqsin&var-cache_type=text - https://alerts.wikimedia.org/?q=alertname%3DHAProxyEdgeTrafficDrop
[17:48:16] <jinxer-wm>	 (VarnishTrafficDrop) firing: (2) Varnish traffic in eqiad has dropped 53.43287656170819% - https://wikitech.wikimedia.org/wiki/Varnish - https://grafana.wikimedia.org/d/000000180/varnish-http-requests?viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DVarnishTrafficDrop
[17:52:56] <jinxer-wm>	 (HAProxyEdgeTrafficDrop) firing: (6) 51% request drop in text@codfw during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop  - https://alerts.wikimedia.org/?q=alertname%3DHAProxyEdgeTrafficDrop
[17:53:16] <jinxer-wm>	 (VarnishTrafficDrop) firing: (8) Varnish traffic in codfw has dropped 26.009064417255107% - https://wikitech.wikimedia.org/wiki/Varnish - https://grafana.wikimedia.org/d/000000180/varnish-http-requests?viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DVarnishTrafficDrop
[17:57:56] <jinxer-wm>	 (HAProxyEdgeTrafficDrop) resolved: (6) 51% request drop in text@codfw during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop  - https://alerts.wikimedia.org/?q=alertname%3DHAProxyEdgeTrafficDrop
[17:58:16] <jinxer-wm>	 (VarnishTrafficDrop) resolved: (10) Varnish traffic in codfw has dropped 25.22607609137403% - https://wikitech.wikimedia.org/wiki/Varnish - https://grafana.wikimedia.org/d/000000180/varnish-http-requests?viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DVarnishTrafficDrop
[20:47:48] <wikibugs>	 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-eqiad: eqiad: Move links to new MPC7E linecard - https://phabricator.wikimedia.org/T304712 (10Papaul) @cmooney hey I was about to set up sub-ports on fpc1 pic0 on both cr1-eqiad and cr2-eqiad and realized that lsw1-[e1-f1] are connected to pic0 of fpc1 o...
[21:05:52] <wikibugs>	 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-eqiad: eqiad: Move links to new MPC7E linecard - https://phabricator.wikimedia.org/T304712 (10cmooney) @papaul hey.  I think it can be done in any order.  Probably best to hard down the port first to be safe (which will cause the CR to down the BGP sessi...