[11:57:57] <jinxer-wm>	 (HAProxyEdgeTrafficDrop) firing: 46% request drop in text@ulsfo during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=ulsfo&var-cache_type=text - https://alerts.wikimedia.org/?q=alertname%3DHAProxyEdgeTrafficDrop
[12:02:57] <jinxer-wm>	 (HAProxyEdgeTrafficDrop) resolved: (2) 60% request drop in text@eqiad during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop  - https://alerts.wikimedia.org/?q=alertname%3DHAProxyEdgeTrafficDrop
[13:57:27] <wikibugs>	 10Traffic, 10SRE, 10Patch-For-Review: Package and deploy ATS 9.1.2 - https://phabricator.wikimedia.org/T309651 (10ssingh)
[15:28:07] <brett>	 Re: The daily pontoon error emails, how naive am I in thinking that a simple puppet agent run would fix it?
[15:47:47] <sukhe>	 brett: only one way to find out... :P
[15:48:15] <brett>	 sukhe: Is that the advised COA? :)
[15:50:05] <sukhe>	 I am curious if it is the COA so I do think it's worth trying it out
[15:50:23] <sukhe>	 I also just noticed that we have been getting these mails periodically 
[16:03:11] <brett>	 I guess YOLO, I'mma do it
[16:07:27] <sukhe>	 lop
[16:07:28] <sukhe>	 lol
[16:11:05] <brett>	 > Error: Could not request certificate: The certificate retrieved from the master does not match the agent's private key.
[16:11:32] <brett>	 If I'm understanding this right, pontoon is the puppetmaster for the cloud servers, right? Does that mean I run something different than 'puppet agent'?
[16:40:35] <wikibugs>	 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10SRE, and 2 others: Allow idrac ftp fetching of firmware updates (either to existing ftp or new solution) - https://phabricator.wikimedia.org/T283771 (10RobH)
[16:44:08] <jinxer-wm>	 (HAProxyEdgeTrafficDrop) firing: 47% request drop in text@eqsin during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=eqsin&var-cache_type=text - https://alerts.wikimedia.org/?q=alertname%3DHAProxyEdgeTrafficDrop
[16:45:44] <wikibugs>	 10netops, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: Finalise design extension of WMCS networks to new cloudsw in Eqiad rows E/F - https://phabricator.wikimedia.org/T304989 (10nskaggs) @cmooney , for the manual override, https://wikitech.wikimedia.org/wiki/Network_design_-_Eqiad_WMCS_Network_...
[16:48:57] <jinxer-wm>	 (HAProxyEdgeTrafficDrop) resolved: 65% request drop in text@eqsin during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=eqsin&var-cache_type=text - https://alerts.wikimedia.org/?q=alertname%3DHAProxyEdgeTrafficDrop
[20:43:51] <wikibugs>	 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-eqiad, and 2 others: Replace labstore100[67] with clouddumps100[12] - https://phabricator.wikimedia.org/T309346 (10wiki_willy) a:03Cmjohnson
[21:29:57] <jinxer-wm>	 (HAProxyEdgeTrafficDrop) firing: 43% request drop in text@eqsin during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=eqsin&var-cache_type=text - https://alerts.wikimedia.org/?q=alertname%3DHAProxyEdgeTrafficDrop
[21:34:57] <jinxer-wm>	 (HAProxyEdgeTrafficDrop) resolved: (3) 61% request drop in text@eqiad during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop  - https://alerts.wikimedia.org/?q=alertname%3DHAProxyEdgeTrafficDrop
[22:51:57] <jinxer-wm>	 (HAProxyEdgeTrafficDrop) firing: (3) 53% request drop in text@drmrs during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop  - https://alerts.wikimedia.org/?q=alertname%3DHAProxyEdgeTrafficDrop
[22:56:57] <jinxer-wm>	 (HAProxyEdgeTrafficDrop) resolved: (4) 59% request drop in text@drmrs during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop  - https://alerts.wikimedia.org/?q=alertname%3DHAProxyEdgeTrafficDrop
[22:58:17] <wikibugs>	 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-eqiad, and 2 others: Replace labstore100[67] with clouddumps100[12] - https://phabricator.wikimedia.org/T309346 (10Andrew) a:05Cmjohnson→03Andrew I think this should be assigned to me, to put the new hosts into service. That's currently blocked by a...