[05:52:56] (HAProxyEdgeTrafficDrop) firing: 47% request drop in text@eqiad during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=eqiad&var-cache_type=text - https://alerts.wikimedia.org/?q=alertname%3DHAProxyEdgeTrafficDrop [05:57:56] (HAProxyEdgeTrafficDrop) resolved: (2) 63% request drop in text@codfw during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://alerts.wikimedia.org/?q=alertname%3DHAProxyEdgeTrafficDrop [08:01:56] (HAProxyEdgeTrafficDrop) firing: 67% request drop in text@ulsfo during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=ulsfo&var-cache_type=text - https://alerts.wikimedia.org/?q=alertname%3DHAProxyEdgeTrafficDrop [08:06:56] (HAProxyEdgeTrafficDrop) resolved: 69% request drop in text@ulsfo during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=ulsfo&var-cache_type=text - https://alerts.wikimedia.org/?q=alertname%3DHAProxyEdgeTrafficDrop [10:34:55] 10Traffic, 10Analytics-Radar, 10Privacy Engineering, 10SRE: Publishing project anomaly data for censorship researchers. Evaluate privacy threats - https://phabricator.wikimedia.org/T183990 (10Aklapper) a:05ssingh→03None Removing task assignee due to inactivity as this open task has been assigned for mo... [11:47:18] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10SRE, and 2 others: Q1:(Need By: TBD) rack/setup/install cloudswift100[12] - https://phabricator.wikimedia.org/T289882 (10taavi) [15:43:24] 10Traffic, 10DC-Ops, 10SRE, 10ops-ulsfo: ulsfo refresh scheduling - https://phabricator.wikimedia.org/T317249 (10RobH) p:05Medium→03High @bblack, Now that we're back from our SRE summit, I'd like to pick this back up! We have the eqsin shipment pending, so getting this tested is now urgent. Can you... [16:58:16] (VarnishTrafficDrop) firing: Varnish traffic in eqsin has dropped 69.46558643210724% - https://wikitech.wikimedia.org/wiki/Varnish - https://grafana.wikimedia.org/d/000000180/varnish-http-requests?viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DVarnishTrafficDrop [17:03:16] (VarnishTrafficDrop) resolved: Varnish traffic in eqsin has dropped 69.46558643210724% - https://wikitech.wikimedia.org/wiki/Varnish - https://grafana.wikimedia.org/d/000000180/varnish-http-requests?viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DVarnishTrafficDrop [19:10:08] bblack: do you have time this week for a ~30m meeting with gehel and I regarding https://phabricator.wikimedia.org/T313751? [19:10:10] 10Traffic, 10RESTBase-API, 10SRE, 10Performance-Team (Radar): Text cluster is being hit with an average of 1.8k PURGE requests per second per host - https://phabricator.wikimedia.org/T318349 (10Krinkle) [19:14:47] ryankemper: yes, I think so, just grab an empty slot on my calendar [19:15:04] ack [19:31:24] 10Traffic, 10DC-Ops, 10SRE, 10ops-ulsfo: ulsfo refresh scheduling - https://phabricator.wikimedia.org/T317249 (10BBlack) CP Nodes' mapping for replacements and where NVMEs go: | cp nodes | Current | Replacement | Disks | text | 21-26, 33, 34 | 37-44 | Single NVME | upload | 27-32, 35, 36 | 45-52 | Dual N... [20:23:00] 10Traffic, 10SRE, 10serviceops, 10Performance-Team (Radar): Split edge caches between php versions - https://phabricator.wikimedia.org/T311479 (10Krinkle) a:05Krinkle→03Joe Given the rollout of the PHP74 cookie campaign, I assume this has since been resolved. [20:23:33] 10Traffic, 10SRE, 10serviceops, 10Performance-Team (Radar): Split edge caches between php versions - https://phabricator.wikimedia.org/T311479 (10Krinkle) 05Open→03Resolved a:03Krinkle