[01:05:57] (HAProxyEdgeTrafficDrop) firing: 40% request drop in text@esams during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=esams&var-cache_type=text - https://alerts.wikimedia.org/?q=alertname%3DHAProxyEdgeTrafficDrop [01:10:57] (HAProxyEdgeTrafficDrop) resolved: (5) 64% request drop in text@codfw during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://alerts.wikimedia.org/?q=alertname%3DHAProxyEdgeTrafficDrop [19:11:16] 10Traffic, 10SRE: All wikis down: error 400 - https://phabricator.wikimedia.org/T308940 (10AlexisJazz) [19:11:40] 10Traffic, 10SRE: All wikis down: error 400 - https://phabricator.wikimedia.org/T308940 (10RhinosF1) This should be resolved now [19:12:11] 10Traffic, 10SRE: All wikis down: error 400 - https://phabricator.wikimedia.org/T308940 (10AlexisJazz) [19:13:28] 10Traffic, 10SRE: All wikis down: error 400 - https://phabricator.wikimedia.org/T308940 (10AlexisJazz) >>! In T308940#7947028, @RhinosF1 wrote: > This should be resolved now I tried to report it sooner, but Phabricator was down! [19:13:38] 10Traffic, 10SRE, 10Wikimedia-Incident: All wikis down: error 400 - https://phabricator.wikimedia.org/T308940 (10Aklapper) [19:17:44] 10Traffic, 10SRE, 10Wikimedia-Incident: All wikis down: error 503 - https://phabricator.wikimedia.org/T308940 (10Krinkle) [19:20:48] 10Traffic, 10SRE, 10Wikimedia-Incident: All wikis down: error 503 - https://phabricator.wikimedia.org/T308940 (10AlexisJazz) [19:30:56] (HAProxyEdgeTrafficDrop) firing: 17% request drop in text@codfw during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=codfw&var-cache_type=text - https://alerts.wikimedia.org/?q=alertname%3DHAProxyEdgeTrafficDrop [19:33:08] 10Traffic, 10SRE, 10Wikimedia-Incident: All wikis down: error 503 - https://phabricator.wikimedia.org/T308940 (10Dzahn) Wikis are back up.. This incident is actively being investigated. [19:34:13] 10Traffic, 10SRE, 10Wikimedia-Incident: All wikis down: error 503 - https://phabricator.wikimedia.org/T308940 (10Krinkle) > Edit: sorry the 400 was due to a typo of mine. [Reporting_a_connectivity_issue](https://wikitech-static.wikimedia.org/wiki/Reporting_a_connectivity_issue) maybe shouldn't use www.wikime... [19:35:56] (HAProxyEdgeTrafficDrop) firing: (6) 7% request drop in text@codfw during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://alerts.wikimedia.org/?q=alertname%3DHAProxyEdgeTrafficDrop [19:40:56] (HAProxyEdgeTrafficDrop) firing: (6) 41% request drop in text@codfw during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://alerts.wikimedia.org/?q=alertname%3DHAProxyEdgeTrafficDrop [19:41:52] 10Traffic, 10SRE, 10Wikimedia-Incident: All wikis down: error 503 - https://phabricator.wikimedia.org/T308940 (10AlexisJazz) [19:45:56] (HAProxyEdgeTrafficDrop) resolved: (6) 55% request drop in text@codfw during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://alerts.wikimedia.org/?q=alertname%3DHAProxyEdgeTrafficDrop [20:06:41] 10Traffic, 10SRE, 10Wikimedia-Incident: All wikis down: error 503 - https://phabricator.wikimedia.org/T308940 (10Dzahn) p:05Triage→03High [20:07:38] 10Traffic, 10SRE, 10Wikimedia-Incident: All wikis down: error 503 - https://phabricator.wikimedia.org/T308940 (10Dzahn) Status High but not UBN anymore. We will follow-up with an incident report but currently no ongoing outage. [20:09:04] 10Traffic, 10SRE, 10Wikimedia-Incident: All wikis down: error 503 (resolved, follow-up pending) - https://phabricator.wikimedia.org/T308940 (10Dzahn) [20:14:01] 10Traffic: Cross-origin request from Wikitech to Wikidata blocked - https://phabricator.wikimedia.org/T308947 (10AlexisJazz) [20:14:42] 10Traffic: Cross-origin request from Wikitech to Wikidata blocked - https://phabricator.wikimedia.org/T308947 (10AlexisJazz) [20:19:12] 10Traffic: Cross-origin request from Wikitech to Wikidata blocked - https://phabricator.wikimedia.org/T308947 (10Legoktm) Wikitech and Wikidata are not linked by SUL, so you need to make an unathenticated request, see https://www.mediawiki.org/wiki/API:Cross-site_requests#Unauthenticated_CORS_Requests [20:25:38] 10Traffic: Cross-origin request from Wikitech to Wikidata blocked - https://phabricator.wikimedia.org/T308947 (10AlexisJazz) 05Open→03Invalid >>! In T308947#7947285, @Legoktm wrote: > Wikitech and Wikidata are not linked by SUL, so you need to make an unathenticated request, see https://www.mediawiki.org/wik... [20:50:09] 10Traffic, 10SRE, 10Wikimedia-Incident: All wikis down: error 503 (resolved, follow-up pending) - https://phabricator.wikimedia.org/T308940 (10Dzahn) This was a failure at the edge / caching layer. All services behind it were not directly affected but appeared down / received no traffic. beta cluster was not... [21:52:57] (HAProxyEdgeTrafficDrop) firing: 59% request drop in text@drmrs during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=drmrs&var-cache_type=text - https://alerts.wikimedia.org/?q=alertname%3DHAProxyEdgeTrafficDrop [21:57:57] (HAProxyEdgeTrafficDrop) resolved: 69% request drop in text@drmrs during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=drmrs&var-cache_type=text - https://alerts.wikimedia.org/?q=alertname%3DHAProxyEdgeTrafficDrop