[02:37:42] 10Traffic, 10Wikidata, 10wdwb-tech: Can not access Wikidata - https://phabricator.wikimedia.org/T290373 (10Bugreporter) [02:42:46] 10Traffic, 10Wikidata, 10wdwb-tech: Can not access Wikidata - https://phabricator.wikimedia.org/T290373 (10Bugreporter) [02:48:09] 10Traffic, 10SRE, 10Wikidata, 10wdwb-tech, 10Wikimedia-Incident: Can not access Wikidata - https://phabricator.wikimedia.org/T290373 (10Bugreporter) [02:54:54] 10Traffic, 10SRE, 10Wikidata, 10wdwb-tech, 10Wikimedia-Incident: Can not access Wikidata - https://phabricator.wikimedia.org/T290373 (10Sunny00217) It doesn't only show wikidata, it can show at zhwiki ( https://zh.wikipedia.org/w/api.php?action=query&format=json&meta=tokens&type=csrf ) and commons ( http... [02:56:41] 10Traffic, 10SRE, 10Wikidata, 10wdwb-tech, 10Wikimedia-Incident: Can not access Wikidata - https://phabricator.wikimedia.org/T290373 (10MBH) I can't even add a page ( https://ru.wikipedia.org/wiki/Воук_%28политика%29 ) to my watchlist. Error 503 or "server doesn't responding". [03:01:40] 10Traffic, 10SRE, 10Wikidata, 10wdwb-tech, 10Wikimedia-Incident: Can not access Wikidata - https://phabricator.wikimedia.org/T290373 (10Legoktm) We have an ongoing outage right now and are investigating. [03:13:18] 10Traffic, 10SRE, 10Wikidata, 10wdwb-tech, 10Wikimedia-Incident: Can not access Wikidata - https://phabricator.wikimedia.org/T290373 (10Legoktm) Things should be back to normal now - if it's not, please let us know. [10:45:57] (VarnishTrafficDrop) firing: 62% GET drop in text@ during the past 30 minutes - https://grafana.wikimedia.org/d/000000180/varnish-http-requests?viewPanel=6 - https://alerts.wikimedia.org [10:50:57] (VarnishTrafficDrop) resolved: 67% GET drop in text@ during the past 30 minutes - https://grafana.wikimedia.org/d/000000180/varnish-http-requests?viewPanel=6 - https://alerts.wikimedia.org [10:51:12] 10netops, 10Infrastructure-Foundations: enwiki was down at 10:44 (UTC) - https://phabricator.wikimedia.org/T290379 (10AlexisJazz) [10:52:00] 10netops, 10Infrastructure-Foundations: enwiki was down at 10:44 (UTC) - https://phabricator.wikimedia.org/T290379 (10AlexisJazz) [10:52:38] 10Traffic, 10Wikimedia-Incident: enwiki was down at 10:44 (UTC) - https://phabricator.wikimedia.org/T290379 (10AlexisJazz) [10:53:50] 10Traffic, 10Wikimedia-Incident: enwiki was down at 10:44 (UTC) - https://phabricator.wikimedia.org/T290379 (10AlexisJazz) [11:35:06] 10Traffic, 10DBA, 10SRE, 10Wikimedia-Incident: enwiki was down at 10:44 (UTC) - https://phabricator.wikimedia.org/T290379 (10Marostegui) I'm on my phone but just to mention that the queries dropping is probably a consequence of something else and not the consequence. There's a huge spike and then the drop,... [12:16:00] 10Traffic, 10DBA, 10SRE, 10Wikimedia-Incident: enwiki was down at 10:44 (UTC) - https://phabricator.wikimedia.org/T290379 (10Marostegui) From what I can see all the API enwiki hosts got this query around the reported time which is pretty crazy (hiding the query for obvious reasons): {P17211} Pasting the e...