[08:05:56] (HAProxyEdgeTrafficDrop) firing: 56% request drop in text@codfw during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=codfw&var-cache_type=text - https://alerts.wikimedia.org/?q=alertname%3DHAProxyEdgeTrafficDrop [08:10:56] (HAProxyEdgeTrafficDrop) resolved: (3) 42% request drop in text@codfw during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://alerts.wikimedia.org/?q=alertname%3DHAProxyEdgeTrafficDrop [08:13:23] Hello, is Phabricator down? It says 503 [08:14:21] Yep [08:14:38] Seems like all WMF sites are affected? [08:14:51] What is happening? [08:15:39] Is it a serious problem? [08:17:12] Yes [08:17:21] Someone will be working on it [08:19:08] 10Traffic, 10SRE, 10Wikimedia-Incident: Unable to view all Wikimedia projects - https://phabricator.wikimedia.org/T310431 (10Bugreporter) See also: https://grafana.wikimedia.org/d/000000170/wikidata-edits [08:22:26] Hello, is this outage due to DDOS attack? [08:25:07] 10Traffic, 10SRE, 10Wikimedia-Incident: Unable to view all Wikimedia projects - https://phabricator.wikimedia.org/T310431 (10GhostInTheMachine) If you report this error to the Wikimedia System Administrators, please include the details below. Request from 2.98.121.99 via cp3064 cp3064, Varnish XID 552143256... [08:25:49] tranve_wiki[m]: we are unlikely to get any explicit confirmation [08:25:59] but I'd take an educated guess [08:26:10] It's at least a DoS [08:27:23] 10Traffic, 10SRE, 10Wikimedia-Incident: Unable to view all Wikimedia projects - https://phabricator.wikimedia.org/T310431 (10MdsShakil) Looks like it's okay at the moment, I can see everything [08:31:02] (HAProxyEdgeTrafficDrop) firing: 66% request drop in text@codfw during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=codfw&var-cache_type=text - https://alerts.wikimedia.org/?q=alertname%3DHAProxyEdgeTrafficDrop [08:35:56] (HAProxyEdgeTrafficDrop) resolved: 64% request drop in text@codfw during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=codfw&var-cache_type=text - https://alerts.wikimedia.org/?q=alertname%3DHAProxyEdgeTrafficDrop [08:37:23] Looks like it is working now [08:38:09] same [08:40:56] (HAProxyEdgeTrafficDrop) firing: 40% request drop in text@codfw during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=codfw&var-cache_type=text - https://alerts.wikimedia.org/?q=alertname%3DHAProxyEdgeTrafficDrop [08:45:56] (HAProxyEdgeTrafficDrop) firing: (4) 16% request drop in text@codfw during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://alerts.wikimedia.org/?q=alertname%3DHAProxyEdgeTrafficDrop [08:46:50] https://media.discordapp.net/attachments/291940905240231937/985465048011010078/unknown.png?width=1200&height=432 [08:46:54] we know why now [08:47:11] https://phabricator.wikimedia.org/T307501 [08:50:22] GooseTheCat: that has nothing to do with today [08:50:33] That comment is from over a month ago [08:50:53] *self trout* yeah I blind [08:50:56] (HAProxyEdgeTrafficDrop) firing: (4) 16% request drop in text@codfw during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://alerts.wikimedia.org/?q=alertname%3DHAProxyEdgeTrafficDrop [08:51:18] Incident is mark as resolved on wikimediastatus.net [09:00:56] (HAProxyEdgeTrafficDrop) resolved: (4) 59% request drop in text@codfw during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://alerts.wikimedia.org/?q=alertname%3DHAProxyEdgeTrafficDrop [09:07:31] 10Traffic, 10SRE, 10Wikimedia-Incident: Unable to view all Wikimedia projects - https://phabricator.wikimedia.org/T310431 (10fgiunchedi) p:05Unbreak!→03Medium Thanks folks, there was indeed widespread unavailability to all sites. We're back now so I'm lowering the severity, there will be followups as well [09:11:12] 10Traffic, 10SRE, 10Wikimedia-Incident: Unable to view all Wikimedia projects - https://phabricator.wikimedia.org/T310431 (10MoritzMuehlenhoff) 05Open→03Resolved a:03MoritzMuehlenhoff The immediate issue has been resolved, closing. There are some actionables, but rather sub tasks to existing tasks and... [11:36:56] (HAProxyEdgeTrafficDrop) firing: 61% request drop in text@eqiad during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=eqiad&var-cache_type=text - https://alerts.wikimedia.org/?q=alertname%3DHAProxyEdgeTrafficDrop [11:41:56] (HAProxyEdgeTrafficDrop) resolved: (2) 66% request drop in text@eqiad during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://alerts.wikimedia.org/?q=alertname%3DHAProxyEdgeTrafficDrop [14:35:57] (HAProxyEdgeTrafficDrop) firing: 56% request drop in text@eqsin during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=eqsin&var-cache_type=text - https://alerts.wikimedia.org/?q=alertname%3DHAProxyEdgeTrafficDrop [14:40:57] (HAProxyEdgeTrafficDrop) resolved: (4) 69% request drop in text@codfw during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://alerts.wikimedia.org/?q=alertname%3DHAProxyEdgeTrafficDrop [15:15:57] (HAProxyEdgeTrafficDrop) firing: 13% request drop in text@drmrs during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=drmrs&var-cache_type=text - https://alerts.wikimedia.org/?q=alertname%3DHAProxyEdgeTrafficDrop [15:25:57] (HAProxyEdgeTrafficDrop) resolved: 69% request drop in text@drmrs during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=drmrs&var-cache_type=text - https://alerts.wikimedia.org/?q=alertname%3DHAProxyEdgeTrafficDrop [20:23:19] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10SRE, and 3 others: Q3:(Need By: TBD) rack/setup/install 2 new labstore hosts - https://phabricator.wikimedia.org/T302981 (10Andrew) 05Open→03Resolved I have these hosts partitioned now (sdb by hand) so closing this task. Thanks for your help papaul! [20:26:11] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-eqiad, 10cloud-services-team (Kanban): hdfs client packages for debian Bullseye - https://phabricator.wikimedia.org/T310451 (10Andrew)