[03:06:56] <jinxer-wm>	 (VarnishTrafficDrop) firing: 65% GET drop in text@esams during the past 30 minutes  - https://grafana.wikimedia.org/d/000000180/varnish-http-requests?viewPanel=6 - https://alerts.wikimedia.org
[03:13:09] <wikibugs>	 10Traffic, 10SRE, 10Wikimedia-production-error: 2021-09-25 Wikimedia sites down - https://phabricator.wikimedia.org/T291765 (10DannyS712)
[03:13:18] <wikibugs>	 10Traffic, 10SRE, 10Wikimedia-production-error: 2021-09-25 Wikimedia sites down - https://phabricator.wikimedia.org/T291765 (10DannyS712) p:05Triage→03Unbreak!
[03:15:57] <jinxer-wm>	 (VarnishTrafficDrop) firing: (2) 64% GET drop in text@eqiad during the past 30 minutes  - https://grafana.wikimedia.org/d/000000180/varnish-http-requests?viewPanel=6 - https://alerts.wikimedia.org
[03:21:40] <jinxer-wm>	 (VarnishTrafficDrop) resolved: (2) 66% GET drop in text@eqiad during the past 30 minutes  - https://grafana.wikimedia.org/d/000000180/varnish-http-requests?viewPanel=6 - https://alerts.wikimedia.org
[03:21:58] <wikibugs>	 10Traffic, 10SRE: 2021-09-25 Wikimedia sites down - https://phabricator.wikimedia.org/T291765 (10Peachey88)
[03:22:33] <wikibugs>	 10Traffic, 10SRE: 2021-09-26 (UTC) Wikimedia sites down - https://phabricator.wikimedia.org/T291765 (10Peachey88)
[03:23:01] <wikibugs>	 10Traffic, 10SRE, 10Wikimedia-Incident: 2021-09-26 (UTC) Wikimedia sites down - https://phabricator.wikimedia.org/T291765 (10AntiCompositeNumber)
[03:32:05] <wikibugs>	 10Traffic, 10SRE, 10Wikimedia-Incident: 2021-09-26 (UTC) Wikimedia sites down - https://phabricator.wikimedia.org/T291765 (10ItsPugle) The production (live) sites seem to be back up now, albeit site loading seems to be a bit slower than usual (presumably due to an influx of requests as the site came back onl...
[03:37:45] <wikibugs>	 10Traffic, 10SRE, 10Wikimedia-Incident: 2021-09-26 (UTC) Wikimedia sites down - https://phabricator.wikimedia.org/T291765 (10PorkchopGMX) This error appeared as HTTP 503 when trying to log into Phabricator via MediaWiki. For a brief moment, the standard “our servers are undergoing maintenance” error appeared...
[03:47:04] <wikibugs>	 10Traffic, 10SRE, 10Wikimedia-Incident: 2021-09-26 (UTC) Wikimedia sites down - https://phabricator.wikimedia.org/T291765 (10Andrew) 05Open→03Resolved a:03Andrew SREs are investigating and responding to this issue; it should be largely resolved by now.  As a DOS-related issue the specifics will not be...
[11:31:44] <wikibugs>	 10Traffic, 10Phabricator: Phabricator search times out - https://phabricator.wikimedia.org/T291775 (10RhinosF1) Is Phab behind envoy?  The plain error rather than 'Wikimedia Error' sounds simmilar to {T287983}
[11:36:10] <wikibugs>	 10Traffic, 10Phabricator: Phabricator search times out - https://phabricator.wikimedia.org/T291775 (101234qwer1234qwer4) "Wikimedia error"? Don't think this appears on Phab. T258803#7378611, where I had mentioned this task, appears to be the design of a typical Phab error.
[11:40:59] <wikibugs>	 10Traffic, 10Phabricator: Phabricator search times out - https://phabricator.wikimedia.org/T291775 (10RhinosF1) >>! In T291775#7378639, @1234qwer1234qwer4 wrote: > "Wikimedia error"? Don't think this appears on Phab. T258803#7378611, where I had mentioned this task, appears to be the design of a typical Phab e...
[11:50:16] <RhinosF1>	 majavah: I'm talking sense right ^
[11:50:40] <RhinosF1>	 (Martin answered)
[14:48:48] <wikibugs>	 10Traffic, 10Phabricator, 10Release-Engineering-Team: Phabricator search times out - https://phabricator.wikimedia.org/T291775 (10Marostegui) p:05Triage→03Medium It does eventually work for me (aka: no timeouts, but it does take a looong time). I have found this query:  ` | 440482799 | phuser          |...