[06:46:56] (EdgeTrafficDrop) firing: 69% request drop in text@codfw during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=codfw&var-cache_type=text - https://alerts.wikimedia.org [06:56:56] (EdgeTrafficDrop) resolved: 69% request drop in text@codfw during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=codfw&var-cache_type=text - https://alerts.wikimedia.org [06:59:56] (EdgeTrafficDrop) firing: 68% request drop in text@codfw during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=codfw&var-cache_type=text - https://alerts.wikimedia.org [07:04:56] (EdgeTrafficDrop) resolved: 69% request drop in text@codfw during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=codfw&var-cache_type=text - https://alerts.wikimedia.org [17:42:35] 10Domains, 10SRE: The phab.wiki domain redirect suddenly outputs "404, this domain is not configured" - https://phabricator.wikimedia.org/T298041 (10MarcoAurelio) [18:04:55] 10Domains, 10Traffic, 10SRE: The phab.wiki domain redirect suddenly outputs "404, this domain is not configured" - https://phabricator.wikimedia.org/T298041 (10MarcoAurelio) [18:33:27] 10Domains, 10Traffic, 10SRE: The phab.wiki domain redirect suddenly outputs "404, this domain is not configured" - https://phabricator.wikimedia.org/T298041 (10Dzahn) ` phab.wiki has address 104.21.94.13 phab.wiki has address 172.67.218.54 phab.wiki has IPv6 address 2606:4700:3033::6815:5e0d phab.wiki has IP... [18:37:21] 10Domains, 10Traffic, 10SRE: The phab.wiki domain redirect suddenly outputs "404, this domain is not configured" - https://phabricator.wikimedia.org/T298041 (10Dzahn) Hi @MarcoAurelio thanks for the report! I wasn't aware of this domain name and after searching in some places (my own email, Phabricator, D... [18:44:28] 10Domains, 10Traffic, 10SRE: The phab.wiki domain redirect suddenly outputs "404, this domain is not configured" - https://phabricator.wikimedia.org/T298041 (10Dzahn) {F34890931} [18:45:14] 10Domains, 10Traffic, 10Phabricator, 10SRE: The phab.wiki domain redirect suddenly outputs "404, this domain is not configured" - https://phabricator.wikimedia.org/T298041 (10Dzahn) [18:46:17] 10Domains, 10Traffic, 10Phabricator, 10SRE: The phab.wiki domain redirect suddenly outputs "404, this domain is not configured" - https://phabricator.wikimedia.org/T298041 (10MarcoAurelio) @Dzahn Searching through my inbox, the first occurence of phab.wiki is from 2019, so it has been around for a short wh... [18:48:02] 10Domains, 10Traffic, 10Phabricator, 10SRE: The phab.wiki domain redirect suddenly outputs "404, this domain is not configured" - https://phabricator.wikimedia.org/T298041 (10Dzahn) Let me try to find the original old list of all the .wiki domains WMF once had. Will get back to you. [18:56:39] 10Domains, 10Traffic, 10Phabricator, 10SRE: The phab.wiki domain redirect suddenly outputs "404, this domain is not configured" - https://phabricator.wikimedia.org/T298041 (10Dzahn) @MarcoAurelio I still have an email from 2014 (!sic) that talks about .wiki domains and has a list that WMF was going to get.... [19:05:25] 10Domains, 10Traffic, 10Phabricator, 10SRE: The phab.wiki domain redirect suddenly outputs "404, this domain is not configured" - https://phabricator.wikimedia.org/T298041 (10Aklapper) I could neither find nor remember any trace of the domain `phab.wiki` either here... [19:08:48] 10Domains, 10Traffic, 10Phabricator, 10SRE: The phab.wiki domain redirect suddenly outputs "404, this domain is not configured" - https://phabricator.wikimedia.org/T298041 (10Dzahn) But I can confirm it does pop up in Google search results. maybe it's possible Phacility owns/owned that and pointed to us as... [19:13:14] 10Domains, 10Traffic, 10Phabricator, 10SRE: The phab.wiki domain redirect suddenly outputs "404, this domain is not configured" - https://phabricator.wikimedia.org/T298041 (10Zabe) Using the global search tool I found https://wikitech.wikimedia.org/wiki/User:Revi/phab.wiki. [19:19:13] 10Wikimedia-Apache-configuration, 10Fundraising-Backlog, 10SRE, 10Thank-You-Page, and 4 others: Deal with donatewiki Thank You page launching in apps - https://phabricator.wikimedia.org/T259312 (10JMinor) 05Open→03Resolved Ok, I think we finally nailed this one down. Thanks all. [19:20:31] 10Domains, 10Traffic, 10Phabricator, 10SRE: The phab.wiki domain redirect suddenly outputs "404, this domain is not configured" - https://phabricator.wikimedia.org/T298041 (10Dzahn) Great detective work @Zabe, Thanks! added @revi Hi @Revi :) [19:23:23] 10Domains, 10Traffic, 10Phabricator, 10SRE: The phab.wiki domain redirect suddenly outputs "404, this domain is not configured" - https://phabricator.wikimedia.org/T298041 (10Dzahn) https://wikitech.wikimedia.org/wiki/User_talk:Revi/phab.wiki [19:26:20] 10Domains, 10Traffic, 10Phabricator, 10SRE: The phab.wiki domain redirect suddenly outputs "404, this domain is not configured" - https://phabricator.wikimedia.org/T298041 (10Sunpriat2) @revi timestamps: https://web.archive.org/web/*/https://phab.wiki/* 1 https://web.archive.org/web/2019*/https://phab.wiki... [19:33:46] 10Domains, 10Phabricator, 10SRE: The phab.wiki domain redirect suddenly outputs "404, this domain is not configured" - https://phabricator.wikimedia.org/T298041 (10Dzahn) [19:34:12] 10Domains, 10Phabricator, 10SRE, 10serviceops-radar: The phab.wiki domain redirect suddenly outputs "404, this domain is not configured" - https://phabricator.wikimedia.org/T298041 (10Dzahn) [23:44:57] (EdgeTrafficDrop) firing: 67% request drop in text@esams during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=esams&var-cache_type=text - https://alerts.wikimedia.org [23:49:56] (EdgeTrafficDrop) resolved: 66% request drop in text@esams during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=esams&var-cache_type=text - https://alerts.wikimedia.org