[03:57:57] <jinxer-wm>	 (HAProxyEdgeTrafficDrop) firing: 69% request drop in text@drmrs during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=drmrs&var-cache_type=text - https://alerts.wikimedia.org/?q=alertname%3DHAProxyEdgeTrafficDrop
[04:02:57] <jinxer-wm>	 (HAProxyEdgeTrafficDrop) resolved: 69% request drop in text@drmrs during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=drmrs&var-cache_type=text - https://alerts.wikimedia.org/?q=alertname%3DHAProxyEdgeTrafficDrop
[06:00:04] <wikibugs>	 10Traffic: DRMRS: Geodns Configuration -- Phase 2 - https://phabricator.wikimedia.org/T311472 (10MMandere)
[06:00:55] <wikibugs>	 10Traffic: DRMRS: Geodns Configuration -- Phase 2 - https://phabricator.wikimedia.org/T311472 (10MMandere)
[06:01:02] <wikibugs>	 10Traffic, 10netops, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: drmrs: initial geodns configuration - https://phabricator.wikimedia.org/T304089 (10MMandere)
[08:48:47] <wikibugs>	 10Traffic, 10serviceops, 10Performance-Team (Radar): Split edge caches between php versions - https://phabricator.wikimedia.org/T311479 (10Joe)
[09:03:15] <wikibugs>	 10Traffic, 10serviceops, 10Performance-Team (Radar): Split edge caches between php versions - https://phabricator.wikimedia.org/T311479 (10Joe) p:05Triage→03High
[12:56:23] <wikibugs>	 10Traffic, 10SRE, 10ops-eqsin: SSH on cp5012.mgmt is flapping (CRITICAL) - https://phabricator.wikimedia.org/T311264 (10ssingh) Thanks for all the help @RobH! Marking this as resolved as the host is now pooled.
[12:56:37] <wikibugs>	 10Traffic, 10SRE, 10ops-eqsin: SSH on cp5012.mgmt is flapping (CRITICAL) - https://phabricator.wikimedia.org/T311264 (10ssingh) 05Open→03Resolved
[14:52:28] <wikibugs>	 10Traffic, 10SRE: pontoon.traffic.eqiad1.wikimedia.cloud unable to run puppet agent due to certificate mismatch - https://phabricator.wikimedia.org/T310303 (10BCornwall) I spoke with @Vgutierrez and they would like to keep these instances around. I'll do a little more digging into fixing this, particularly sin...
[15:36:56] <jinxer-wm>	 (HAProxyEdgeTrafficDrop) firing: (2) 60% request drop in text@eqsin during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop  - https://alerts.wikimedia.org/?q=alertname%3DHAProxyEdgeTrafficDrop
[15:41:56] <jinxer-wm>	 (HAProxyEdgeTrafficDrop) resolved: (3) 66% request drop in text@eqiad during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop  - https://alerts.wikimedia.org/?q=alertname%3DHAProxyEdgeTrafficDrop
[16:04:05] <wikibugs>	 10Traffic, 10DNS, 10SRE, 10WMF-Legal, and 3 others: Setup redirect of policy.wikimedia.org to Advocacy portal on Foundation website - https://phabricator.wikimedia.org/T310738 (10LSobanski) @Varnent After chatting about this some more, how about we do the following:  - Redirect policy.wikimedia.org as requ...
[18:01:41] <wikibugs>	 10Traffic, 10netops, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: Upgrade to Bird 2 - https://phabricator.wikimedia.org/T310574 (10ssingh) Notes from today's deployment:  - We were missing an additional bird config, notably, bird2 rejects by default so we need to explicitly set `export all` f...
[20:37:12] <brett>	 Any reason pontoon cannot connect to webproxy.eqiad.net:8080?
[20:37:33] <brett>	 pings are fine but looks like some firewalling might be preventing it from proxying
[20:41:07] <XioNoX>	 brett: pontoon is in wmcs, right?
[20:41:17] <brett>	 XioNoX: Yeah
[20:41:37] <brett>	 and I just realized I wrote net
[20:41:43] <brett>	 But I tried wmnet too :)
[20:42:45] <XioNoX>	 brett: https://wikitech.wikimedia.org/wiki/HTTP_proxy proxies are for prod to reach the internet
[20:43:05] <XioNoX>	 cloud VMs use NAT
[20:43:27] <XioNoX>	 then it comes down to the project's setting
[20:44:00] <brett>	 XioNoX: Interesting, so is that to say that these cloud instances are not supposed to ever reach out? By running "apt update" I saw "0% [Connecting to webproxy.eqiad.wmnet[...]" where it would hang
[20:44:51] <XioNoX>	 I think they're missconfigured, and should try to reach out directly and not try to talk to the proxies
[20:44:57] <mutante>	 I think the real fix is that where the proxy address shows up in code you'd need to make it a parameter and move it to Hiera. then set it there differently based on prod or cloud 
[20:45:13] <brett>	 makes sense
[20:46:17] <brett>	 vgutierrez: This might be revealing the value in blowing away/starting testing stuff from scratch on the regular - I don't know how much of this is manual misconfiguration and how much is puppet
[20:46:22] <mutante>	 cloud VPS instances can talk to external web directly.so basically "set HTTP_PROXY or not"
[20:46:25] <mutante>	 and HTTPS_PROXY
[20:48:03] <brett>	 thanks for the hints, mutante. I see that apt.conf.d/80security-*-proxy files are responsible.
[20:49:49] <mutante>	 brett: 
[20:49:49] <mutante>	 /puppet/hieradata$ grep -r webproxy *
[20:49:50] <mutante>	 common/profile/statograph.yaml:    http: http://webproxy:8080
[20:49:50] <mutante>	 common/profile/statograph.yaml:    https: http://webproxy:8080
[20:50:26] <mutante>	 I dont know what statograph does but it does this thing you need I believe
[20:50:43] <mutante>	 profile::statograph passes on proxies parameter
[20:50:58] <mutante>	 and in the module statograph it defaults to nothing:
[20:51:07] <mutante>	     Hash[String, Statograph::Proxy] $proxies = {},
[20:51:27] <mutante>	 but then it gives it specific values in prod Hiera
[20:51:52] <mutante>	 that way it should work in both places without using  "if $realm" 
[20:52:07] <mutante>	 which would also work but is a bit frowned upon
[23:55:09] <wikibugs>	 10Traffic, 10DNS, 10SRE, 10WMF-Legal, and 3 others: Setup redirect of policy.wikimedia.org to Advocacy portal on Foundation website - https://phabricator.wikimedia.org/T310738 (10Varnent) >>! In T310738#8030481, @LSobanski wrote: > @Varnent could we get a clarification of the timeline for this request? The...
[23:58:40] <wikibugs>	 10Traffic, 10DNS, 10SRE, 10WMF-Legal, and 4 others: Setup redirect of policy.wikimedia.org to Advocacy portal on Foundation website - https://phabricator.wikimedia.org/T310738 (10Dzahn)
[23:59:31] <wikibugs>	 10Traffic, 10DNS, 10SRE, 10WMF-Legal, and 4 others: Setup redirect of policy.wikimedia.org to Advocacy portal on Foundation website - https://phabricator.wikimedia.org/T310738 (10Dzahn) >>! In T310738#8033789, @LSobanski wrote: > @Varnent After chatting about this some more, how about we do the following:...