[05:26:27] 06Traffic, 06SRE: Intermittent access issues to English Wikipedia on desktop/laptop - https://phabricator.wikimedia.org/T402142#11092761 (10Joe) This task seems to coalesce different issues; having said that, we've had quite a few abusers lately that used old versions of Chrome as their user agent, so we had t... [05:31:19] 06Traffic, 06SRE: Intermittent access issues to English Wikipedia on desktop/laptop - https://phabricator.wikimedia.org/T402142#11092766 (10Joe) I should also add - if this has happened in the last week, that might be connected to T400119 - if users have any browser extension that modifies their user-agent, fo... [06:00:55] 06Traffic, 06SRE: Intermittent access issues to English Wikipedia on desktop/laptop - https://phabricator.wikimedia.org/T402142#11092799 (10Josve05a) I just noticed a pattern and wanted to surface it. For context: in the VRT (the volunteer helpdesk) we normally only see one such “Wikipedia inaccessible” email... [06:57:06] 06Traffic, 06SRE: Intermittent access issues to English Wikipedia on desktop/laptop - https://phabricator.wikimedia.org/T402142#11092835 (10Joe) >>! In T402142#11092799, @Josve05a wrote: > I just noticed a pattern and wanted to surface it. For context: in the VRT (the volunteer helpdesk) we normally only see o... [07:16:18] <_joe_> I think we need to have a form to report connectivity issues [07:22:51] FIRING: FermMSS: Unexpected MSS value on 10.2.2.27:80 @ ms-fe1009 - https://wikitech.wikimedia.org/wiki/LVS#LVSRealserverMSS_alert - https://grafana.wikimedia.org/d/Y9-MQxNSk/ipip-encapsulated-services?orgId=1&viewPanel=4&var-site=eqiad&var-cluster=swift - https://alerts.wikimedia.org/?q=alertname%3DFermMSS [07:27:51] RESOLVED: FermMSS: Unexpected MSS value on 10.2.2.27:80 @ ms-fe1009 - https://wikitech.wikimedia.org/wiki/LVS#LVSRealserverMSS_alert - https://grafana.wikimedia.org/d/Y9-MQxNSk/ipip-encapsulated-services?orgId=1&viewPanel=4&var-site=eqiad&var-cluster=swift - https://alerts.wikimedia.org/?q=alertname%3DFermMSS [07:50:08] 06Traffic, 06SRE: Intermittent access issues to English Wikipedia on desktop/laptop - https://phabricator.wikimedia.org/T402142#11092939 (10A_smart_kitten) >* Detailed error message the users are receiving - it is included at the bottom of the error page. Interestingly (IMO), from the screenshots shared in thi... [08:58:51] FIRING: FermMSS: Unexpected MSS value on 10.2.2.27:80 @ ms-fe1009 - https://wikitech.wikimedia.org/wiki/LVS#LVSRealserverMSS_alert - https://grafana.wikimedia.org/d/Y9-MQxNSk/ipip-encapsulated-services?orgId=1&viewPanel=4&var-site=eqiad&var-cluster=swift - https://alerts.wikimedia.org/?q=alertname%3DFermMSS [09:03:51] RESOLVED: FermMSS: Unexpected MSS value on 10.2.2.27:80 @ ms-fe1009 - https://wikitech.wikimedia.org/wiki/LVS#LVSRealserverMSS_alert - https://grafana.wikimedia.org/d/Y9-MQxNSk/ipip-encapsulated-services?orgId=1&viewPanel=4&var-site=eqiad&var-cluster=swift - https://alerts.wikimedia.org/?q=alertname%3DFermMSS [09:34:44] 06Traffic, 06SRE: Intermittent access issues to English Wikipedia on desktop/laptop - https://phabricator.wikimedia.org/T402142#11093380 (10XXBlackburnXx) >>! In T402142#11092766, @Joe wrote: > I should also add - if this has happened in the last week, that might be connected to T400119 @Joe This seems to be... [10:44:39] Hey :) Is there anything specific we need to be aware of for merging https://gerrit.wikimedia.org/r/c/operations/puppet/+/1171994 ? Should we leave deployment to scheduled puppet runs or actively run puppet on P:trafficserver::backend ? [10:44:58] * vgutierrez looking [10:45:05] I think I'd like to merge that tomorrow during the UTC morning infra window [10:46:13] given you're not introducing a change in behavior (just the ability of doing it) you could let puppet run organically on the CDN [10:46:37] Cool, I'll schedule that for tomorrow morning then, thanks <3 [10:46:40] np [15:26:31] 06Traffic, 10Page Content Service, 06Wikipedia-Android-App-Backlog, 10Content-Transform-Team (Work In Progress): [[2025 Coeur d'Alene shooting]] showing old version in Android app - https://phabricator.wikimedia.org/T398243#11094672 (10Jgiannelos) This should be fixed by now after T397072 [15:28:11] 06Traffic, 10Page Content Service, 06Wikipedia-Android-App-Backlog, 10Content-Transform-Team (Work In Progress): [[2025 Coeur d'Alene shooting]] showing old version in Android app - https://phabricator.wikimedia.org/T398243#11094682 (10Jgiannelos) 05Open→03Resolved a:03Jgiannelos [19:45:11] 06Traffic, 06SRE: Intermittent access issues to English Wikipedia on desktop/laptop - https://phabricator.wikimedia.org/T402142#11095772 (10Josve05a) Yet another report Ticket#2025081810008602: * Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36 * E... [19:51:16] 06Traffic, 06SRE: Intermittent access issues to English Wikipedia on desktop/laptop - https://phabricator.wikimedia.org/T402142#11095786 (10Josve05a) [21:06:42] 06Traffic, 10MediaWiki-extensions-QuickInstantCommons, 10MediaWiki-File-management, 06MediaWiki-Platform-Team, and 2 others: Make InstantCommons and other uses of ForeignApiRepo use WMF policy-compliant user agents - https://phabricator.wikimedia.org/T400881#11096139 (10Tgr) Updated T400881#11072676 and th... [22:03:33] 06Traffic, 06SRE: Intermittent access issues to English Wikipedia on desktop/laptop - https://phabricator.wikimedia.org/T402142#11096267 (10sbassett) I've recently removed one of our private security mitigations that may have been causing some of these unintended consequences (the incident had been resolved an... [23:40:12] 06Traffic, 06SRE: Intermittent access issues to English Wikipedia on desktop/laptop - https://phabricator.wikimedia.org/T402142#11096515 (10Josve05a) >>! In T402142#11092492, @Josve05a wrote: > Additional report from another user (Ticket #2025081710003225) - second person today: > > - Device: HP Pavilion x360...