[01:46:16] 10Traffic, 10SRE: fawiki user reports getting 503 errors with message "upstream connect error or disconnect before headers" - https://phabricator.wikimedia.org/T310450 (10Bugreporter) [03:29:27] 10Traffic, 10Community-Tech, 10MediaWiki-Parser, 10SRE, and 4 others: Show SVGs in page language if available - https://phabricator.wikimedia.org/T205040 (10Winston_Sung) [03:58:47] 10Traffic, 10Community-Tech, 10MediaWiki-Parser, 10SRE, and 3 others: Show SVGs in page view language for language variants if available - https://phabricator.wikimedia.org/T310453 (10Winston_Sung) [04:16:36] 10Traffic, 10SRE, 10Wikimedia-Incident: Unable to view all Wikimedia projects - https://phabricator.wikimedia.org/T310431 (10Liz) This happened again in the past 15 minutes and lasted about 4 or 5 minutes. [04:21:42] 10Traffic, 10Community-Tech, 10MediaWiki-Parser, 10SRE, and 4 others: Show SVGs in page view language for language variants if available - https://phabricator.wikimedia.org/T310453 (10PatchDemoBot) Test wiki **created** on [[ https://patchdemo.wmflabs.org | Patch demo ]] by Winston Sung using patch(es) lin... [04:22:56] (HAProxyEdgeTrafficDrop) firing: (5) 48% request drop in text@codfw during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://alerts.wikimedia.org/?q=alertname%3DHAProxyEdgeTrafficDrop [04:27:56] (HAProxyEdgeTrafficDrop) resolved: (6) 53% request drop in text@codfw during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://alerts.wikimedia.org/?q=alertname%3DHAProxyEdgeTrafficDrop [07:18:39] !log Manually rerun webrequest_text laod for hour 2022-06-12T08:00 [07:18:42] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:18:46] Arf sorry wrong chan [13:40:07] 10Traffic, 10DC-Ops, 10SRE, 10ops-eqiad: cp1089 memory errors on DIMM_B1 - https://phabricator.wikimedia.org/T310387 (10SLyngshede-WMF) p:05Triage→03Medium [17:28:07] 10Traffic, 10Thumbor: Thumbor URLs are too permissive - https://phabricator.wikimedia.org/T310528 (10AntiCompositeNumber) The last part of URLs should already be normalized by the upload-frontend proxy, {https://phabricator.wikimedia.org/source/operations-puppet/browse/production/modules/varnish/templates/uplo... [18:52:30] 10Traffic, 10SRE, 10Thumbor: Thumbor URLs are too permissive - https://phabricator.wikimedia.org/T310528 (10TheDJ) This shouldn't be a problem as long as MediaWiki only generates url fragments that are lowercase (which is what it should be doing). In general, thumbor is a tad more permissive than MediaWiki (... [22:30:15] 10Traffic, 10SRE: pontoon.traffic.eqiad1.wikimedia.cloud unable to run puppet agent due to certificate mismatch - https://phabricator.wikimedia.org/T310303 (10BCornwall) @ssingh @KOfori Is there a need/desire to have these three instances around? If so, is there any objection to following the above and termin... [22:37:53] 10Traffic, 10SRE: pontoon.traffic.eqiad1.wikimedia.cloud unable to run puppet agent due to certificate mismatch - https://phabricator.wikimedia.org/T310303 (10Dzahn) I also saw certificate errors pop up in a different project that uses a local puppetmaster. And we felt like we had not touched anything. Did not... [22:55:10] 10Traffic, 10SRE: fawiki user reports getting 503 errors with message "upstream connect error or disconnect before headers" - https://phabricator.wikimedia.org/T310450 (10Huji) Likely. But the point about an error message shown which appears to only exist in unit test code is also worth investigating.