[01:19:39] 06Traffic: Handle exception raised when OCSP response is an error - https://phabricator.wikimedia.org/T435104#12239995 (10BCornwall) The issue we had stems from the existing certificate having been issued from before google pulled OCSP support, so it was checking for a non-existent property. While technically th... [05:55:46] 06Traffic, 06Commons: Varnish cache purges of file assets sometimes seem to be failing, at least sometimes - https://phabricator.wikimedia.org/T435283#12240215 (10Bawolff) >>! In T435283#12239059, @ssingh wrote: > Thanks for reporting. This has been reported to us in other tasks as well and while they resolved... [06:24:18] 06Traffic, 06Commons: Varnish cache purges of file assets sometimes seem to be failing, at least sometimes - https://phabricator.wikimedia.org/T435283#12240228 (10Bawolff) One thing to note here, based on mediawiki logs, MW is issuing purges for the following URLs: ` https://upload.wikimedia.org/wikipedia/com... [08:34:58] 06Traffic, 06Infrastructure-Foundations, 06ServiceOps, 06SRE, 13Patch-For-Review: Scaling urldownloaders by adding redundancy and load balancing - https://phabricator.wikimedia.org/T429175#12240442 (10MoritzMuehlenhoff) There's two still things, that need to happen, let me reopen the task until they... [09:00:26] 10netops, 06Infrastructure-Foundations, 06SRE: Aug 2026: Packet loss on new HE transports to magru - https://phabricator.wikimedia.org/T435543#12240494 (10cmooney) [09:03:19] 10netops, 06Infrastructure-Foundations, 06SRE: Aug 2026: Packet loss on new HE transports to magru - https://phabricator.wikimedia.org/T435543#12240509 (10cmooney) The HE NOC came back to advise they had several sub sea fibre breaks at once which affected capacity to the region. ` On Thu, 20 Aug 2026 a... [09:21:35] 10netops, 06Infrastructure-Foundations, 06SRE: Aug 2026: Packet loss on new HE transports to magru - https://phabricator.wikimedia.org/T435543#12240542 (10cmooney) [09:23:44] 10netops, 06Infrastructure-Foundations, 06SRE: Aug 2026: Packet loss on new HE transports to magru - https://phabricator.wikimedia.org/T435543#12240544 (10cmooney) [09:38:36] 10netops, 06Infrastructure-Foundations, 06SRE: Power alert for cr2-eqiad old line cards - https://phabricator.wikimedia.org/T435506#12240576 (10cmooney) 05Open→03Resolved Closing, cr2-eqiad still seems happy. Though as discussed on irc with @robh it is somewhat worrying these both alarmed at the sam... [10:05:36] 10netops, 06Infrastructure-Foundations, 06SRE: Alert on packet loss to core sites from POPs - https://phabricator.wikimedia.org/T435592 (10cmooney) 03NEW p:05Triage→03Medium [10:11:11] 10netops, 06Infrastructure-Foundations, 06SRE: Alert on packet loss to core sites from POPs - https://phabricator.wikimedia.org/T435592#12240731 (10cmooney) [10:12:33] 10netops, 06Infrastructure-Foundations, 06SRE: Alert on packet loss to core sites from POPs - https://phabricator.wikimedia.org/T435592#12240734 (10cmooney) [14:30:28] 06Traffic, 13Patch-For-Review: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#12241480 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cdobbins@cumin1003 for host dns5004.wikimedia.org with OS trixie [15:57:04] 06Traffic, 13Patch-For-Review: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#12241939 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cdobbins@cumin1003 for host dns5004.wikimedia.org with OS trixie completed: - dns5004 (**FAIL**) - Downtimed on Icinga/Ale... [15:57:05] 06Traffic, 13Patch-For-Review: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#12241940 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cdobbins@cumin1003 for host dns5004.wikimedia.org with OS trixie executed with errors: - dns5004 (**FAIL**) - Downtimed on... [16:12:58] 06Traffic, 06MediaWiki-Media-Platform-Team, 10Thumbor, 13Patch-For-Review: Add `thumb_generated` from X-Analytics to webrequest_sampled_live - https://phabricator.wikimedia.org/T435634#12242025 (10Ladsgroup) I don't know who owns turnilo and how this can be deployed. [16:16:21] 06Traffic, 06MediaWiki-Media-Platform-Team, 10Thumbor, 13Patch-For-Review: Add `thumb_generated` from X-Analytics to webrequest_sampled_live - https://phabricator.wikimedia.org/T435634#12242038 (10Ladsgroup) Probably more patches needed btw. [16:26:37] 10netops, 06Traffic, 06Infrastructure-Foundations, 06SRE: BFD session fails from Anycast hosts over IPv6 on boot - https://phabricator.wikimedia.org/T434806#12242098 (10cmooney) So actually I may have spoken too soon here. We've had a couple of DNS hosts reimaged/rebooted in the past few days, and none of... [17:38:23] 06Traffic, 10Observability-Alerting: urldownloader service should be a paging alert - https://phabricator.wikimedia.org/T435648 (10ssingh) 03NEW [17:39:01] 06Traffic, 06Infrastructure-Foundations, 06ServiceOps, 06SRE, 13Patch-For-Review: Scaling urldownloaders by adding redundancy and load balancing - https://phabricator.wikimedia.org/T429175#12242324 (10ssingh) 05Resolved→03Open [17:39:02] 06Traffic, 10Observability-Alerting: urldownloader service should be a paging alert - https://phabricator.wikimedia.org/T435648#12242325 (10ssingh) [17:39:10] 06Traffic, 06Infrastructure-Foundations, 06ServiceOps, 06SRE, 13Patch-For-Review: Scaling urldownloaders by adding redundancy and load balancing - https://phabricator.wikimedia.org/T429175#12242326 (10ssingh) [19:12:56] 06Traffic, 06DC-Ops, 10ops-drmrs: hw troubleshooting: Memory failure for cp6008.drmrs.wmnet - https://phabricator.wikimedia.org/T431651#12242534 (10RobH) Updates: * Chatted with a rep today via 1-800-225-5345 * The shipment landed in France on August 7th - https://mydhl.express.dhl/us/en/tracking.html#/resu... [19:20:57] 06Traffic, 06DC-Ops, 10ops-drmrs: hw troubleshooting: Memory failure for cp6008.drmrs.wmnet - https://phabricator.wikimedia.org/T431651#12242550 (10RobH) DHL just called me back the case is open and they have a request into DHL france delivery directly. They should have an update by Monday August 24th no la...