[06:39:39] <wikibugs>	 10netops, 06Infrastructure-Foundations: HE transport eqiad-eqsin packet loss - october 2026 #2 or #3 ? - https://phabricator.wikimedia.org/T442591#12487905 (10ayounsi) >  We are seeing congestion between Marseille and Singapore due to 3rd party fiber circuit maintenance.  They have routed eqsin-eqiad through S...
[11:42:50] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06SRE, 06Traffic: Mark all traffic from upload.wikimedia.org as low-priority for QoS - https://phabricator.wikimedia.org/T440990#12488873 (10cmooney) Just to document the results here this has had a pronounced affect on the profiling of traffic on our internet ed...
[12:13:01] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06SRE, 06Traffic: Mark all traffic from upload.wikimedia.org as low-priority for QoS - https://phabricator.wikimedia.org/T440990#12489222 (10cmooney) Actually @ayounsi alerted me to a better stat to share.  We actually had a saturation incident earlier today on o...
[12:21:51] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06SRE, 06Traffic: Mark all traffic from upload.wikimedia.org as low-priority for QoS - https://phabricator.wikimedia.org/T440990#12489247 (10BBlack) Super-cool outcome, nice work all!
[12:48:37] <wikibugs>	 10netops, 06Infrastructure-Foundations, 07Wikimedia-Incident: Temporary loss of connectivity to eqiad row A - https://phabricator.wikimedia.org/T442705#12489392 (10MLechvien-WMF) Could we attach a [[ https://wikitech.wikimedia.org/wiki/Incident_response/Incident_severities | severity ]] tag to this task?
[12:50:39] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06SRE, 06Traffic: Mark all traffic from upload.wikimedia.org as low-priority for QoS - https://phabricator.wikimedia.org/T440990#12489400 (10Fabfur) Very nice to hear this, good job!
[16:38:29] <wikibugs>	 10Packaging, 10SRE-tools, 06Infrastructure-Foundations: Upgrade prometheus-atlas-exporter - https://phabricator.wikimedia.org/T429672#12490988 (10Lizimedia) a:03Lizimedia
[17:40:40] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06SRE, 06Traffic: Mark all traffic from upload.wikimedia.org as low-priority for QoS - https://phabricator.wikimedia.org/T440990#12491346 (10ssingh) Great work all around, folks! <3
[20:02:41] <wikibugs>	 10Mail, 10fundraising-tech-ops, 06Infrastructure-Foundations: mx-out* relay access denied for fundraising subnets - https://phabricator.wikimedia.org/T442940 (10Jgreen) 03NEW p:05Triage→03Unbreak!
[20:24:23] <wikibugs>	 10Mail, 10fundraising-tech-ops, 06Infrastructure-Foundations: mx-out* relay access denied for fundraising subnets - https://phabricator.wikimedia.org/T442940#12492254 (10Jgreen) @jhathaway I think it could be this change?  ` commit f0cd9ba04de0b7938ca9839eff14fb36d82cd109 Author:     Jesse Hathaway <jhathawa...
[20:53:51] <wikibugs>	 10Mail, 10fundraising-tech-ops, 06Infrastructure-Foundations: mx-out* relay access denied for fundraising subnets - https://phabricator.wikimedia.org/T442940#12492343 (10LSobanski) The CR mentioned above is https://gerrit.wikimedia.org/r/c/operations/puppet/+/1358850 and the related task is {T442063}
[21:34:06] <wikibugs>	 10Mail, 10fundraising-tech-ops, 06Infrastructure-Foundations: mx-out* relay access denied for fundraising subnets - https://phabricator.wikimedia.org/T442940#12492479 (10BCornwall) @Jgreen, I've reverted the CR due to UBN and deployed to `crm2001.codfw.wmnet,mx-in[1001,2001].wikimedia.org,mx-out[1001,2001].w...
[21:34:16] <wikibugs>	 10Mail, 10fundraising-tech-ops, 06Infrastructure-Foundations: mx-out* relay access denied for fundraising subnets - https://phabricator.wikimedia.org/T442940#12492480 (10BCornwall) 05Open→03In progress a:03BCornwall
[22:31:38] <wikibugs>	 10Mail, 10fundraising-tech-ops, 06Infrastructure-Foundations: mx-out* relay access denied for fundraising subnets - https://phabricator.wikimedia.org/T442940#12492758 (10BCornwall) 05In progress→03Resolved Confirmed with @Jgreen that the revert fixed the issue. We'll follow up with re-implementation...
[22:52:06] <wikibugs>	 10Mail, 06Infrastructure-Foundations, 10MediaWiki-Email, 10MediaWiki-extensions-EmailAuth, and 9 others: Could not send confirmation email: Unknown error in PHP's mail() function. - https://phabricator.wikimedia.org/T383047#12492898 (10Quiddity) Noting a fresh bug-report of someone using the `Special:Email...