[12:41:24] 10netops, 06Infrastructure-Foundations, 06SRE: rancid: message has lines too long for transport - https://phabricator.wikimedia.org/T410606#11419139 (10MoritzMuehlenhoff) Rancid is a bit of a maze of scripts calling each other, but I could eventually track it down to /usr/bin/control_rancid. In our case, the... [13:17:14] 10netops, 06Infrastructure-Foundations, 06SRE: rancid: message has lines too long for transport - https://phabricator.wikimedia.org/T410606#11419237 (10MoritzMuehlenhoff) Also reported to Debian as https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1121730 [13:25:31] 10netops, 06Infrastructure-Foundations, 06SRE: rancid: message has lines too long for transport - https://phabricator.wikimedia.org/T410606#11419246 (10MoritzMuehlenhoff) 05Open→03Resolved a:03MoritzMuehlenhoff Updates have been rolled out and diffs are being sent again. [13:46:56] 10SRE-tools, 06Data-Persistence, 06Infrastructure-Foundations, 06serviceops-radar: Automation to change a server's vlan - https://phabricator.wikimedia.org/T350152#11419328 (10ayounsi) 05Open→03Resolved a:03ayounsi I think this is all done, we have cookbooks in place. [13:56:20] 10Mail, 06Infrastructure-Foundations, 06SRE: Emails to Google group no-reply@wikimedia.org are not being delivered - SMTP server issue? - https://phabricator.wikimedia.org/T411027#11419367 (10JKelsoteel-WMF) Just an update - Noah (the requester) needs the email address by December 3. [14:02:10] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-magru, and 2 others: MAGRU power maint - CHG0262056 - October 29-30, 2025 - https://phabricator.wikimedia.org/T408589#11419402 (10ayounsi) 05Open→03Resolved a:03ayounsi I guess we're good here. [15:29:16] 10Mail, 06Infrastructure-Foundations, 06SRE: Emails to Google group no-reply@wikimedia.org are not being delivered - SMTP server issue? - https://phabricator.wikimedia.org/T411027#11419795 (10jhathaway) p:05Triage→03Medium [15:29:55] 10netops, 06Infrastructure-Foundations: Remove second network connection for cloudcephosd hosts with single uplink - https://phabricator.wikimedia.org/T410989#11419801 (10ayounsi) p:05Triage→03Low [15:31:19] 10SRE-tools, 06Infrastructure-Foundations, 06SRE, 06Traffic: Reboot cookbook workflow leaves Puppet disabled - https://phabricator.wikimedia.org/T410944#11419817 (10elukey) p:05Triage→03Medium [16:24:50] 10Mail, 06Infrastructure-Foundations, 06SRE: Emails to Google group no-reply@wikimedia.org are not being delivered - SMTP server issue? - https://phabricator.wikimedia.org/T411027#11420022 (10jhathaway) @JKelsoteel-WMF the addresses `no-reply` or `noreply` are used to indicate that the sender does not expect... [17:06:14] 10SRE-tools, 06Infrastructure-Foundations, 06SRE, 07IPv6: Some Foundation clusters do not appear to support IPv6 - https://phabricator.wikimedia.org/T271136#11420281 (10Dzahn) epic !:) [17:09:36] 10Mail, 06Infrastructure-Foundations, 06SRE: Emails to Google group no-reply@wikimedia.org are not being delivered - SMTP server issue? - https://phabricator.wikimedia.org/T411027#11420300 (10JKelsoteel-WMF) Hi @jhathaway, Noah's team is sending out a high-visibility/high-priority email to donors, and for th... [17:29:46] 10Mail, 06Infrastructure-Foundations, 06SRE: Emails to Google group no-reply@wikimedia.org are not being delivered - SMTP server issue? - https://phabricator.wikimedia.org/T411027#11420456 (10JKelsoteel-WMF) @jhathaway I did create a group called "noreply@wikimedia.org" to see if messages to that address wer... [17:51:05] 10Mail, 06Infrastructure-Foundations, 06SRE: Emails to Google group no-reply@wikimedia.org are not being delivered - SMTP server issue? - https://phabricator.wikimedia.org/T411027#11420579 (10jhathaway) >>! In T411027#11420456, @JKelsoteel-WMF wrote: > @jhathaway I did create a group called "noreply@wikimedi... [18:29:55] 10CAS-SSO, 06cloud-services-team, 10Striker, 13Patch-For-Review: Use IDP for authentication in Striker - https://phabricator.wikimedia.org/T359554#11420828 (10Arendpieter) >>! In T359554#11374327, @taavi wrote: > > This is pending a response to https://gerrit.wikimedia.org/r/c/labs/striker/+/1189915/comme... [19:33:25] FIRING: SystemdUnitFailed: update-ubuntu-mirror.service on mirror1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [23:33:40] FIRING: SystemdUnitFailed: update-ubuntu-mirror.service on mirror1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed