[06:03:21] 10netops, 06Infrastructure-Foundations, 06SRE, 06Traffic, 13Patch-For-Review: BFD session fails from Anycast hosts over IPv6 on boot - https://phabricator.wikimedia.org/T434806#12224818 (10ayounsi) We should reach out to the BIRD team, even if there is a race condition, BIRD's BFD probably needs to come... [06:06:25] FIRING: SystemdUnitFailed: dump_proxy_ranges.service on puppetserver1003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [06:30:46] https://github.com/netbox-community/netbox/releases/tag/v4.7.0-beta1 [06:31:03] so many breaking changes [06:56:15] it's not a breaking change, if you don't update [06:56:26] follow me for more inspiring enterprise IT advice! [07:02:32] :) [10:06:40] FIRING: SystemdUnitFailed: dump_proxy_ranges.service on puppetserver1003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [10:47:46] 10SRE-tools, 10Cumin, 06DC-Ops, 06Infrastructure-Foundations, and 2 others: add dcops group to run sre.hosts.downtime cookbook - https://phabricator.wikimedia.org/T433409#12225729 (10JMeybohm) 05Open→03Resolved >>! In T433409#12216045, @JMeybohm wrote: > From the sudo rule added I would assume you... [11:57:33] 10CAS-SSO, 06Infrastructure-Foundations: Change database encoding for CAS database - https://phabricator.wikimedia.org/T435191 (10SLyngshede-WMF) 03NEW [12:00:51] 10CAS-SSO, 06Infrastructure-Foundations: Change database encoding for CAS database - https://phabricator.wikimedia.org/T435191#12226000 (10SLyngshede-WMF) p:05Triage→03Medium [12:01:18] 10CAS-SSO, 06Data-Persistence: Change database encoding for CAS database - https://phabricator.wikimedia.org/T435191#12226001 (10SLyngshede-WMF) [12:02:14] 07Puppet, 06Collaboration-Services, 10Gerrit, 06Infrastructure-Foundations, 13Patch-For-Review: Change puppet-merge git origin to use gerrit.discovery.wmnet instead of gerrit.wikimedia.org - https://phabricator.wikimedia.org/T420184#12226002 (10ABran-WMF) [12:06:25] RESOLVED: SystemdUnitFailed: dump_proxy_ranges.service on puppetserver1003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [12:08:00] 10Mail, 06Infrastructure-Foundations: Failed email communication for an User account - https://phabricator.wikimedia.org/T434744#12226009 (10LSobanski) a:03jhathaway [12:08:27] 10Mail, 06Infrastructure-Foundations: Failed email communication for an User account - https://phabricator.wikimedia.org/T434744#12226012 (10LSobanski) p:05Triage→03Medium [12:08:45] 10Mail, 06Infrastructure-Foundations: Failed email communication for an User account - https://phabricator.wikimedia.org/T434744#12226016 (10LSobanski) 05Open→03In progress [14:01:40] 10CAS-SSO, 06cloud-services-team, 10Striker, 06tools-platform-team, 13Patch-For-Review: Use IDP for authentication in Striker - https://phabricator.wikimedia.org/T359554#12226452 (10Arendpieter) [14:01:55] 10CAS-SSO, 06cloud-services-team, 10Striker, 06tools-platform-team, 13Patch-For-Review: Use IDP for authentication in Striker - https://phabricator.wikimedia.org/T359554#12226456 (10Arendpieter) 05Open→03In progress [14:08:21] 10netops, 10fundraising-tech-ops, 06Infrastructure-Foundations, 06SRE: Upgrade JunOS on pfw1-eqiad and pfw1-codfw - https://phabricator.wikimedia.org/T434865#12226495 (10cmooney) >>! In T434865#12221432, @Jgreen wrote: > @cmooney this happens to be a FR maintenance week, is it feasible to plan it sometime... [14:19:46] 10Mail, 06Infrastructure-Foundations: Email sent from wikipedia UI seems to use nondeliverable sender: 550 Administrative prohibition - https://phabricator.wikimedia.org/T207650#12226634 (10LSobanski) 05Open→03Resolved Considering the date of the last update I'm resolving this task, please reopen if th... [14:20:17] 10Mail, 06Infrastructure-Foundations: Troubleshooting Mail Delivery Issues from Coupa - https://phabricator.wikimedia.org/T306472#12226641 (10LSobanski) 05Open→03Declined [14:28:40] 10Mail, 06Infrastructure-Foundations: Ensure Wikimedia complies with Google's 2024 email sender guidelines - https://phabricator.wikimedia.org/T355712#12226715 (10LSobanski) a:05jhathaway→03None [14:30:42] 10Mail, 06Infrastructure-Foundations, 06SRE, 10Wikimedia-Mailing-lists, 07Upstream: lists.wikimedia.org - adhere to RFC8048 (one-click unsubscribe) dkim guidelines - https://phabricator.wikimedia.org/T355802#12226738 (10LSobanski) a:05jhathaway→03None [18:25:19] 10netops, 10fundraising-tech-ops, 06Infrastructure-Foundations, 06SRE: Upgrade JunOS on pfw1-eqiad and pfw1-codfw - https://phabricator.wikimedia.org/T434865#12228113 (10cmooney) As discussed on irc I'll kick off the upgrade on pfw1-codfw at 09:00am UTC tomrorow (Wed Aug 19th).