[11:36:31] 10netops, 06Infrastructure-Foundations, 06SRE, 06Traffic, 07Wikimedia-Incident: Wikimedia's eqsin datacenter (Asia Pacific) had network connectivity issues - https://phabricator.wikimedia.org/T284986#12307394 (10MLechvien-WMF) [11:38:10] 10netops, 06Infrastructure-Foundations, 06SRE, 07Wikimedia-Incident: Improve resiliency of the eqsin transport link - https://phabricator.wikimedia.org/T236878#12307437 (10MLechvien-WMF) [11:38:31] 10netops, 06Infrastructure-Foundations, 06SRE, 07Wikimedia-Incident: asw2-d2-eqiad crash - https://phabricator.wikimedia.org/T233645#12307447 (10MLechvien-WMF) [11:39:18] 10netops, 06Infrastructure-Foundations, 06SRE, 07Wikimedia-Incident: asw2-a-eqiad FPC5 gets disconnected every 10 minutes - https://phabricator.wikimedia.org/T201145#12307473 (10MLechvien-WMF) [13:17:45] 10Mail, 06Infrastructure-Foundations, 10MediaWiki-Email, 10MediaWiki-extensions-EmailAuth, and 9 others: Could not send confirmation email: Unknown error in PHP's mail() function. - https://phabricator.wikimedia.org/T383047#12307960 (10JTweed-WMF) [14:25:25] FIRING: SystemdUnitFailed: requestctl-credential-refresh.service on puppetserver1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [15:25:25] RESOLVED: SystemdUnitFailed: requestctl-credential-refresh.service on puppetserver1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [15:26:28] 👀 hm [16:14:57] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: eqiad row A/B switch refresh prep - https://phabricator.wikimedia.org/T418012#12309022 (10RobH) IRC Sync update: * All switches except the lsw and ssw in A8 are racked, connected to SCS * I've added columns for the switch fiber cable... [16:34:51] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: eqiad row A/B switch refresh prep - https://phabricator.wikimedia.org/T418012#12309116 (10cmooney) >>! In T418012#12309022, @RobH wrote: > * All switches except the lsw and ssw in A8 are racked, connected to SCS Can we confirm what po... [16:39:28] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: eqiad row A/B switch refresh prep - https://phabricator.wikimedia.org/T418012#12309140 (10RobH) >>! In T418012#12309116, @cmooney wrote: >>>! In T418012#12309022, @RobH wrote: >> * All switches except the lsw and ssw in A8 are racked,... [20:16:49] 10Mail, 06Infrastructure-Foundations, 10Observability-Logging: Allow IT Services to view inbound email logs - https://phabricator.wikimedia.org/T419906#12309896 (10jhathaway) Debian's Rsyslog now has support for open telemetry, https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1130542