[00:30:57] 10HTTPS, 10SRE, 10Traffic-Icebox, 10Upstream: Support ECH on Wikimedia servers - https://phabricator.wikimedia.org/T205378 (10ssingh) [00:31:05] 10Traffic, 10SRE, 10Epic: Deploy Wikimedia DNS: DNS-over-HTTPS (DoH) and DNS-over-TLS (DoT) public resolver - https://phabricator.wikimedia.org/T252132 (10ssingh) [06:34:05] 10Traffic, 10Anti-Harassment, 10Data-Engineering, 10SRE, and 2 others: Include User-Agent Client Hints in WebRequest logs - https://phabricator.wikimedia.org/T337947 (10kostajh) [08:56:39] vgutierrez, heads up, I'm about to start pushing 4% of traffic to mw-on-k8s, tell me if it's a bad time [08:57:35] not in terms of load, it's Friday though :) [08:58:22] Yeah I know, that's why I'm doing it in the morning while I'm on call :D [08:58:54] I'll send the usual "how to revert" email just in case something blows up [09:12:26] claime: ack [11:09:23] 10netops, 10Infrastructure-Foundations, 10SRE, 10SRE-tools, 10Patch-For-Review: Setup zero touch provisioning (ZTP) for network devices - https://phabricator.wikimedia.org/T336485 (10cmooney) Just an update. The cookbook is now working to both add the initial configuration and upgrade/downgrade the devi... [15:55:38] 10Traffic, 10SRE, 10Patch-For-Review: Upgrade Traffic hosts to bookworm - https://phabricator.wikimedia.org/T342154 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by brett@cumin2002 for host doh5001.wikimedia.org with OS bookworm [17:06:34] 10Traffic, 10SRE, 10Patch-For-Review: Upgrade Traffic hosts to bookworm - https://phabricator.wikimedia.org/T342154 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by brett@cumin2002 for host doh5001.wikimedia.org with OS bookworm executed with errors: - doh5001 (**FAIL**) - Downtimed o... [17:22:22] 10Traffic, 10SRE, 10Patch-For-Review: Upgrade Traffic hosts to bookworm - https://phabricator.wikimedia.org/T342154 (10BCornwall) [17:31:20] 10Traffic, 10SRE, 10Patch-For-Review: Upgrade Traffic hosts to bookworm - https://phabricator.wikimedia.org/T342154 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by brett@cumin2002 for host doh3004.wikimedia.org with OS bookworm [18:04:20] 10Traffic, 10SRE, 10Patch-For-Review: Upgrade Traffic hosts to bookworm - https://phabricator.wikimedia.org/T342154 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by brett@cumin2002 for host doh3004.wikimedia.org with OS bookworm executed with errors: - doh3004 (**FAIL**) - Downtimed o... [18:48:50] 10Traffic, 10SRE, 10Patch-For-Review: Upgrade Traffic hosts to bookworm - https://phabricator.wikimedia.org/T342154 (10BCornwall) [18:48:52] 10Traffic, 10SRE, 10Patch-For-Review: Upgrade Traffic hosts to bookworm - https://phabricator.wikimedia.org/T342154 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by brett@cumin2002 for host doh3003.wikimedia.org with OS bookworm [19:23:55] 10Traffic, 10SRE, 10Patch-For-Review: Upgrade Traffic hosts to bookworm - https://phabricator.wikimedia.org/T342154 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by brett@cumin2002 for host doh3003.wikimedia.org with OS bookworm executed with errors: - doh3003 (**FAIL**) - Downtimed o... [20:46:20] 10netops, 10Infrastructure-Foundations, 10SRE: Plan codfw row A/B top-of-rack switch refresh - https://phabricator.wikimedia.org/T327938 (10cmooney) @papaul thanks for the work documenting the cable IDs. I've put the ones from above in Netbox now. There is one discrepancy, the same label is listed for two... [21:29:45] 10Traffic, 10SRE, 10Patch-For-Review: Upgrade Traffic hosts to bookworm - https://phabricator.wikimedia.org/T342154 (10BCornwall) [21:32:17] 10Traffic, 10SRE, 10Patch-For-Review: Upgrade Traffic hosts to bookworm - https://phabricator.wikimedia.org/T342154 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by brett@cumin2002 for host doh4002.wikimedia.org with OS bookworm [21:54:59] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Upgrade new codfw switches to Juniper recommended - https://phabricator.wikimedia.org/T341670 (10cmooney) 05Open→03Resolved a:03cmooney All are now upgraded to JUNOS 22.2R3.15. I used the opportunity to test the ZTP cookbook which is workin... [21:55:07] 10netops, 10Infrastructure-Foundations, 10SRE: TLS certificates for network devices - https://phabricator.wikimedia.org/T334594 (10cmooney) [21:55:15] 10netops, 10Infrastructure-Foundations, 10SRE: Plan codfw row A/B top-of-rack switch refresh - https://phabricator.wikimedia.org/T327938 (10cmooney) [22:02:29] 10Traffic, 10SRE, 10Patch-For-Review: Upgrade Traffic hosts to bookworm - https://phabricator.wikimedia.org/T342154 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by brett@cumin2002 for host doh4002.wikimedia.org with OS bookworm executed with errors: - doh4002 (**FAIL**) - Downtimed o... [22:04:42] (SystemdUnitFailed) firing: anycast-healthchecker.service Failed on doh4002:9100- https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [22:09:42] (SystemdUnitFailed) firing: (2) anycast-healthchecker.service Failed on doh4002:9100- https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [22:19:42] (SystemdUnitFailed) resolved: (2) anycast-healthchecker.service Failed on doh4002:9100- https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [22:22:32] 10Traffic, 10SRE, 10Patch-For-Review: Upgrade Traffic hosts to bookworm - https://phabricator.wikimedia.org/T342154 (10BCornwall)