[09:01:03] 10Traffic, 10SRE, 10SRE-swift-storage, 10Patch-For-Review: Revisit CDN<-->Swift communication - https://phabricator.wikimedia.org/T317616 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by mvernon@cumin2002 for host ms-fe2013.codfw.wmnet with OS bullseye [09:13:19] 10Acme-chief, 10Traffic: acme-chief service started on a passive node after reimage - https://phabricator.wikimedia.org/T351655 (10Vgutierrez) 05Open→03Resolved Fixed by masking the systemd service before acme-chief package is installed on passive hosts [09:25:31] 10Traffic, 10SRE, 10Patch-For-Review: Enable IPIP encapsulation for ncredir - https://phabricator.wikimedia.org/T351069 (10Vgutierrez) [09:30:36] 10Traffic, 10SRE, 10SRE-swift-storage: Revisit CDN<-->Swift communication - https://phabricator.wikimedia.org/T317616 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by mvernon@cumin2002 for host ms-fe2013.codfw.wmnet with OS bullseye completed: - ms-fe2013 (**PASS**) - Downtimed on Ici... [09:49:18] 10Traffic, 10SRE, 10SRE-swift-storage: Revisit CDN<-->Swift communication - https://phabricator.wikimedia.org/T317616 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by mvernon@cumin1001 for host ms-fe1013.eqiad.wmnet with OS bullseye completed: - ms-fe1013 (**PASS**) - Downtimed on Ici... [11:35:40] 10Traffic, 10SRE, 10SRE-swift-storage, 10Patch-For-Review: Revisit CDN<-->Swift communication - https://phabricator.wikimedia.org/T317616 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by mvernon@cumin1001 for host ms-fe1012.eqiad.wmnet with OS bullseye [11:35:55] 10Traffic, 10SRE, 10SRE-swift-storage, 10Patch-For-Review: Revisit CDN<-->Swift communication - https://phabricator.wikimedia.org/T317616 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by mvernon@cumin2002 for host ms-fe2012.codfw.wmnet with OS bullseye [12:02:00] 10Traffic, 10MW-on-K8s, 10SRE, 10serviceops, and 2 others: Move 25% of mediawiki external requests to mw on k8s - https://phabricator.wikimedia.org/T348122 (10Clement_Goubert) [12:05:38] 10Traffic, 10SRE, 10SRE-swift-storage, 10Patch-For-Review: Revisit CDN<-->Swift communication - https://phabricator.wikimedia.org/T317616 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by mvernon@cumin1001 for host ms-fe1012.eqiad.wmnet with OS bullseye completed: - ms-fe1012 (**PASS**... [12:10:21] 10Traffic, 10SRE, 10SRE-swift-storage, 10Patch-For-Review: Revisit CDN<-->Swift communication - https://phabricator.wikimedia.org/T317616 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by mvernon@cumin2002 for host ms-fe2012.codfw.wmnet with OS bullseye completed: - ms-fe2012 (**PASS**... [12:47:44] 10Traffic, 10SRE, 10SRE-swift-storage, 10Patch-For-Review: Revisit CDN<-->Swift communication - https://phabricator.wikimedia.org/T317616 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by mvernon@cumin1001 for host ms-fe1011.eqiad.wmnet with OS bullseye [12:48:00] 10Traffic, 10SRE, 10SRE-swift-storage, 10Patch-For-Review: Revisit CDN<-->Swift communication - https://phabricator.wikimedia.org/T317616 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by mvernon@cumin2002 for host ms-fe2011.codfw.wmnet with OS bullseye [13:17:35] 10Traffic, 10SRE, 10SRE-swift-storage, 10Patch-For-Review: Revisit CDN<-->Swift communication - https://phabricator.wikimedia.org/T317616 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by mvernon@cumin1001 for host ms-fe1011.eqiad.wmnet with OS bullseye completed: - ms-fe1011 (**PASS**... [13:21:33] 10Traffic, 10SRE, 10SRE-swift-storage, 10Patch-For-Review: Revisit CDN<-->Swift communication - https://phabricator.wikimedia.org/T317616 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by mvernon@cumin2002 for host ms-fe2011.codfw.wmnet with OS bullseye completed: - ms-fe2011 (**PASS**... [13:29:33] 10Traffic, 10SRE, 10SRE-swift-storage, 10Patch-For-Review: Revisit CDN<-->Swift communication - https://phabricator.wikimedia.org/T317616 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by mvernon@cumin1001 for host ms-fe1010.eqiad.wmnet with OS bullseye [13:29:47] 10Traffic, 10SRE, 10SRE-swift-storage, 10Patch-For-Review: Revisit CDN<-->Swift communication - https://phabricator.wikimedia.org/T317616 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by mvernon@cumin2002 for host ms-fe2010.codfw.wmnet with OS bullseye [13:59:09] 10Traffic, 10SRE, 10SRE-swift-storage, 10Patch-For-Review: Revisit CDN<-->Swift communication - https://phabricator.wikimedia.org/T317616 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by mvernon@cumin1001 for host ms-fe1010.eqiad.wmnet with OS bullseye completed: - ms-fe1010 (**PASS**... [14:03:14] 10Traffic, 10SRE, 10SRE-swift-storage, 10Patch-For-Review: Revisit CDN<-->Swift communication - https://phabricator.wikimedia.org/T317616 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by mvernon@cumin2002 for host ms-fe2010.codfw.wmnet with OS bullseye completed: - ms-fe2010 (**PASS**... [14:20:50] 10Traffic, 10SRE, 10SRE-swift-storage, 10Patch-For-Review: Revisit CDN<-->Swift communication - https://phabricator.wikimedia.org/T317616 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by mvernon@cumin1001 for host ms-fe1009.eqiad.wmnet with OS bullseye [14:21:13] 10Traffic, 10SRE, 10SRE-swift-storage, 10Patch-For-Review: Revisit CDN<-->Swift communication - https://phabricator.wikimedia.org/T317616 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by mvernon@cumin2002 for host ms-fe2009.codfw.wmnet with OS bullseye [14:36:18] 10Traffic, 10SRE, 10SRE-swift-storage, 10Patch-For-Review: Revisit CDN<-->Swift communication - https://phabricator.wikimedia.org/T317616 (10MatthewVernon) 05Stalled→03In progress [14:54:58] 10Traffic, 10SRE, 10SRE-swift-storage, 10Patch-For-Review: Revisit CDN<-->Swift communication - https://phabricator.wikimedia.org/T317616 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by mvernon@cumin1001 for host ms-fe1009.eqiad.wmnet with OS bullseye completed: - ms-fe1009 (**PASS**... [14:57:17] 10Traffic, 10SRE, 10SRE-swift-storage, 10Patch-For-Review: Revisit CDN<-->Swift communication - https://phabricator.wikimedia.org/T317616 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by mvernon@cumin2002 for host ms-fe2009.codfw.wmnet with OS bullseye completed: - ms-fe2009 (**WARN**... [15:04:45] 10Traffic, 10SRE, 10SRE-swift-storage, 10Patch-For-Review: Revisit CDN<-->Swift communication - https://phabricator.wikimedia.org/T317616 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by mvernon@cumin1001 for host moss-fe1001.eqiad.wmnet with OS bullseye [15:05:02] 10Traffic, 10SRE, 10SRE-swift-storage, 10Patch-For-Review: Revisit CDN<-->Swift communication - https://phabricator.wikimedia.org/T317616 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by mvernon@cumin2002 for host moss-fe2001.codfw.wmnet with OS bullseye [15:07:02] 10Traffic, 10DC-Ops, 10SRE, 10ops-eqiad: Q1:rack/setup/install cp11[00-15] - https://phabricator.wikimedia.org/T342159 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by fabfur@cumin1001 for hosts: `cp1113.eqiad.wmnet` - cp1113.eqiad.wmnet (**PASS**) - Downtimed host on Icinga/Alertmanag... [15:24:30] 10Traffic, 10SRE, 10SRE-swift-storage, 10Patch-For-Review: Revisit CDN<-->Swift communication - https://phabricator.wikimedia.org/T317616 (10MatthewVernon) [15:28:26] 10Traffic, 10SRE, 10SRE-swift-storage, 10Patch-For-Review: Revisit CDN<-->Swift communication - https://phabricator.wikimedia.org/T317616 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by mvernon@cumin2002 for host moss-fe2001.codfw.wmnet with OS bullseye executed with errors: - moss-f... [15:28:40] 10Traffic, 10SRE, 10SRE-swift-storage, 10Patch-For-Review: Revisit CDN<-->Swift communication - https://phabricator.wikimedia.org/T317616 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by mvernon@cumin2002 for host moss-fe2001.codfw.wmnet with OS bullseye [15:28:48] 10Traffic, 10SRE, 10SRE-swift-storage, 10Patch-For-Review: Revisit CDN<-->Swift communication - https://phabricator.wikimedia.org/T317616 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by mvernon@cumin1001 for host moss-fe1001.eqiad.wmnet with OS bullseye executed with errors: - moss-f... [15:28:59] 10Traffic, 10SRE, 10SRE-swift-storage, 10Patch-For-Review: Revisit CDN<-->Swift communication - https://phabricator.wikimedia.org/T317616 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by mvernon@cumin1001 for host moss-fe1001.eqiad.wmnet with OS bullseye [15:54:40] 10Traffic, 10SRE, 10SRE-swift-storage, 10Patch-For-Review: Revisit CDN<-->Swift communication - https://phabricator.wikimedia.org/T317616 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by mvernon@cumin2002 for host moss-fe2001.codfw.wmnet with OS bullseye executed with errors: - moss-f... [15:56:02] 10Traffic, 10SRE, 10SRE-swift-storage, 10Patch-For-Review: Revisit CDN<-->Swift communication - https://phabricator.wikimedia.org/T317616 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by mvernon@cumin2002 for host moss-fe2001.codfw.wmnet with OS bullseye [15:58:28] 10Traffic, 10SRE, 10SRE-swift-storage, 10Patch-For-Review: Revisit CDN<-->Swift communication - https://phabricator.wikimedia.org/T317616 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by mvernon@cumin1001 for host moss-fe1001.eqiad.wmnet with OS bullseye completed: - moss-fe1001 (**WA... [16:11:22] 10Traffic, 10DC-Ops, 10SRE, 10ops-eqiad: Q1:Install cp11[00-15] and rotate into production - https://phabricator.wikimedia.org/T349244 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by fabfur@cumin1001 for host cp1113.eqiad.wmnet with OS bullseye [16:14:02] 10Traffic, 10SRE, 10SRE-swift-storage, 10Patch-For-Review: Revisit CDN<-->Swift communication - https://phabricator.wikimedia.org/T317616 (10MatthewVernon) [16:18:14] 10Traffic, 10SRE, 10SRE-swift-storage, 10Patch-For-Review: Revisit CDN<-->Swift communication - https://phabricator.wikimedia.org/T317616 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by mvernon@cumin1001 for host ms-fe1014.eqiad.wmnet with OS bullseye [16:20:51] 10Traffic, 10SRE, 10SRE-swift-storage, 10Patch-For-Review: Revisit CDN<-->Swift communication - https://phabricator.wikimedia.org/T317616 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by mvernon@cumin2002 for host moss-fe2001.codfw.wmnet with OS bullseye executed with errors: - moss-f... [16:30:56] 10Traffic, 10SRE, 10SRE-swift-storage: Revisit CDN<-->Swift communication - https://phabricator.wikimedia.org/T317616 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by mvernon@cumin2002 for host moss-fe2001.codfw.wmnet with OS bullseye [16:34:33] 10Traffic, 10SRE, 10SRE-swift-storage: Revisit CDN<-->Swift communication - https://phabricator.wikimedia.org/T317616 (10MatthewVernon) (perhaps the moss-fe2001 puppet failures are due to T350809 ) [16:44:52] 10Traffic, 10SRE, 10SRE-swift-storage: Revisit CDN<-->Swift communication - https://phabricator.wikimedia.org/T317616 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by mvernon@cumin2002 for host moss-fe2001.codfw.wmnet with OS bullseye completed: - moss-fe2001 (**PASS**) - Downtimed on... [16:47:30] 10Traffic, 10SRE, 10SRE-swift-storage: Revisit CDN<-->Swift communication - https://phabricator.wikimedia.org/T317616 (10MatthewVernon) [16:47:54] 10Traffic, 10DC-Ops, 10SRE, 10ops-eqiad: Q1:Install cp11[00-15] and rotate into production - https://phabricator.wikimedia.org/T349244 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by fabfur@cumin1001 for host cp1113.eqiad.wmnet with OS bullseye completed: - cp1113 (**PASS**) - Remo... [16:55:17] 10Traffic, 10SRE, 10SRE-swift-storage: Revisit CDN<-->Swift communication - https://phabricator.wikimedia.org/T317616 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by mvernon@cumin1001 for host ms-fe1014.eqiad.wmnet with OS bullseye completed: - ms-fe1014 (**PASS**) - Downtimed on Ici... [16:57:48] 10Traffic, 10SRE, 10SRE-swift-storage: Revisit CDN<-->Swift communication - https://phabricator.wikimedia.org/T317616 (10MatthewVernon) [17:44:59] 10Traffic, 10MW-on-K8s, 10SRE, 10serviceops, 10Release-Engineering-Team (Seen): Serve production traffic via Kubernetes - https://phabricator.wikimedia.org/T290536 (10Clement_Goubert) [17:45:59] 10Traffic, 10MW-on-K8s, 10SRE, 10serviceops, 10Release-Engineering-Team (Seen): Move 25% of mediawiki external requests to mw on k8s - https://phabricator.wikimedia.org/T348122 (10Clement_Goubert) 05In progress→03Resolved a:03Clement_Goubert [18:10:30] I can't reach wikipedia... ping 185.15.59.224 gives me nothing. [18:10:44] traceroute stops after 62.214.42.186 [18:12:29] ah, looks like it'S fixed. Probably was just an intermediate routing issue at my provider [18:12:35] duesen: things look OK here fwiw so yeah [23:49:42] (SystemdUnitFailed) firing: export_smart_data_dump.service Failed on cp4037:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed