[09:20:00] 10Traffic, 10SRE: purged issues while kafka brokers are restarted - https://phabricator.wikimedia.org/T334078 (10Vgutierrez) p:05Triage→03High We had two servers (cp1089 and cp3069) having purged issues over the weekend, after losing connection to the kafka cluster and logging: ` Oct 28 05:19:11 cp1089 pur... [10:15:59] 10Traffic, 10SRE: purged issues while kafka brokers are restarted - https://phabricator.wikimedia.org/T334078 (10Fabfur) Adding, for complete information, that the list of hosts impacted with the same purged error this weekend were: - cp1078 - cp1089 - cp6005 - cp3069 [11:03:21] 10Traffic, 10SRE: purged issues while kafka brokers are restarted - https://phabricator.wikimedia.org/T334078 (10Vgutierrez) We need to work on purged Kafka consumer. I've already spotted the issue on our codebase [12:38:02] 10Traffic, 10Infrastructure-Foundations, 10Puppet-Infrastructure, 10SRE, and 2 others: find solution for acmechief in puppet7 - https://phabricator.wikimedia.org/T349915 (10jbond) [14:54:33] 10Traffic, 10DNS, 10SRE: DNS Update, Google Postmaster Tools - https://phabricator.wikimedia.org/T349942 (10NMariano-WMF) The ITS System team will set this up and manage permissions for Noah Israel (@nisrae)l and Danny Bu (@DBu-WMF). [15:49:50] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate mr1-codfw from asw-a1-codfw to lsw1-a1-codfw - https://phabricator.wikimedia.org/T348164 (10Papaul) @cmooney cable is place from mr1-codfw ge0/0/3 to lsw1-a2-codfw ge-0/0/47 ID 00745 [15:51:46] 10Traffic, 10SRE: Q1:Install cp11[00-15] and rotate into production - https://phabricator.wikimedia.org/T349244 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by fabfur@cumin1001 for host cp1103.eqiad.wmnet with OS bullseye [16:07:44] 10Traffic, 10SRE: Q1:Install cp11[00-15] and rotate into production - https://phabricator.wikimedia.org/T349244 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by fabfur@cumin1001 for host cp1103.eqiad.wmnet with OS bullseye executed with errors: - cp1103 (**FAIL**) - Removed from Puppet... [16:08:02] 10Traffic, 10SRE: Q1:Install cp11[00-15] and rotate into production - https://phabricator.wikimedia.org/T349244 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by fabfur@cumin1001 for host cp1103.eqiad.wmnet with OS bullseye [16:26:44] 10Traffic, 10DNS, 10SRE: DNS Update, Google Postmaster Tools - https://phabricator.wikimedia.org/T349942 (10ssingh) Hi, this is for wikimedia.org, correct? [16:28:28] 10Traffic, 10DNS, 10SRE: DNS Update, Google Postmaster Tools - https://phabricator.wikimedia.org/T349942 (10NMariano-WMF) Correct [16:56:58] 10Traffic, 10DNS, 10SRE, 10Patch-For-Review: DNS Update, Google Postmaster Tools - https://phabricator.wikimedia.org/T349942 (10ssingh) 05Open→03Resolved a:03ssingh wikimedia.org. 600 IN TXT "google-site-verification=uzfgD0YiIqSQgRdSQXlkA7NByyyOZDp-n0SZ3nozpDM" [17:12:31] 10Traffic, 10SRE, 10Patch-For-Review: Upgrade Traffic hosts to bookworm - https://phabricator.wikimedia.org/T342154 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by brett@cumin2002 for host dns3003.wikimedia.org with OS bookworm [17:16:52] 10Traffic, 10SRE: Q1:Install cp11[00-15] and rotate into production - https://phabricator.wikimedia.org/T349244 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by fabfur@cumin1001 for host cp1103.eqiad.wmnet with OS bullseye executed with errors: - cp1103 (**FAIL**) - Removed from Puppet... [17:53:41] 10Traffic, 10API Platform, 10MediaWiki-REST-API, 10SRE, and 2 others: Use relative URLs in redirects emitted by rest.php - https://phabricator.wikimedia.org/T349001 (10daniel) 05Open→03Resolved a:03daniel [18:10:52] 10Traffic, 10SRE: Q1:Install cp11[00-15] and rotate into production - https://phabricator.wikimedia.org/T349244 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by fabfur@cumin1001 for host cp1103.eqiad.wmnet with OS bullseye [18:22:37] 10Traffic, 10SRE: Q1:Install cp11[00-15] and rotate into production - https://phabricator.wikimedia.org/T349244 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by fabfur@cumin1001 for host cp1103.eqiad.wmnet with OS bullseye executed with errors: - cp1103 (**FAIL**) - Downtimed on Icinga/... [18:38:26] 10Traffic, 10SRE, 10Patch-For-Review: Upgrade Traffic hosts to bookworm - https://phabricator.wikimedia.org/T342154 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by brett@cumin2002 for host dns3003.wikimedia.org with OS bookworm completed: - dns3003 (**PASS**) - Downtimed on Icinga/Al... [19:21:17] 10Traffic, 10SRE, 10Patch-For-Review: Upgrade Traffic hosts to bookworm - https://phabricator.wikimedia.org/T342154 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by brett@cumin2002 for host dns3004.wikimedia.org with OS bookworm [20:21:01] 10Traffic, 10SRE, 10Patch-For-Review: Upgrade Traffic hosts to bookworm - https://phabricator.wikimedia.org/T342154 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by brett@cumin2002 for host dns3004.wikimedia.org with OS bookworm completed: - dns3004 (**PASS**) - Downtimed on Icinga/Al... [20:50:32] 10Traffic, 10SRE, 10Patch-For-Review: Upgrade Traffic hosts to bookworm - https://phabricator.wikimedia.org/T342154 (10BCornwall)