[06:21:52] 10SRE-tools, 10Infrastructure-Foundations, 10SRE, 10serviceops-radar: SVC DNS zonefiles and source of truth - https://phabricator.wikimedia.org/T270071 (10akosiaris) >>! In T270071#7371192, @Volans wrote: > Has been a while since we discussed this but the problem still stands and I think we need to get som... [07:42:53] 10Mail, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: Upgrade MXes to Bullseye - https://phabricator.wikimedia.org/T286911 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by jmm@cumin2002 for hosts: `mx1002.wikimedia.org` - mx1002.wikimedia.org (**WARN**) - //Host not found on... [07:53:28] in terms of the fastest network connection to either codfw/eqiad, do we have some data what's preferable for esams/ulsfo/eqsin? (internal smarthosts currently all point to mx1001, but if codfw is preferable to either edge site we can set the Hiera flag to prefer mx2001 over mx1001) [08:09:01] 10Mail, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: Upgrade MXes to Bullseye - https://phabricator.wikimedia.org/T286911 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by jmm@cumin2002 for hosts: `mx2002.wikimedia.org` - mx2002.wikimedia.org (**PASS**) - Downtimed host on I... [08:20:25] 10SRE-tools, 10Infrastructure-Foundations: Introduce Spicerack.kafka module, along with the method to transfer offset state between consumer groups and clusters - https://phabricator.wikimedia.org/T291681 (10Zbyszko) [08:21:40] 10SRE-tools, 10Infrastructure-Foundations: Introduce Spicerack.kafka module, along with the method to transfer offset state between consumer groups and clusters - https://phabricator.wikimedia.org/T291681 (10Zbyszko) Configuration is added in this patch - https://gerrit.wikimedia.org/r/c/operations/puppet/+/72... [08:23:00] 10SRE-tools, 10Infrastructure-Foundations: Introduce Spicerack.kafka module, along with the method to transfer offset state between consumer groups and clusters - https://phabricator.wikimedia.org/T291681 (10Zbyszko) [08:31:35] 10SRE-tools, 10Infrastructure-Foundations: sre.decom.host fails if the mgmt interface's DNS records have already been removed - https://phabricator.wikimedia.org/T268965 (10Volans) 05Open→03Resolved The cookbook is now falling back to the asset-tag based management record in case the hostname-based one fai... [08:45:42] 10Puppet, 10Infrastructure-Foundations: Temporary failures for prometheus_puppet_agent_stats - https://phabricator.wikimedia.org/T290726 (10fgiunchedi) 05Open→03Resolved a:03fgiunchedi Yes @jbond I'll resolve this! Let's followup re: removing git_sha altogether from prometheus metrics since it is in logs... [09:49:05] 10netops, 10Data-Services, 10Infrastructure-Foundations, 10SRE, and 2 others: wikireplicas last-minute infra work to discuss / resolve - https://phabricator.wikimedia.org/T273248 (10Marostegui) p:05High→03Medium Can this be closed? [09:57:34] 10SRE-tools, 10Infrastructure-Foundations, 10SRE, 10observability: HP RAID failed on ms-be1054 didn't open a task - https://phabricator.wikimedia.org/T269563 (10Marostegui) 05Open→03Resolved a:03jbond I am closing this for now - reopen if it is not fixed [10:50:45] 10Mail, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: Upgrade MXes to Bullseye - https://phabricator.wikimedia.org/T286911 (10MoritzMuehlenhoff) The two VMs (mx1002/mx2002) which were used to test the Bullseye setup have been taken down. [13:46:39] moritzm: you should prefer codfw in ulsfo and eqsin [13:46:58] https://wikitech.wikimedia.org/wiki/Network_design#/media/File:Wikimedia_network_overview.png [13:47:21] not a huge difference, but measureable :) [20:20:00] cdanis: ack, thx