[10:25:09] <wikibugs>	 06Traffic, 06SRE, 06SRE Observability, 10SRE-SLO: Page on ATS backend errors relative to traffic - https://phabricator.wikimedia.org/T400675#12211488 (10fgiunchedi) Conversely, alerting on ratio will also cover low-traffic services that are erroring out just below the threshold. As it was the case today fo...
[10:33:44] <wikibugs>	 06Traffic, 06SRE, 06SRE Observability, 10SRE-SLO: Page on ATS backend errors relative to traffic - https://phabricator.wikimedia.org/T400675#12211522 (10fgiunchedi) Today also a swift.discovery.wmnet page came in, although IMHO it should have been a thumbor page instead. Swift itself was fine compared to i...
[11:18:48] <jinxer-wm>	 FIRING: PuppetConstantChange: Puppet performing a change on every puppet run on cp7009:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange
[11:33:55] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06SRE: Problems using Homer with SSH Agent on Debian 13 - https://phabricator.wikimedia.org/T434773 (10cmooney) 03NEW p:05Triage→03Medium
[11:45:04] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: Problems using Homer with SSH Agent on Debian 13 - https://phabricator.wikimedia.org/T434773#12211776 (10cmooney)
[12:26:10] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: Problems using Homer with SSH Agent on Debian 13 - https://phabricator.wikimedia.org/T434773#12211966 (10cmooney) p:05Medium→03Low
[13:30:45] <wikibugs>	 06Traffic, 06DC-Ops, 10decommission-hardware, 10ops-eqiad, 06SRE: decommission lvs10[13-15] - https://phabricator.wikimedia.org/T433508#12212220 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by sukhe@cumin1003 for hosts: `lvs1013.eqiad.wmnet` - lvs1013.eqiad.wmnet (**PASS**)   - Downti...
[13:40:45] <wikibugs>	 06Traffic, 06DC-Ops, 10decommission-hardware, 10ops-eqiad, and 2 others: decommission lvs10[13-15] - https://phabricator.wikimedia.org/T433508#12212255 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by sukhe@cumin1003 for hosts: `lvs1014.eqiad.wmnet` - lvs1014.eqiad.wmnet (**PASS**)   - D...
[13:50:36] <wikibugs>	 06Traffic, 06DC-Ops, 10decommission-hardware, 10ops-eqiad, and 2 others: decommission lvs10[13-15] - https://phabricator.wikimedia.org/T433508#12212330 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by sukhe@cumin1003 for hosts: `lvs1015.eqiad.wmnet` - lvs1015.eqiad.wmnet (**PASS**)   - D...
[13:53:27] <wikibugs>	 06Traffic, 06DC-Ops, 10decommission-hardware, 10ops-eqiad, and 2 others: decommission lvs10[13-15] - https://phabricator.wikimedia.org/T433508#12212337 (10ssingh)
[13:53:37] <wikibugs>	 06Traffic, 06DC-Ops, 10decommission-hardware, 10ops-eqiad, and 2 others: decommission lvs10[13-15] - https://phabricator.wikimedia.org/T433508#12212339 (10ssingh) a:03VRiley-WMF
[14:48:49] <jinxer-wm>	 RESOLVED: PuppetConstantChange: Puppet performing a change on every puppet run on cp7009:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange
[15:17:03] <wikibugs>	 06Traffic, 10DNS, 06SRE, 13Patch-For-Review: [Update DNS Record Request] - wikimedia.org - Add TXT verification for Mentimeter - https://phabricator.wikimedia.org/T434620#12212808 (10CDobbins) I just updated the DNS records, @bcampbell. Let me know if there's any unexpected behavior or if I can close the t...
[15:29:09] <wikibugs>	 10netops, 06Traffic, 06Infrastructure-Foundations, 06SRE: BFD session fails from Anycast hosts over IPv6 on boot - https://phabricator.wikimedia.org/T434806 (10cmooney) 03NEW p:05Triage→03Low
[15:32:52] <wikibugs>	 10netops, 06Traffic, 06Infrastructure-Foundations, 06SRE: BFD session fails from Anycast hosts over IPv6 on boot - https://phabricator.wikimedia.org/T434806#12212903 (10cmooney) I should say this has no particular production impact, BGP establishes and everything works ok.  It's only if BFD was already in...
[15:33:43] <wikibugs>	 10netops, 06Traffic, 06Infrastructure-Foundations, 06SRE: BFD session fails from Anycast hosts over IPv6 on boot - https://phabricator.wikimedia.org/T434806#12212931 (10cmooney)
[15:42:53] <wikibugs>	 10netops, 06Traffic, 06Infrastructure-Foundations, 06SRE: BFD session fails from Anycast hosts over IPv6 on boot - https://phabricator.wikimedia.org/T434806#12212971 (10cmooney)
[17:18:28] <wikibugs>	 06Traffic, 13Patch-For-Review: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#12213423 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cdobbins@cumin1003 for host dns3004.wikimedia.org with OS trixie completed: - dns3004 (**WARN**)   - Downtimed on Icinga/Ale...
[17:21:17] <wikibugs>	 06Traffic, 07PM: Create useful Phabricator task workflow for Traffic - https://phabricator.wikimedia.org/T370557#12213430 (10BCornwall) I think a few things could be helpful for us:  1. Routine gardening of the board: [[ https://phabricator.wikimedia.org/project/1201/item/view/18890/ | This PoC gardening board...
[17:32:50] <wikibugs>	 06Traffic, 06DC-Ops, 10decommission-hardware, 10ops-eqiad, and 2 others: decommission lvs10[13-15] - https://phabricator.wikimedia.org/T433508#12213469 (10ssingh)
[17:37:35] <wikibugs>	 06Traffic, 06DC-Ops, 10decommission-hardware, 10ops-eqiad, and 2 others: decommission lvs10[13-16] - https://phabricator.wikimedia.org/T433508#12213505 (10ssingh)
[20:43:52] <wikibugs>	 06Traffic, 10DNS, 06SRE: [Update DNS Record Request] - wikimedia.org - Add TXT verification for Mentimeter - https://phabricator.wikimedia.org/T434620#12214153 (10bcampbell) @Aklapper Will do, I'll add that step to our process doc.  @CDobbins All looks good, I see domain control is active now for Mentimeter,...