[00:00:34] 06Traffic, 06DC-Ops, 10ops-ulsfo, 13Patch-For-Review: Q4: install PCIe NVMe SSDs into ulsfo text cp40(3[789]|4[01234] - https://phabricator.wikimedia.org/T364891#9901770 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by brett@cumin2002 for host cp4044.ulsfo.wmnet with OS bullseye [00:10:20] 06Traffic, 06DC-Ops, 10ops-ulsfo, 13Patch-For-Review: Q4: install PCIe NVMe SSDs into ulsfo text cp40(3[789]|4[01234] - https://phabricator.wikimedia.org/T364891#9901798 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by brett@cumin2002 for host cp4044.ulsfo.wmnet with OS bullseye execu... [00:10:34] 06Traffic, 06DC-Ops, 10ops-ulsfo, 13Patch-For-Review: Q4: install PCIe NVMe SSDs into ulsfo text cp40(3[789]|4[01234] - https://phabricator.wikimedia.org/T364891#9901799 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by brett@cumin2002 for host cp4044.ulsfo.wmnet with OS bullseye [00:49:28] 06Traffic: Deprecate TLS 1.2 - https://phabricator.wikimedia.org/T367821#9901851 (10Bugreporter) This will mean Chrome<70 and Firefox<63 users will no longer be able to view Wikimedia projects. [00:57:18] 06Traffic, 06DC-Ops, 10ops-ulsfo: Q4: install PCIe NVMe SSDs into ulsfo text cp40(3[789]|4[01234] - https://phabricator.wikimedia.org/T364891#9901859 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by brett@cumin2002 for host cp4044.ulsfo.wmnet with OS bullseye completed: - cp4044 (**PASS... [01:11:55] 06Traffic, 06DC-Ops, 10ops-ulsfo: Q4: install PCIe NVMe SSDs into ulsfo text cp40(3[789]|4[01234] - https://phabricator.wikimedia.org/T364891#9901869 (10BCornwall) [08:09:21] 10netops, 06Data-Persistence, 06DBA, 06Infrastructure-Foundations, and 2 others: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-e1-eqiad - https://phabricator.wikimedia.org/T365993#9902257 (10ABran-WMF) [08:12:52] 10netops, 06Data-Persistence, 06DBA, 06Infrastructure-Foundations, and 2 others: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-e2-eqiad - https://phabricator.wikimedia.org/T365994#9902282 (10ABran-WMF) [08:46:52] 06Traffic, 06Data-Platform-SRE: Migrate Cloudelastic load balancing to IPIP encapsulation (LVS) - https://phabricator.wikimedia.org/T367511#9902384 (10Gehel) p:05Triage→03Medium [08:55:49] 06Traffic, 10Elasticsearch, 06Infrastructure-Foundations, 10Data-Platform-SRE (2024.06.17 - 2024.07.07): Migrate services behind high-traffic2 LVS to IPIP encapsulation - https://phabricator.wikimedia.org/T367312#9902416 (10Gehel) [09:05:49] 06Traffic, 10Data-Platform-SRE (2024.06.17 - 2024.07.07), 13Patch-For-Review: Consider migrating Search Platform-owned clusters to IPIP encapsulation (LVS) - https://phabricator.wikimedia.org/T365616#9902462 (10Gehel) 05In progress→03Resolved Decision is made, implementation following on T367511 [09:32:22] 06Traffic, 06Infrastructure-Foundations: Migrate ldap-ro and ldap-ro-ssl to IPIP encapsulation - https://phabricator.wikimedia.org/T367861 (10Vgutierrez) 03NEW [10:05:10] 06Traffic, 06Data-Engineering, 10Observability-Logging, 13Patch-For-Review: Upgrade hosts to haproxy 2.8.10 - https://phabricator.wikimedia.org/T367756#9902659 (10Fabfur) [10:30:02] 06Traffic, 10Phabricator (Upstream), 10Release-Engineering-Team (Priority Backlog 📥), 07Upstream: Consider using preconnect for https://phab.wmfusercontent.org CDN - https://phabricator.wikimedia.org/T367290#9902731 (10Aklapper) [10:37:27] 06Traffic, 10Phabricator (Upstream), 10Release-Engineering-Team (Priority Backlog 📥), 07Upstream: Consider using preconnect for https://phab.wmfusercontent.org CDN - https://phabricator.wikimedia.org/T367290#9902756 (10Aklapper) 05Open→03Stalled [10:46:22] 06Traffic, 10Phabricator (Upstream), 10Release-Engineering-Team (Priority Backlog 📥), 07Upstream: Consider using preconnect for https://phab.wmfusercontent.org CDN - https://phabricator.wikimedia.org/T367290#9902778 (10Vgutierrez) we already leverage preconnect on some cases but not as an HTTP Header but u... [11:06:24] 06Traffic, 06serviceops, 13Patch-For-Review, 10Release Pipeline (Blubber), 10Release-Engineering-Team (Priority Backlog 📥): Remove blubberoid LVS/k8s service - https://phabricator.wikimedia.org/T365742#9902842 (10JMeybohm) [11:16:12] 06Traffic, 10MoveComms-Support, 10MW-on-K8s, 06serviceops, and 2 others: Move 100% of external traffic to Kubernetes - https://phabricator.wikimedia.org/T362323#9902871 (10Clement_Goubert) [12:14:20] 10netops, 06Data-Persistence, 06DBA, 06Infrastructure-Foundations, and 2 others: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-e2-eqiad - https://phabricator.wikimedia.org/T365994#9903045 (10ABran-WMF) [14:05:52] kwakuofori: hello! how can I help move https://gerrit.wikimedia.org/r/c/operations/puppet/+/1042278 along? any eta for a review? thank you! [14:07:10] hey ottomata, sorry for the delay. following up on it [14:08:16] no worries! thank you. [14:08:50] there isn't really any urgency except my excitement to decommission a system I've been trying to decom for i dunno, 3 or 4 years? [14:19:50] :D [14:37:01] <_joe_> is puppet disabled in ulsfo? [14:37:06] <_joe_> on cp-text [14:37:22] <_joe_> I am asking because it's interfering with the switch of 100% of traffic to k8s [14:37:34] _joe_: only on cp4037 it was, which is enabled now [14:37:37] should not be anywhere else in ulsfo [14:37:47] <_joe_> uhm ok [14:37:56] and enabled on cp4037 [14:47:31] 06Traffic: Deprecate TLS 1.2 - https://phabricator.wikimedia.org/T367821#9903649 (10BCornwall) [14:47:49] 10netops, 06Data-Persistence, 06DBA, 06Infrastructure-Foundations, and 2 others: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-f7-eqiad - https://phabricator.wikimedia.org/T365984#9903652 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=0039bfdd-84ad-4638-9b4c-c0c23984e401) se... [14:56:53] 10netops, 06Data-Persistence, 06DBA, 06Infrastructure-Foundations, and 2 others: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-f7-eqiad - https://phabricator.wikimedia.org/T365984#9903691 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=b16e0477-5d40-4e59-950e-09e82271c822) se... [14:57:44] 10netops, 06Data-Persistence, 06DBA, 06Infrastructure-Foundations, and 2 others: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-f7-eqiad - https://phabricator.wikimedia.org/T365984#9903694 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=80e189d2-8757-4138-ad14-1e0cf5cfbbdb) se... [14:59:31] 06Traffic: Deprecate TLS 1.2 - https://phabricator.wikimedia.org/T367821#9903710 (10BCornwall) [15:01:23] 06Traffic, 10MoveComms-Support, 10MW-on-K8s, 06serviceops, and 2 others: Move 100% of external traffic to Kubernetes - https://phabricator.wikimedia.org/T362323#9903731 (10Clement_Goubert) [15:03:44] 06Traffic, 10MoveComms-Support, 10MW-on-K8s, 06serviceops, and 2 others: Move 100% of external traffic to Kubernetes - https://phabricator.wikimedia.org/T362323#9903729 (10Clement_Goubert) {F55438321} 🚀🚀🚀 [15:08:30] 06Traffic, 10MoveComms-Support, 10MW-on-K8s, 06serviceops, and 2 others: Move 100% of external traffic to Kubernetes - https://phabricator.wikimedia.org/T362323#9903765 (10Ladsgroup) {meme, src=itshappening} [15:13:28] 06Traffic, 10Elasticsearch, 06Infrastructure-Foundations, 10Data-Platform-SRE (2024.06.17 - 2024.07.07): Migrate services behind high-traffic2 LVS to IPIP encapsulation - https://phabricator.wikimedia.org/T367312#9903778 (10Vgutierrez) [15:18:40] 10netops, 06Data-Persistence, 06DBA, 06Infrastructure-Foundations, and 2 others: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-f7-eqiad - https://phabricator.wikimedia.org/T365984#9903792 (10cmooney) Switch is back online after upgrade, everything looks good at first glance. [15:24:08] 10netops, 06Data-Persistence, 06DBA, 06Infrastructure-Foundations, and 2 others: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - lsw1-f7-eqiad - https://phabricator.wikimedia.org/T365984#9903811 (10MatthewVernon) ms swift looks good, thanks. [15:27:40] 06Traffic: Discovery: Deprecation of TLS 1.2 - https://phabricator.wikimedia.org/T367821#9903820 (10BCornwall) [15:29:12] 06Traffic, 06DC-Ops, 10ops-ulsfo, 13Patch-For-Review: Q4: install PCIe NVMe SSDs into ulsfo text cp40(3[789]|4[01234] - https://phabricator.wikimedia.org/T364891#9903823 (10BCornwall) 05In progress→03Resolved [17:47:21] hi traffic folks! wanted to give you a heads up that I'll be updating conftool on cp and ncredir hosts in cache sites shortly. [17:47:21] no disruptions are expected, as there are no functional changes to confctl in this release (only changes to logging severity). [17:47:21] core sites have already been updated, with no issues encountered thus far. [17:47:37] swfrench-wmf: thanks for the heads-up [17:47:47] we also have DNS hosts that depend on confctl, just as an FYI [17:48:06] that being more critical in some ways since the pooled state determines if the service announcements and authdns-update can be pushed or not [17:50:33] basically [17:50:33] {"dns1004.wikimedia.org": {"weight": 100, "pooled": "yes"}, "tags": "dc=eqiad,cluster=dnsbox,service=authdns-update"} [17:50:37] {"dns1004.wikimedia.org": {"weight": 100, "pooled": "yes"}, "tags": "dc=eqiad,cluster=dnsbox,service=ntp"} [17:50:40] {"dns1004.wikimedia.org": {"weight": 100, "pooled": "yes"}, "tags": "dc=eqiad,cluster=dnsbox,service=authdns-ns0"} [17:50:43] {"dns1004.wikimedia.org": {"weight": 100, "pooled": "yes"}, "tags": "dc=eqiad,cluster=dnsbox,service=recdns"} [17:50:46] {"dns1004.wikimedia.org": {"weight": 100, "pooled": "yes"}, "tags": "dc=eqiad,cluster=dnsbox,service=authdns-ns2"} [17:50:56] sukhe: great, thanks! [17:51:29] so, just to clarify, what hosts are you talking about specifically where the conftool debian package might be installed [17:51:43] e.g., which hosts might be querying the pooled status you're referring to [17:52:52] basically, I don't see python3-conftool installed on dns1004 for example, so I'm wondering if it might instead be confd that's reading pooled status? [17:53:52] swfrench-wmf: sorry, I should have clarified. you are right, the conftool package doesn't actually exist on the DNS hosts [17:54:15] but for puppetmasters and such from where we control the state, I wanted to mention that the DNS hosts also fall under that in case it wasn't obvious [17:54:21] since that's a more recent shift than the other ones [17:54:29] but sorry, go ahead please for the cp and ncredir ones :) [17:56:04] sukhe: ah, interesting! got it, thanks for clarifying :) I have already updated the puppetmasters after smoke-testing confctl, dbctl, and requestctl, so I guess that ship may have already sailed, heh [17:56:43] nice, thanks! [17:57:24] happy future sailiing [18:45:41] these updates are now done [19:19:29] nicely done! [21:19:21] 06Traffic, 06Data-Platform-SRE, 13Patch-For-Review: Migrate Cloudelastic load balancing to IPIP encapsulation (LVS) - https://phabricator.wikimedia.org/T367511#9905254 (10bking) [21:23:54] 06Traffic, 06Data-Platform-SRE, 13Patch-For-Review: Migrate Cloudelastic load balancing to IPIP encapsulation (LVS) - https://phabricator.wikimedia.org/T367511#9905273 (10bking) [21:24:10] 06Traffic, 06Data-Platform-SRE, 13Patch-For-Review: Migrate Cloudelastic load balancing to IPIP encapsulation (LVS) - https://phabricator.wikimedia.org/T367511#9905279 (10bking)