[09:28:44] 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - https://phabricator.wikimedia.org/T348977#9824816 (10cmooney) We also now have the issue from T365204 that we can resolve with an upgrade of JunOS. Not essential in eqiad but still I think we need to stop proc... [09:38:56] 06Traffic: Provide a TCP MSS clamping mechanism for real servers - https://phabricator.wikimedia.org/T350462#9824913 (10Vgutierrez) 05Open→03Resolved tcp-mss-clamper is being already used to perform MSS clamping on ncredir and CDN upload clusters [09:42:35] 06Traffic: Provide a ferm based alternative to tcp-mss-clamper - https://phabricator.wikimedia.org/T365689 (10Vgutierrez) 03NEW [12:12:28] 10netops, 06Infrastructure-Foundations: Arelion IPv6 transit renumbering - https://phabricator.wikimedia.org/T365697 (10ayounsi) 03NEW [12:19:12] 10netops, 06Infrastructure-Foundations, 13Patch-For-Review: Arelion IPv6 transit renumbering - https://phabricator.wikimedia.org/T365697#9825281 (10ayounsi) [13:00:34] 06Traffic, 10MW-on-K8s, 06serviceops, 06SRE, 10Release-Engineering-Team (Seen): Rename X-Wikimedia-Debug k8s-experimental option - https://phabricator.wikimedia.org/T362662#9825484 (10Jdforrester-WMF) 05In progress→03Resolved [13:01:24] 06Traffic, 13Patch-For-Review: Use IPIP encapsulation on lvs<-->upload cluster - https://phabricator.wikimedia.org/T357257#9825490 (10Vgutierrez) 05In progress→03Resolved [13:34:57] 06Traffic: rp_filter should be disabled on puppet apply - https://phabricator.wikimedia.org/T365354#9825719 (10Vgutierrez) 05Open→03Resolved [14:24:04] 06Traffic, 06Data-Engineering, 10Observability-Logging: Switch HAProxy/Benthos to rfc5424 - https://phabricator.wikimedia.org/T365718 (10Fabfur) 03NEW [15:06:41] 06Traffic, 10Data-Platform-SRE (2024.05.06 - 2024.05.26), 13Patch-For-Review, 10Sustainability (Incident Followup): LVS hosts: Monitor/alert when pooled nodes are outside broadcast domain - https://phabricator.wikimedia.org/T363702#9826080 (10Gehel) [15:08:33] 06Traffic, 06Content-Transform-Team, 06MW-Interfaces-Team, 10RESTBase Sunsetting: Remove long term caching and active purging for Parsoid endpoints in RESTBase - https://phabricator.wikimedia.org/T365630#9826088 (10daniel) [15:20:23] 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - https://phabricator.wikimedia.org/T348977#9826145 (10cmooney) [15:41:15] vgutierrez: hey we have an upcoming network maintenance in eqiad rack F1 (provisionally scheduled for July 11th) [15:41:30] lvs1013-1016 are in that rack, I think you were using for testing? [15:42:01] do we need to take any action before the switch upgrade (T348977), they'd be offline about 30 mins (probably half that) [15:42:02] T348977: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - https://phabricator.wikimedia.org/T348977 [15:42:17] topranks: no problem at all :) [15:42:23] cool! [15:55:23] 06Traffic, 06Content-Transform-Team, 06MW-Interfaces-Team, 10RESTBase Sunsetting: Remove long term caching and active purging for Parsoid endpoints in RESTBase - https://phabricator.wikimedia.org/T365630#9826415 (10FJoseph-WMF) p:05Triage→03High [15:57:55] 06Traffic, 06MW-Interfaces-Team, 06serviceops: map the /api/ prefix to /w/rest.php - https://phabricator.wikimedia.org/T364400#9826433 (10FJoseph-WMF) p:05Triage→03High [16:13:27] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: codfw row C/D upgrade racking task - https://phabricator.wikimedia.org/T360789#9826511 (10Papaul) [16:26:46] 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade EVPN switches Eqiad row E-F to JunOS 22.2 - https://phabricator.wikimedia.org/T348977#9826583 (10cmooney) [17:03:18] 06Traffic, 10Release Pipeline (Blubber), 10Release-Engineering-Team (Priority Backlog 📥): Remove blubberoid LVS/k8s service - https://phabricator.wikimedia.org/T365742 (10dduvall) 03NEW [17:07:42] hey all 0/ [17:08:09] i'm currently looking to decommission the blubberoid service which is no longer used and need help with steps in https://wikitech.wikimedia.org/wiki/LVS#Remove_a_load_balanced_service [17:14:05] hi, dduvall! Are you wanting help in performing the steps? [17:14:15] yes, please :D [17:14:40] i looked for blubberoid in the instance dropdown when creating a new silence but i can't seem to find it [17:15:00] according to the docs it should be `blubberoid:4666` but that doesn't seem to match anything [17:15:58] karma/alerts.wikimedia.org? [17:16:00] if it helps, the alerts from about 35 minutes ago were the ones i triggered :| [17:16:24] i'm looking at alerts.wikimedia.org yeah [17:16:32] karma's interface is kinda awful [17:16:34] * brett looks [17:18:19] lol, it crashed my firefox [17:19:02] haha oh no [17:33:58] This is painful [17:35:32] I can't find a way to create an alert for it as well. worse case, it will just alert and then we can silence it. it doesn't page, so it's good [17:35:52] dduvall: What was the alert? I'm seeing lvs alerting [17:39:04] brett: pybal backends health check was what went off for dduvall [17:39:37] yeah that's expected [17:39:47] but Traffic is happy to help regardless [17:40:30] I like the idea of just silencing if/when it fires instead of traipsing [17:45:03] shall i move to the next step then? "Remove the discovery DNS record" ? [17:46:33] Sure [17:46:46] alrighty [17:52:48] 06Traffic, 10Release Pipeline (Blubber), 10Release-Engineering-Team (Priority Backlog 📥): Remove blubberoid LVS/k8s service - https://phabricator.wikimedia.org/T365742#9827010 (10dduvall) 05Open→03In progress p:05Triage→03Medium [17:57:32] 06Traffic, 06Content-Transform-Team, 06MW-Interfaces-Team, 10RESTBase Sunsetting: Remove long term caching and active purging for Parsoid endpoints in RESTBase - https://phabricator.wikimedia.org/T365630#9827033 (10BBlack) >>! In T365630#9822721, @daniel wrote: >>>! In T365630#9822532, @BBlack wrote: >> Re... [18:10:18] dduvall: +1ed [18:12:11] brett: thanks! i don't have +2 or deployment access there so will wait for someone with that access while i digest the step steps :) [18:12:33] dduvall: Okay, I'll deploy [18:13:45] nice, thanks [18:14:27] dduvall: applied [18:16:32] awesome. i have https://gerrit.wikimedia.org/r/c/operations/puppet/+/1035543 now as well [18:19:27] +1 [18:24:04] ok, i'll try to prepare all the other patches as best i can. thanks for your help brett [18:24:11] of course [18:30:22] 06Traffic, 06DC-Ops, 10ops-ulsfo, 06SRE: Q4: install PCIe NVMe SSDs into ulsfo text cp40(3[789]|4[01234] - https://phabricator.wikimedia.org/T364891#9827152 (10BCornwall) [20:30:19] 10netops, 06Infrastructure-Foundations: Arelion BGP sessions - IPv6 reconfiguration - - https://phabricator.wikimedia.org/T365762 (10Dzahn) 03NEW [20:30:29] 10netops, 06Infrastructure-Foundations: Arelion BGP sessions - IPv6 reconfiguration - https://phabricator.wikimedia.org/T365762#9827554 (10Dzahn) [20:32:46] 10netops, 06Infrastructure-Foundations, 13Patch-For-Review: Arelion IPv6 transit renumbering - https://phabricator.wikimedia.org/T365697#9827579 (10Volans) [20:32:55] 10netops, 06Infrastructure-Foundations: Arelion BGP sessions - IPv6 reconfiguration - https://phabricator.wikimedia.org/T365762#9827577 (10Volans) →14Duplicate dup:03T365697 [21:16:56] 06Traffic, 06DC-Ops, 10ops-eqsin: Q#:rack/setup/install X - https://phabricator.wikimedia.org/T365763 (10RobH) 03NEW [21:17:26] 06Traffic, 06DC-Ops, 10ops-eqsin: Q4: install PCIe NVMe SSDs into eqsin text cp50(1[789]|2[01234] - https://phabricator.wikimedia.org/T365763#9827683 (10RobH) [21:25:20] 06Traffic, 06DC-Ops, 10ops-eqsin: Q4: install PCIe NVMe SSDs into eqsin text cp50(1[789]|2[01234] - https://phabricator.wikimedia.org/T365763#9827741 (10RobH) [21:25:32] 06Traffic, 06DC-Ops, 10ops-eqsin: Q4: install PCIe NVMe SSDs into eqsin text cp50(1[789]|2[01234] - https://phabricator.wikimedia.org/T365763#9827744 (10RobH) [21:39:20] 06Traffic, 06DC-Ops, 10ops-eqsin, 06SRE: Q4: install PCIe NVMe SSDs into eqsin text cp50(1[789]|2[01234] - https://phabricator.wikimedia.org/T365763#9827789 (10RobH) [21:42:06] 06Traffic, 06DC-Ops, 10ops-eqsin, 06SRE: Q4: install PCIe NVMe SSDs into eqsin text cp50(1[789]|2[01234] - https://phabricator.wikimedia.org/T365763#9827795 (10RobH) [21:54:46] 06Traffic, 06DC-Ops, 10ops-ulsfo, 06SRE: Q4: install PCIe NVMe SSDs into ulsfo text cp40(3[789]|4[01234] - https://phabricator.wikimedia.org/T364891#9827823 (10RobH) SSDs confirmed onsite by shipping, so I can go onsite whenever we schedule to take and install the SSD upgrades.