[04:44:09] FIRING: LVSHighRX: Excessive RX traffic on lvs2013:9100 (eno12399np0) - https://bit.ly/wmf-lvsrx - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=lvs2013 - https://alerts.wikimedia.org/?q=alertname%3DLVSHighRX [04:49:09] RESOLVED: LVSHighRX: Excessive RX traffic on lvs2013:9100 (eno12399np0) - https://bit.ly/wmf-lvsrx - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=lvs2013 - https://alerts.wikimedia.org/?q=alertname%3DLVSHighRX [08:13:09] FIRING: LVSHighRX: Excessive RX traffic on lvs2013:9100 (eno12399np0) - https://bit.ly/wmf-lvsrx - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=lvs2013 - https://alerts.wikimedia.org/?q=alertname%3DLVSHighRX [08:18:09] RESOLVED: LVSHighRX: Excessive RX traffic on lvs2013:9100 (eno12399np0) - https://bit.ly/wmf-lvsrx - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=lvs2013 - https://alerts.wikimedia.org/?q=alertname%3DLVSHighRX [09:45:54] 06Traffic, 10Liberica, 13Patch-For-Review: Replace pybal with liberica on the PoPs - https://phabricator.wikimedia.org/T384477#10640779 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by vgutierrez@cumin1002 for host lvs3010.esams.wmnet with OS bookworm [10:34:09] FIRING: LVSHighRX: Excessive RX traffic on lvs2013:9100 (eno12399np0) - https://bit.ly/wmf-lvsrx - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=lvs2013 - https://alerts.wikimedia.org/?q=alertname%3DLVSHighRX [10:37:35] 06Traffic, 10Liberica, 13Patch-For-Review: Replace pybal with liberica on the PoPs - https://phabricator.wikimedia.org/T384477#10640973 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by vgutierrez@cumin1002 for host lvs3010.esams.wmnet with OS bookworm executed with errors: - lvs3010 (**... [10:38:13] 06Traffic, 10Liberica, 13Patch-For-Review: Replace pybal with liberica on the PoPs - https://phabricator.wikimedia.org/T384477#10640974 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by vgutierrez@cumin1002 for host lvs3010.esams.wmnet with OS bookworm [10:44:09] RESOLVED: LVSHighRX: Excessive RX traffic on lvs2013:9100 (eno12399np0) - https://bit.ly/wmf-lvsrx - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=lvs2013 - https://alerts.wikimedia.org/?q=alertname%3DLVSHighRX [11:20:37] 06Traffic, 10Liberica, 13Patch-For-Review: Replace pybal with liberica on the PoPs - https://phabricator.wikimedia.org/T384477#10641177 (10Vgutierrez) [11:22:28] 06Traffic, 10Liberica, 13Patch-For-Review: Replace pybal with liberica on the PoPs - https://phabricator.wikimedia.org/T384477#10641180 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by vgutierrez@cumin1002 for host lvs3010.esams.wmnet with OS bookworm completed: - lvs3010 (**PASS**) -... [13:08:11] 06Traffic, 10Liberica, 13Patch-For-Review: Replace pybal with liberica on the PoPs - https://phabricator.wikimedia.org/T384477#10641553 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=00120896-d6ec-4aac-9b71-59479cad308d) set by vgutierrez@cumin1002 for 0:30:00 on 1 host(s) and their serv... [13:13:03] 10netops, 06Infrastructure-Foundations, 10ops-drmrs: cr1-drmrs to asw1-b12-drmrs link down - https://phabricator.wikimedia.org/T389071 (10ayounsi) 03NEW p:05Triage→03High [13:18:52] 06Traffic, 10Liberica, 13Patch-For-Review: Replace pybal with liberica on the PoPs - https://phabricator.wikimedia.org/T384477#10641610 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by vgutierrez@cumin1002 for host lvs3009.esams.wmnet with OS bookworm [13:23:17] 10netops, 06Infrastructure-Foundations, 10ops-drmrs: cr1-drmrs to asw1-b12-drmrs link down - https://phabricator.wikimedia.org/T389071#10641632 (10ayounsi) [13:36:37] 06Traffic: Evaluate HAProxy 3.1 - https://phabricator.wikimedia.org/T386796#10641695 (10Vgutierrez) [13:57:52] 06Traffic, 10Observability-Alerting, 06SRE, 10SRE Observability (FY2024/2025-Q3): Icinga check_curl plugin is broken on bullseye and bookworm hosts - https://phabricator.wikimedia.org/T388680#10641773 (10tappof) 05Open→03Resolved It looks like the patch has fixed the problem. I'm closing the task.... [14:00:07] 06Traffic, 10Observability-Alerting, 06SRE, 10SRE Observability (FY2024/2025-Q3): Icinga check_curl plugin is broken on bullseye and bookworm hosts - https://phabricator.wikimedia.org/T388680#10641781 (10ssingh) Thanks for taking care of it; can confirm resolved! [14:00:09] FIRING: LVSHighRX: Excessive RX traffic on lvs2013:9100 (eno12399np0) - https://bit.ly/wmf-lvsrx - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=lvs2013 - https://alerts.wikimedia.org/?q=alertname%3DLVSHighRX [14:01:15] 06Traffic, 10Liberica, 13Patch-For-Review: Replace pybal with liberica on the PoPs - https://phabricator.wikimedia.org/T384477#10641783 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by vgutierrez@cumin1002 for host lvs3009.esams.wmnet with OS bookworm completed: - lvs3009 (**PASS**) -... [14:05:09] RESOLVED: LVSHighRX: Excessive RX traffic on lvs2013:9100 (eno12399np0) - https://bit.ly/wmf-lvsrx - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=lvs2013 - https://alerts.wikimedia.org/?q=alertname%3DLVSHighRX [14:10:57] 06Traffic, 10Liberica, 13Patch-For-Review: Replace pybal with liberica on the PoPs - https://phabricator.wikimedia.org/T384477#10641826 (10Vgutierrez) [14:26:19] 06Traffic, 10Liberica, 13Patch-For-Review: Replace pybal with liberica on the PoPs - https://phabricator.wikimedia.org/T384477#10641913 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=84eaa5ca-ad49-419d-9f2f-eb1dda5bf75d) set by vgutierrez@cumin1002 for 0:30:00 on 1 host(s) and their serv... [14:29:13] 06Traffic, 10Liberica, 13Patch-For-Review: Replace pybal with liberica on the PoPs - https://phabricator.wikimedia.org/T384477#10641917 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by vgutierrez@cumin1002 for host lvs3008.esams.wmnet with OS bookworm [14:38:31] 06Traffic: Evaluate HAProxy 3.1 - https://phabricator.wikimedia.org/T386796#10641982 (10Vgutierrez) 05Open→03In progress [15:11:47] 06Traffic, 10Liberica: Replace pybal with liberica on the PoPs - https://phabricator.wikimedia.org/T384477#10642180 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by vgutierrez@cumin1002 for host lvs3008.esams.wmnet with OS bookworm completed: - lvs3008 (**PASS**) - Downtimed on Icinga/A... [15:14:30] 10netops, 06Infrastructure-Foundations, 10ops-drmrs: cr1-drmrs to asw1-b12-drmrs link down - https://phabricator.wikimedia.org/T389071#10642215 (10RobH) Draft of directions: > Support, > > We just had an optic fail on one of our router to switch links, and need the switch side optic swapped out with spa... [15:18:02] 06Traffic, 10Liberica: Replace pybal with liberica on the PoPs - https://phabricator.wikimedia.org/T384477#10642245 (10Vgutierrez) [15:20:39] 10netops, 06Infrastructure-Foundations, 10ops-drmrs: cr1-drmrs to asw1-b12-drmrs link down - https://phabricator.wikimedia.org/T389071#10642258 (10RobH) Had the option for 'normal work' which must be planned in work hours and 24 hours in advance (with time zone changes that means if I entered it now, it woul... [15:21:26] 10netops, 06Infrastructure-Foundations, 10ops-drmrs: cr1-drmrs to asw1-b12-drmrs link down - https://phabricator.wikimedia.org/T389071#10642262 (10RobH) a:03RobH [15:28:11] sukhe: o/ [15:28:19] first of all, thanks a ton for the reviews <3 [15:28:39] I am a bit doubtful about https://gerrit.wikimedia.org/r/c/operations/puppet/+/1128343 - will it need a pybal restart? [15:28:40] 06Traffic: requestctl bandwidth limit has incorrect syntax - https://phabricator.wikimedia.org/T388529#10642312 (10Volans) [15:29:23] elukey: I think so but also, we can quickly check when we roll out [15:30:05] elukey: yes, pybal doesn't reload any bits of configuration without a restart [15:30:26] he is just selling liberica here ^ [15:30:28] :P [15:30:36] and of course the new etcd cluster/service needs to be populated with some realservers before that restart [15:30:59] vgutierrez: yeah the kubesvc bit is already the one that all k8s services use [15:31:43] ok then I'd say I will not do anything now, and we have the switchover lined up, so I'll find a slot later on during the week :) [15:31:59] elukey: anytime, just ping us. thanks for structuring the patches the right way <3 [15:32:51] sukhe: np! I usually picture Valentin staring at me when sending patches for service.yaml and that helps preventing mistakes :D [15:33:26] that's a guiding principle for me as well [15:33:34] are you implying I'm that ugly that I'm scary? [15:33:38] * vgutierrez cries in the corner [15:34:08] there is volint and there is valentined [15:36:17] vgutierrez: nono nothing related to that, just a reminder of proper procedures and tidy patches :D Something like "how likely is that Valentin will ask me why I am doing this? 110%? Probably better to refactor :D" [16:09:40] 06Traffic, 10Liberica, 13Patch-For-Review: Replace pybal with liberica on the PoPs - https://phabricator.wikimedia.org/T384477#10642502 (10Vgutierrez) 05In progress→03Resolved [16:09:46] ^^ 🍻 [16:14:35] Congrats! [16:15:33] ! [16:26:08] 10netops, 06Infrastructure-Foundations, 10ops-drmrs: cr1-drmrs to asw1-b12-drmrs link down - https://phabricator.wikimedia.org/T389071#10642593 (10ayounsi) remote hands replaced the optic, but the issue persists. Looking closer at it it converts the 40G port into 4x10G lanes. This might be because lane 1 is... [16:38:10] 10netops, 06Infrastructure-Foundations, 10ops-drmrs: cr1-drmrs to asw1-b12-drmrs link down - https://phabricator.wikimedia.org/T389071#10642675 (10RobH) IRC update: We asked them to swap both optic and fiber patch to reduce complexity in troubleshooting. > Support, > > Background: For some reason this li... [16:45:01] 06Traffic, 10Maps, 06SRE: Allow Wikimedia Maps usage on  - https://phabricator.wikimedia.org/T389096 (10Olivierpeyronnet) 03NEW Closing this task as invalid due to missing information. [16:47:58] 06Traffic, 10Maps, 06SRE: Allow Wikimedia Maps usage on my research about Bosnia - https://phabricator.wikimedia.org/T389099 (10Olivierpeyronnet) 03NEW [16:48:52] 06Traffic, 10Maps, 06SRE: Allow Wikimedia Maps usage on my research about Bosnia - https://phabricator.wikimedia.org/T389099#10642769 (10Olivierpeyronnet) I am a master's student working on my thesis about the postal and telegraph network in Bosnia-Herzegovina before World War I. My research requires mapping... [16:49:01] 06Traffic, 13Patch-For-Review: Evaluate HAProxy 3.1 - https://phabricator.wikimedia.org/T386796#10642770 (10Vgutierrez) [16:50:09] 06Traffic, 13Patch-For-Review: Evaluate HAProxy 3.1 - https://phabricator.wikimedia.org/T386796#10642779 (10Vgutierrez) 05In progress→03Stalled downgraded after seeing the following issue in cp5024: ` Mar 17 16:00:08 cp5024 systemd[1]: Reloaded HAProxy Load Balancer. Mar 17 16:00:08 cp5024 haproxy[3828921]... [16:56:04] 06Traffic, 10Maps, 06SRE: Allow Wikimedia Maps usage on  - https://phabricator.wikimedia.org/T389096#10642818 (10Aklapper) →14Duplicate dup:03T389099 [16:56:06] 06Traffic, 10Maps, 06SRE: Allow Wikimedia Maps usage on my research about Bosnia - https://phabricator.wikimedia.org/T389099#10642820 (10Aklapper) [16:57:11] 06Traffic, 10Maps, 06SRE: Allow Wikimedia Maps usage on my research about Bosnia - https://phabricator.wikimedia.org/T389099#10642826 (10Aklapper) 05Open→03Declined Hi @Olivierpeyronnet, maps.wikimedia.org tiles may only be used by Wikimedia wikis, and sites hosted by Wikimedia Affiliates. We are not... [17:11:55] 06Traffic: upgrade to trafficserver 9.2.9 - https://phabricator.wikimedia.org/T388035#10642973 (10BCornwall) [20:46:43] 06Traffic, 13Patch-For-Review: Upgrade Varnish from 6.0 to 7.1 - https://phabricator.wikimedia.org/T378737#10644009 (10BCornwall)