[09:43:46] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06SRE: magru network setup - https://phabricator.wikimedia.org/T362421#9811967 (10cmooney) >>! In T362421#9808627, @ayounsi wrote: > The Telxius community doesn't seem to be of any effect so far, I'll wait for their reply, maybe they changed or need to be enabled on th...
[12:04:34] <wikibugs>	 06Traffic: rp_filter should be disabled on puppet apply - https://phabricator.wikimedia.org/T365354 (10Vgutierrez) 03NEW
[12:05:03] <wikibugs>	 06Traffic: rp_filter should be disabled on puppet apply - https://phabricator.wikimedia.org/T365354#9812255 (10Vgutierrez) p:05Triage→03Medium
[12:48:30] <wikibugs>	 10netops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE, 13Patch-For-Review: Problem re-imaging hosts on row-wide vlan on EVPN switches - https://phabricator.wikimedia.org/T365204#9812357 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cmooney@cumin1002 for host sretest2002....
[13:58:34] <wikibugs>	 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, and 2 others: Problem re-imaging hosts on row-wide vlan on EVPN switches - https://phabricator.wikimedia.org/T365204#9812556 (10cmooney) So some interesting findings when testing today.  I was able to reproduce the issue with sretest2002, and t...
[14:02:09] <wikibugs>	 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, and 2 others: Problem re-imaging hosts on row-wide vlan on EVPN switches - https://phabricator.wikimedia.org/T365204#9812566 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cmooney@cumin1002 for host sretest2002.wikimedia...
[14:15:06] <wikibugs>	 06Traffic, 13Patch-For-Review: Use IPIP encapsulation on lvs<-->upload cluster - https://phabricator.wikimedia.org/T357257#9812581 (10Vgutierrez)
[14:58:12] <wikibugs>	 06Traffic, 13Patch-For-Review: Use IPIP encapsulation on lvs<-->upload cluster - https://phabricator.wikimedia.org/T357257#9812720 (10Vgutierrez)
[15:39:23] <wikibugs>	 06Traffic, 13Patch-For-Review: Use IPIP encapsulation on lvs<-->upload cluster - https://phabricator.wikimedia.org/T357257#9812839 (10Vgutierrez)
[15:43:54] <wikibugs>	 06Traffic, 10MoveComms-Support, 10MW-on-K8s, 06serviceops, and 2 others: Move 100% of external traffic to Kubernetes (excluding Votewiki and Commons) - https://phabricator.wikimedia.org/T362323#9812866 (10hnowlan)
[16:10:14] <wikibugs>	 06Traffic, 10MoveComms-Support, 10MW-on-K8s, 06serviceops, and 2 others: Move 100% of external traffic to Kubernetes (excluding Votewiki and Commons) - https://phabricator.wikimedia.org/T362323#9813003 (10Jdforrester-WMF)
[16:14:26] <wikibugs>	 06Traffic, 10MoveComms-Support, 10MW-on-K8s, 06serviceops, and 2 others: Move 100% of external traffic to Kubernetes (excluding Votewiki and Commons) - https://phabricator.wikimedia.org/T362323#9813010 (10hnowlan)
[16:31:58] <wikibugs>	 10Wikimedia-Apache-configuration, 06Security-Team, 07Documentation, 07SecTeam-Processed, 07Security: Add security.txt to Wikimedia sites? (2023 edition) - https://phabricator.wikimedia.org/T337949#9813114 (10mmartorana) 05In progress→03Resolved
[16:36:25] <jinxer-wm>	 FIRING: SystemdUnitFailed: ncmonitor.service on ncmonitor1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[20:36:40] <jinxer-wm>	 FIRING: SystemdUnitFailed: ncmonitor.service on ncmonitor1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[20:37:16] <brett>	 ^quieting
[20:41:25] <jinxer-wm>	 RESOLVED: SystemdUnitFailed: ncmonitor.service on ncmonitor1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[23:17:10] <wikibugs>	 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, and 2 others: Problem re-imaging hosts on row-wide vlan on EVPN switches - https://phabricator.wikimedia.org/T365204#9814633 (10Jhancock.wm) @Papaul still getting an error on provisioning of the new server.   100.0% (1/1) success ratio (>= 100....
[23:39:16] <wikibugs>	 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, and 2 others: Problem re-imaging hosts on row-wide vlan on EVPN switches - https://phabricator.wikimedia.org/T365204#9814670 (10Papaul) @Jhancock.wm it looks like we have another sretest2002 setup in b7 the switch has that configuration already...