[00:16:12] 10netops, 06Infrastructure-Foundations, 10ops-magru: Flapping OSFP between cr1-magru and cr2-eqiad - https://phabricator.wikimedia.org/T413415 (10Papaul) 03NEW [06:00:31] 10netops, 06Infrastructure-Foundations, 06SRE: InboundInterfaceErrors alerts firing for Nokia switches on v25.10.1 - https://phabricator.wikimedia.org/T412733#11482748 (10VRiley-WMF) [09:27:43] FIRING: [9x] HaproxyKafkaSocketDroppedMessages: Sustained high rate of dropped messages from HaproxyKafka - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaSocketDroppedMessages - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaSocketDroppedMessages [09:32:43] FIRING: [16x] HaproxyKafkaSocketDroppedMessages: Sustained high rate of dropped messages from HaproxyKafka - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaSocketDroppedMessages - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaSocketDroppedMessages [09:37:43] FIRING: [16x] HaproxyKafkaSocketDroppedMessages: Sustained high rate of dropped messages from HaproxyKafka - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaSocketDroppedMessages - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaSocketDroppedMessages [09:42:43] RESOLVED: [16x] HaproxyKafkaSocketDroppedMessages: Sustained high rate of dropped messages from HaproxyKafka - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaSocketDroppedMessages - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaSocketDroppedMessages [09:49:09] FIRING: LVSHighRX: Excessive RX traffic on lvs7001:9100 (eno12399np0) - https://bit.ly/wmf-lvsrx - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=lvs7001 - https://alerts.wikimedia.org/?q=alertname%3DLVSHighRX [09:54:09] RESOLVED: LVSHighRX: Excessive RX traffic on lvs7001:9100 (eno12399np0) - https://bit.ly/wmf-lvsrx - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=lvs7001 - https://alerts.wikimedia.org/?q=alertname%3DLVSHighRX [10:05:58] 06Traffic, 10DNS, 06SRE: [Update DNS Record Request] - wikimedia.org - https://phabricator.wikimedia.org/T413259#11482958 (10Fabfur) 05Open→03Resolved a:03Fabfur [14:06:53] 10netops, 06Infrastructure-Foundations, 10ops-magru: Flapping OSFP between cr1-magru and cr2-eqiad - https://phabricator.wikimedia.org/T413415#11483229 (10Papaul) 05Open→03Invalid [14:23:40] 10netops, 06Infrastructure-Foundations, 06SRE: InboundInterfaceErrors alerts firing for Nokia switches on v25.10.1 - https://phabricator.wikimedia.org/T412733#11483270 (10Papaul) I am still waiting for Nokia to get back in touch with me. [22:45:58] 10netops, 06FR-donorrelations, 06Fundraising-Backlog, 10fundraising-tech-ops, and 2 others: Routing issue from payments hosts to 62.146.255.205 - https://phabricator.wikimedia.org/T413382#11483888 (10Dwisehaupt) More details from adyen, this particular IP is behind cloudflare: ` This IP address 62.146.255....