[04:35:02] 10Traffic, 10Wikidata, 10wmde-wikidata-tech, 10Wikimedia-production-error: 503 on Wikidata - https://phabricator.wikimedia.org/T352094 (10AlexisJazz) 05Open→03Resolved a:03ssingh Nope, seems fine now. [08:37:07] 10Traffic, 10SRE: Firewall rules prevent IPIP/IP6IP6 encapsulated traffic from reaching realservers - https://phabricator.wikimedia.org/T352143 (10Vgutierrez) [08:37:27] 10Traffic, 10SRE: Firewall rules prevent IPIP/IP6IP6 encapsulated traffic from reaching realservers - https://phabricator.wikimedia.org/T352143 (10Vgutierrez) p:05Triage→03High [08:45:29] 10Traffic, 10SRE: Firewall rules prevent IPIP/IP6IP6 encapsulated traffic from reaching realservers - https://phabricator.wikimedia.org/T352143 (10Vgutierrez) using the syntax on the good old iptables, this should work: ` iptables -A INPUT -s 172.16.0.0/10 -p ipencap -j ACCEPT ip6tables -A INPUT -s 0100::/64 -... [09:33:07] 10Traffic, 10Data-Persistence, 10Infrastructure-Foundations, 10conftool, 10serviceops: Switch conftool to use the version 3 etcd datastore - https://phabricator.wikimedia.org/T350565 (10Volans) Untagged sre-tools and spicerack as I've created the dedicated sub-tasks for them. [10:40:36] 10Traffic, 10SRE: Firewall rules prevent IPIP/IP6IP6 encapsulated traffic from reaching realservers - https://phabricator.wikimedia.org/T352143 (10Vgutierrez) 05Open→03Resolved [10:40:41] 10Traffic, 10SRE, 10Patch-For-Review: Enable IPIP encapsulation for ncredir - https://phabricator.wikimedia.org/T351069 (10Vgutierrez) [10:41:28] 10Traffic, 10SRE, 10Patch-For-Review: Enable IPIP encapsulation for ncredir - https://phabricator.wikimedia.org/T351069 (10Vgutierrez) [10:43:52] 10Traffic, 10SRE: RP filtering drops requests incoming via IPIP tunnels on ncredir realservers - https://phabricator.wikimedia.org/T352160 (10Vgutierrez) [10:44:15] 10Traffic, 10SRE: RP filtering drops requests incoming via IPIP tunnels on ncredir realservers - https://phabricator.wikimedia.org/T352160 (10Vgutierrez) p:05Triage→03High [11:33:04] 10netops, 10Infrastructure-Foundations: cr2-esams Transit Tele2 down - https://phabricator.wikimedia.org/T352163 (10Volans) [11:33:24] 10netops, 10Infrastructure-Foundations: cr2-esams Transit Tele2 down - https://phabricator.wikimedia.org/T352163 (10ops-monitoring-bot) ===== Automated diagnostic for Netbox interface ID cr2-esams:xe-0/1/2 --- **Interface cr2-esams:xe-0/1/2** - admin-status: up - ⚠️ oper-status: down - interface-flapped: 20... [11:33:51] 10netops, 10Infrastructure-Foundations: cr2-esams Transit Tele2 down - https://phabricator.wikimedia.org/T352163 (10cmooney) Thanks @volans, I'll have a look and reach out to them. [11:34:56] 10netops, 10Infrastructure-Foundations: cr2-esams Transit Tele2 down - https://phabricator.wikimedia.org/T352163 (10Volans) p:05Triage→03High [11:39:27] XioNoX: https://gitlab.wikimedia.org/-/snippets/107 scapy is becoming my new best friend [11:44:22] wow, your best friend bar must be pretty low if scapy met the criteria :-P [11:48:54] volans: LOL [11:49:12] *meets [11:56:42] 10netops, 10Infrastructure-Foundations: cr2-esams Transit Tele2 down - https://phabricator.wikimedia.org/T352163 (10cmooney) 05Open→03Resolved a:03cmooney Port seems to have come back up while I was trying to sort out a v6 issue on my laptop: ` Nov 28 11:22:52 cr2-esams mib2d[18462]: SNMP_TRAP_LINK_DOWN... [11:59:52] 10Traffic, 10SRE, 10Patch-For-Review: RP filtering drops requests incoming via IPIP tunnels on ncredir realservers - https://phabricator.wikimedia.org/T352160 (10Vgutierrez) [12:45:06] haha yeah [12:49:19] vgutierrez: for the rp_filter, you might need to turn it off for the specific interface too eg. `net.ipv4.conf.eno1.rp_filter` [12:58:00] yup.. that's what we got on LVS instances though [12:58:25] I'm assuming it's ok as long as you reboot the server [13:00:27] also I don't think the rp_filter is useful on most of our servers, it's quite a niche attack surface [13:13:06] 10Traffic, 10Beta-Cluster-Infrastructure, 10Beta-Cluster-reproducible: HTTP 504 connection timeout error accessing MW API on Beta cluster - https://phabricator.wikimedia.org/T351930 (10Jgiannelos) I am getting test failures on restbase because of enwiki beta Action API returning 504. I think I first encounte... [13:52:03] XioNoX: in servers with one NIC is basically useless IMHO [14:47:28] 10Traffic, 10DC-Ops, 10SRE, 10ops-eqiad, 10Patch-For-Review: Q1:Install cp11[00-15] and rotate into production - https://phabricator.wikimedia.org/T349244 (10CDanis) >>! In T349244#9360471, @Fabfur wrote: > Looping in @CDanis as the original author for the [[ https://gerrit.wikimedia.org/r/c/operations/p... [20:51:42] 10Traffic, 10Patch-For-Review: Simplify maintenance of DNS/NTP hosts to reduce toil around reboots, reimages, and other work - https://phabricator.wikimedia.org/T347054 (10KOfori)