[06:04:09] FIRING: LVSHighRX: Excessive RX traffic on lvs2013:9100 (eno12399np0) - https://bit.ly/wmf-lvsrx - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=lvs2013 - https://alerts.wikimedia.org/?q=alertname%3DLVSHighRX [06:24:09] RESOLVED: LVSHighRX: Excessive RX traffic on lvs2013:9100 (eno12399np0) - https://bit.ly/wmf-lvsrx - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=lvs2013 - https://alerts.wikimedia.org/?q=alertname%3DLVSHighRX [07:48:09] FIRING: LVSHighRX: Excessive RX traffic on lvs2013:9100 (eno12399np0) - https://bit.ly/wmf-lvsrx - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=lvs2013 - https://alerts.wikimedia.org/?q=alertname%3DLVSHighRX [08:08:09] RESOLVED: LVSHighRX: Excessive RX traffic on lvs2013:9100 (eno12399np0) - https://bit.ly/wmf-lvsrx - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=lvs2013 - https://alerts.wikimedia.org/?q=alertname%3DLVSHighRX [08:56:09] FIRING: LVSHighRX: Excessive RX traffic on lvs2013:9100 (eno12399np0) - https://bit.ly/wmf-lvsrx - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=lvs2013 - https://alerts.wikimedia.org/?q=alertname%3DLVSHighRX [09:06:09] RESOLVED: LVSHighRX: Excessive RX traffic on lvs2013:9100 (eno12399np0) - https://bit.ly/wmf-lvsrx - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=lvs2013 - https://alerts.wikimedia.org/?q=alertname%3DLVSHighRX [09:27:31] 06Traffic: Private TLS material (TLS keys) should be stored in volatile storage only - https://phabricator.wikimedia.org/T384227#10681833 (10Fabfur) [10:47:28] 10netops, 06Infrastructure-Foundations, 10ops-drmrs: cr1-drmrs to asw1-b12-drmrs link down - https://phabricator.wikimedia.org/T389071#10682142 (10cmooney) This has been stable ever since the replacement fwiw, no drops or errors etc. So I think we are good to close it {F58929621 width=600} [10:52:09] FIRING: LVSHighRX: Excessive RX traffic on lvs2013:9100 (eno12399np0) - https://bit.ly/wmf-lvsrx - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=lvs2013 - https://alerts.wikimedia.org/?q=alertname%3DLVSHighRX [10:57:09] RESOLVED: LVSHighRX: Excessive RX traffic on lvs2013:9100 (eno12399np0) - https://bit.ly/wmf-lvsrx - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=lvs2013 - https://alerts.wikimedia.org/?q=alertname%3DLVSHighRX [11:10:09] FIRING: LVSHighRX: Excessive RX traffic on lvs2013:9100 (eno12399np0) - https://bit.ly/wmf-lvsrx - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=lvs2013 - https://alerts.wikimedia.org/?q=alertname%3DLVSHighRX [11:15:09] RESOLVED: LVSHighRX: Excessive RX traffic on lvs2013:9100 (eno12399np0) - https://bit.ly/wmf-lvsrx - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=lvs2013 - https://alerts.wikimedia.org/?q=alertname%3DLVSHighRX [11:36:09] FIRING: LVSHighRX: Excessive RX traffic on lvs2013:9100 (eno12399np0) - https://bit.ly/wmf-lvsrx - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=lvs2013 - https://alerts.wikimedia.org/?q=alertname%3DLVSHighRX [11:41:09] RESOLVED: LVSHighRX: Excessive RX traffic on lvs2013:9100 (eno12399np0) - https://bit.ly/wmf-lvsrx - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=lvs2013 - https://alerts.wikimedia.org/?q=alertname%3DLVSHighRX [13:29:56] 10netops, 06Infrastructure-Foundations, 06SRE, 07IPv6: WMCS Eqiad: Enable IPv6 in cloud vrf on switches - https://phabricator.wikimedia.org/T389958#10682999 (10aborrero) there was a major network outage as a result of the operations that affected all WMCS systems, including Ceph and Toolforge kubernetes. [14:19:12] 06Traffic, 06serviceops, 06Wikimedia Enterprise, 10Content-Transform-Team (Work In Progress), 13Patch-For-Review: Restbase API returns 404 for some articles with revision - https://phabricator.wikimedia.org/T389628#10683237 (10SLopes-WMF) [14:36:51] 10netops, 06Infrastructure-Foundations, 10ops-drmrs: cr1-drmrs to asw1-b12-drmrs link down - https://phabricator.wikimedia.org/T389071#10683332 (10RobH) 05Open→03Resolved Awesome, I've updated the ticket to Interxion so they can close it. [14:58:43] 10netops, 06Infrastructure-Foundations, 06SRE, 07IPv6: WMCS Eqiad: Enable IPv6 in cloud vrf on switches - https://phabricator.wikimedia.org/T389958#10683425 (10cmooney) Just to confirm the timeline of events: (IMPORTANT) Mar 27 13:06:57: IBGP configuration commited on all 4 cloudsw, enabling IBGP in the... [15:26:54] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: Install and cable Nokia test devices and test servers in codfw - https://phabricator.wikimedia.org/T385217#10683573 (10Jhancock.wm) [15:33:08] 10netops, 06Infrastructure-Foundations, 06SRE, 07IPv6: WMCS Eqiad: Enable IPv6 in cloud vrf on switches - https://phabricator.wikimedia.org/T389958#10683594 (10cmooney) @aborrero @taavi one thing we could maybe try, if we wanted to make progress sooner (i.e. without replicating the setup elsewhere): * Add... [16:10:02] 10netops, 06Infrastructure-Foundations, 10Observability-Alerting, 13Patch-For-Review: Migrate network icinga alerts to gNMI/prometheus - https://phabricator.wikimedia.org/T388641#10683734 (10cmooney) >>! In T388641#10678444, @ayounsi wrote: > @cmooney another question is that if the service is not present... [16:24:42] 06Traffic, 10RESTBase, 10RESTBase Sunsetting, 06serviceops-radar, 10Content-Transform-Team (Work In Progress): Block traffic to RESTBase /page/related endpoint and sunset it - https://phabricator.wikimedia.org/T376297#10683766 (10Jgiannelos) 05Open→03Resolved [16:27:53] hello! I have modified gateway-check a little bit to add the ability to ignore rules, so that we can roll out changes to most of our wikis without needing to manually list each one. Would someone mind having a look please? I realise it's fairly late on a thursday though so I can hold on rolling this out until next week https://gerrit.wikimedia.org/r/c/operations/puppet/+/1131748 [16:33:38] 06Traffic, 10Beta-Cluster-Infrastructure, 07Documentation: Create a runbook for troubleshooting the CDN in deployment-prep - https://phabricator.wikimedia.org/T390213 (10bd808) 03NEW [17:06:49] 06Traffic: Unify CDN ats/haproxy/varnish upgrade cookbooks - https://phabricator.wikimedia.org/T390094#10683999 (10BCornwall) Okay! That's what I had figured. Thanks for the clarification. My vote would be to continue on as in the current CR - while having unification in __init__ is tempting, I think it will be... [17:46:30] 06Traffic, 06serviceops, 06Wikimedia Enterprise, 10Content-Transform-Team (Work In Progress), 13Patch-For-Review: Restbase API returns 404 for some articles with revision - https://phabricator.wikimedia.org/T389628#10684130 (10hnowlan) r/1130728 has been deployed and pages now appear to render correctly. [17:49:51] 06Traffic, 06serviceops, 06Wikimedia Enterprise, 10Content-Transform-Team (Work In Progress), 13Patch-For-Review: Restbase API returns 404 for some articles with revision - https://phabricator.wikimedia.org/T389628#10684161 (10ABreault-WMF) 05Open→03Resolved a:03ABreault-WMF @hnowlan Thanks for... [17:50:48] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-ulsfo, 06SRE: InboundInterfaceErrors - https://phabricator.wikimedia.org/T389884#10684167 (10RobH) [17:53:34] 06Traffic, 06serviceops, 06Wikimedia Enterprise, 10Content-Transform-Team (Work In Progress), 13Patch-For-Review: Restbase API returns 404 for some articles with revision - https://phabricator.wikimedia.org/T389628#10684172 (10ABreault-WMF) > Franklin D. Roosevelt: https://en.wikipedia.org/api/rest_v... [19:41:34] 06Traffic, 06Fundraising-Backlog, 06Fundraising-Tech-Roadmap, 07fr-acoustic: Acoustic SMS: Domain needed for short links - https://phabricator.wikimedia.org/T379318#10684614 (10AKanji-WMF) @greg, @XenoRyet and I are reviewing this against the next quarter's SMS projects - what are the next steps on this? [19:48:32] 06Traffic, 06DC-Ops, 10ops-eqiad, 06SRE: Q3:test NIC for lvs1017 or lvs1018 - https://phabricator.wikimedia.org/T387145#10684631 (10wiki_willy) [21:17:15] 06Traffic, 13Patch-For-Review: Upgrade Varnish from 6.0 to 7.1 - https://phabricator.wikimedia.org/T378737#10684826 (10BCornwall)