[10:12:25] 10netops, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: Gaps in gNMI network statistics in eqiad - https://phabricator.wikimedia.org/T386807#10570490 (10cmooney) I increased the "cache timeout" for stats received from routers in eqiad, and upped the number of threads for the prometheus output fr... [10:42:16] 10netops, 06Infrastructure-Foundations, 10ops-magru, 06SRE: cr2-magru errors on xe-0/1/0 (EdgeUno Transit) - https://phabricator.wikimedia.org/T387006 (10cmooney) 03NEW p:05Triage→03High [10:42:53] 10netops, 06Infrastructure-Foundations, 10ops-magru, 06SRE: cr2-magru errors on xe-0/1/0 (EdgeUno Transit) - https://phabricator.wikimedia.org/T387006#10570549 (10cmooney) [10:46:17] 10netops, 06Infrastructure-Foundations, 10ops-magru, 06SRE: cr2-magru errors on xe-0/1/0 (EdgeUno Transit) - https://phabricator.wikimedia.org/T387006#10570552 (10cmooney) [11:40:39] 10netops, 06Infrastructure-Foundations, 10ops-magru, 06SRE: cr2-magru errors on xe-0/1/0 (EdgeUno Transit) - https://phabricator.wikimedia.org/T387006#10570708 (10cmooney) [13:21:52] 10netops, 06Infrastructure-Foundations, 10ops-magru, 06SRE: cr2-magru errors on xe-0/1/0 (EdgeUno Transit) - https://phabricator.wikimedia.org/T387006#10570948 (10RobH) @cmooney, Can we have the carrier check their end first, since we'll incur hourly billing for our check? [13:35:36] 10netops, 06Infrastructure-Foundations, 06SRE: gNMIc connection not working for cloudsw2-d5-eqiad - https://phabricator.wikimedia.org/T387018 (10cmooney) 03NEW p:05Triage→03Low [13:51:30] 06Traffic, 10Maps, 06SRE, 13Patch-For-Review: Allow Wikimedia Maps usage on schoolwiki.in - https://phabricator.wikimedia.org/T383210#10571019 (10MSantos) Approved. [14:12:48] 06Traffic, 10Maps, 06SRE, 13Patch-For-Review: Allow Wikimedia Maps usage on schoolwiki.in - https://phabricator.wikimedia.org/T383210#10571103 (10ssingh) Thanks @MSantos! @Gnoeee: We will merge this on Monday (Feb 24). [14:39:00] 10netops, 06Infrastructure-Foundations, 10ops-magru, 06SRE: cr2-magru errors on xe-0/1/0 (EdgeUno Transit) - https://phabricator.wikimedia.org/T387006#10571213 (10cmooney) >>! In T387006#10570948, @RobH wrote: > @cmooney, > > Can we have the carrier check their end first, since we'll incur hourly billing... [14:48:18] 10netops, 06Infrastructure-Foundations, 10ops-magru, 06SRE: cr2-magru errors on xe-0/1/0 (EdgeUno Transit) - https://phabricator.wikimedia.org/T387006#10571241 (10cmooney) Ticket #341504 created. [16:28:09] FIRING: LVSHighRX: Excessive RX traffic on lvs2013:9100 (eno12399np0) - https://bit.ly/wmf-lvsrx - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=lvs2013 - https://alerts.wikimedia.org/?q=alertname%3DLVSHighRX [16:33:09] RESOLVED: LVSHighRX: Excessive RX traffic on lvs2013:9100 (eno12399np0) - https://bit.ly/wmf-lvsrx - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=lvs2013 - https://alerts.wikimedia.org/?q=alertname%3DLVSHighRX [19:20:10] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: Install and cable Nokia test devices and test servers in codfw - https://phabricator.wikimedia.org/T385217#10572041 (10cmooney) Bit of an update on this one. I was able to get on to three of the devices with the default password for t... [19:21:17] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: Install and cable Nokia test devices and test servers in codfw - https://phabricator.wikimedia.org/T385217#10572043 (10cmooney) Also @Jhancock.wm not sure if you want to close this task at this point, or if we want to keep it open to d...