[01:03:08] 10Traffic, 10SRE: oom killed varnish on cp4052 - https://phabricator.wikimedia.org/T325797 (10BBlack) Summarizing some of the lengthy IRC discussion and investigation on this topic (most of which was @Vgutierrez !): We seem to have a likely candidate mechanism for how this is happening, and it has to do with... [08:11:32] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-drmrs: cr2-drmrs:xe-0/1/1 stuck optic - https://phabricator.wikimedia.org/T324555 (10ayounsi) Thanks for those great pictures! As a first step (and I think that's what you suggested previously!) it might be worth asking Interxion's remote hands if that'... [12:04:49] 10netops, 10Data-Engineering, 10Infrastructure-Foundations, 10Product-Analytics, and 3 others: Maybe restrict domains accessible by webproxy - https://phabricator.wikimedia.org/T300977 (10ayounsi) [12:29:58] 10netops, 10Data-Engineering, 10Infrastructure-Foundations, 10Product-Analytics, and 3 others: Maybe restrict domains accessible by webproxy - https://phabricator.wikimedia.org/T300977 (10ayounsi) [12:30:45] 10netops, 10Data-Engineering, 10Infrastructure-Foundations, 10Product-Analytics, and 3 others: Maybe restrict domains accessible by webproxy - https://phabricator.wikimedia.org/T300977 (10ayounsi) [12:31:04] 10netops, 10Data-Engineering, 10Infrastructure-Foundations, 10Product-Analytics, and 3 others: Maybe restrict domains accessible by webproxy - https://phabricator.wikimedia.org/T300977 (10ayounsi) [13:07:27] hi traffic could someone please take a look at this change https://gerrit.wikimedia.org/r/c/operations/puppet/+/875888 its to remove profile::cache::envoy which appears to be unsued and also broken (cc kwakuofori) [13:13:22] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-eqiad: [eqiad] faulty VC optics - https://phabricator.wikimedia.org/T325803 (10ayounsi) [13:16:21] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-eqiad: [eqiad] faulty VC optics - https://phabricator.wikimedia.org/T325803 (10ayounsi) [13:16:33] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-eqiad: [eqiad] faulty VC optics - https://phabricator.wikimedia.org/T325803 (10ayounsi) I had a look at the eqiad counters and updated the task description with what has increased. We should replace the optic on the 5 interfaces that stand out. [13:44:43] jbond: sure [13:52:14] thanks [14:15:50] 10netops, 10Data-Engineering, 10Infrastructure-Foundations, 10Product-Analytics, and 3 others: Maybe restrict domains accessible by webproxy - https://phabricator.wikimedia.org/T300977 (10ayounsi) [14:24:22] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-eqiad: [eqiad] faulty VC optics - https://phabricator.wikimedia.org/T325803 (10ayounsi) Everything has been replaced, thanks @Jclark-ctr! I'll check it on Monday to see if there are any ongoing errors and close it if it's good! [14:40:35] 10netops, 10Infrastructure-Foundations, 10SRE: Add per-output queue graphing for Juniper network devices in LibreNMS - https://phabricator.wikimedia.org/T326322 (10cmooney) p:05Triage→03Medium [17:35:22] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-drmrs: cr2-drmrs:xe-0/1/1 stuck optic - https://phabricator.wikimedia.org/T324555 (10RobH) >>! In T324555#8500823, @ayounsi wrote: > Thanks for those great pictures! > > As a first step (and I think that's what you suggested previously!) it might be wor... [18:01:09] 10Traffic, 10SRE: Review cp2041 and cp2042 running bullseye - https://phabricator.wikimedia.org/T325557 (10ssingh) Filed a bug report with Debian: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1027994 [18:02:25] 10Traffic, 10SRE, 10Upstream: Review cp2041 and cp2042 running bullseye - https://phabricator.wikimedia.org/T325557 (10Vgutierrez) [20:02:51] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-eqiad: [eqiad] faulty VC optics - https://phabricator.wikimedia.org/T325803 (10Jclark-ctr) Thanks Ayounsi! if any issues come back i still do have more spare optics. would Monday be a good day to schedule line card moves also if no errors return?