[08:11:30] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-drmrs: cr2-drmrs:xe-0/1/1 stuck optic - https://phabricator.wikimedia.org/T324555 (10ayounsi) Thanks for those great pictures! As a first step (and I think that's what you suggested previously!) it might be worth asking Interxion's remote hands if that'... [12:04:47] 10netops, 10Data-Engineering, 10Infrastructure-Foundations, 10Product-Analytics, and 3 others: Maybe restrict domains accessible by webproxy - https://phabricator.wikimedia.org/T300977 (10ayounsi) [12:29:56] 10netops, 10Data-Engineering, 10Infrastructure-Foundations, 10Product-Analytics, and 3 others: Maybe restrict domains accessible by webproxy - https://phabricator.wikimedia.org/T300977 (10ayounsi) [12:30:43] 10netops, 10Data-Engineering, 10Infrastructure-Foundations, 10Product-Analytics, and 3 others: Maybe restrict domains accessible by webproxy - https://phabricator.wikimedia.org/T300977 (10ayounsi) [12:31:02] 10netops, 10Data-Engineering, 10Infrastructure-Foundations, 10Product-Analytics, and 3 others: Maybe restrict domains accessible by webproxy - https://phabricator.wikimedia.org/T300977 (10ayounsi) [13:13:20] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-eqiad: [eqiad] faulty VC optics - https://phabricator.wikimedia.org/T325803 (10ayounsi) [13:16:19] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-eqiad: [eqiad] faulty VC optics - https://phabricator.wikimedia.org/T325803 (10ayounsi) [13:16:31] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-eqiad: [eqiad] faulty VC optics - https://phabricator.wikimedia.org/T325803 (10ayounsi) I had a look at the eqiad counters and updated the task description with what has increased. We should replace the optic on the 5 interfaces that stand out. [14:15:48] 10netops, 10Data-Engineering, 10Infrastructure-Foundations, 10Product-Analytics, and 3 others: Maybe restrict domains accessible by webproxy - https://phabricator.wikimedia.org/T300977 (10ayounsi) [14:24:21] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-eqiad: [eqiad] faulty VC optics - https://phabricator.wikimedia.org/T325803 (10ayounsi) Everything has been replaced, thanks @Jclark-ctr! I'll check it on Monday to see if there are any ongoing errors and close it if it's good! [14:40:33] 10netops, 10Infrastructure-Foundations, 10SRE: Add per-output queue graphing for Juniper network devices in LibreNMS - https://phabricator.wikimedia.org/T326322 (10cmooney) p:05Triage→03Medium [14:47:26] 10CAS-SSO, 10Infrastructure-Foundations, 10SRE, 10User-jbond: CLI tools for CAS administration - https://phabricator.wikimedia.org/T233940 (10jbond) also possibly useful: https://apereo.github.io/cas/6.5.x/installation/Configuring-Commandline-Shell.html [15:41:01] I lost my internet connection [15:41:55] we're waiting for ou, no worries [17:35:20] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-drmrs: cr2-drmrs:xe-0/1/1 stuck optic - https://phabricator.wikimedia.org/T324555 (10RobH) >>! In T324555#8500823, @ayounsi wrote: > Thanks for those great pictures! > > As a first step (and I think that's what you suggested previously!) it might be wor... [17:57:15] 10Puppet, 10Infrastructure-Foundations, 10SRE: Knock down puppet 4 deprecation warnings - https://phabricator.wikimedia.org/T193664 (10herron) [17:57:32] 10Mail, 10Infrastructure-Foundations, 10SRE, 10observability, 10Sustainability (Incident Followup): Graph outbound mail volume on per-service or hostgroup level - https://phabricator.wikimedia.org/T197171 (10herron) [17:57:58] 10Mail, 10Infrastructure-Foundations, 10SRE, 10observability, 10Sustainability (Incident Followup): Improve outbound mail service alerting - https://phabricator.wikimedia.org/T197172 (10herron) [17:58:22] 10Mail, 10Growth-Team, 10Growth-Team-Filtering, 10Infrastructure-Foundations, and 3 others: SRE query: Is it possible to measure how many e-mails are sent to "black hole" e-mail addresses? - https://phabricator.wikimedia.org/T202329 (10herron) [20:02:50] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-eqiad: [eqiad] faulty VC optics - https://phabricator.wikimedia.org/T325803 (10Jclark-ctr) Thanks Ayounsi! if any issues come back i still do have more spare optics. would Monday be a good day to schedule line card moves also if no errors return?