[03:11:46] 10Traffic, 10DNS, 10SRE, 10WMF-Communications: Setup subdomain for Foundation messaging site - https://phabricator.wikimedia.org/T296570 (10Varnent) I also have the certificates info from VIP - and can share that with whomever will need it - presuming that is something we will need. [09:01:56] (EdgeTrafficDrop) firing: 57% request drop in text@codfw during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=codfw&var-cache_type=text - https://alerts.wikimedia.org [09:06:56] (EdgeTrafficDrop) resolved: 66% request drop in text@codfw during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=codfw&var-cache_type=text - https://alerts.wikimedia.org [13:59:35] 10netops, 10Infrastructure-Foundations, 10SRE: Enable NTP for drmrs network devices - https://phabricator.wikimedia.org/T296623 (10cmooney) Ok so this has been addressed for CR routers. You can view the NTP status as follows: ` cmooney@cr2-eqord> show ntp associations remote refid st t... [14:00:49] 10netops, 10Infrastructure-Foundations, 10SRE: Enable NTP for drmrs network devices - https://phabricator.wikimedia.org/T296623 (10cmooney) 05Open→03Resolved Scrap that it does seem to be working, perhaps it only failed to query against itself after the initial change. ` cmooney@mr1-drmrs> show ntp assoc... [14:12:25] 10Traffic, 10SRE, 10ops-drmrs: Degraded RAID on cp6002 - https://phabricator.wikimedia.org/T295747 (10BBlack) 05Open→03Invalid This was a false alarm due to monitoring anomalies while first bringing up the host. [14:19:43] 10netops, 10Data-Engineering, 10Infrastructure-Foundations, 10SRE, and 2 others: Collect netflow data for internal traffic - https://phabricator.wikimedia.org/T263277 (10BTullis) Tagging #data-engineering because we will likely be managing the Gobblin and/or Druid ingestion parts of this pipeline. Should w... [15:27:56] (EdgeTrafficDrop) firing: 68% request drop in text@eqiad during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=eqiad&var-cache_type=text - https://alerts.wikimedia.org [15:32:56] (EdgeTrafficDrop) resolved: 68% request drop in text@eqiad during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=eqiad&var-cache_type=text - https://alerts.wikimedia.org [15:34:56] (EdgeTrafficDrop) firing: 66% request drop in text@eqiad during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=eqiad&var-cache_type=text - https://alerts.wikimedia.org [15:39:56] (EdgeTrafficDrop) firing: (2) 60% request drop in text@eqiad during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://alerts.wikimedia.org [15:44:56] (EdgeTrafficDrop) resolved: (2) 68% request drop in text@eqiad during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://alerts.wikimedia.org [17:13:26] 10HTTPS, 10SRE, 10Traffic-Icebox: HTTPS for internal service traffic - https://phabricator.wikimedia.org/T108580 (10Majavah) [17:13:30] 10HTTPS, 10Continuous-Integration-Infrastructure, 10SRE, 10Traffic-Icebox: contint.wikimedia.org: add TLS termination - https://phabricator.wikimedia.org/T263830 (10Majavah) 05Open→03Resolved a:03Majavah [18:03:30] 10netops, 10Infrastructure-Foundations, 10SRE, 10cloud-services-team (Kanban): cloud: decide on general idea for having cloud-dedicated hardware provide service in the cloud realm & the internet - https://phabricator.wikimedia.org/T296411 (10aborrero) I did not forget this task, but have been busy the last... [18:04:09] 10netops, 10Data-Engineering, 10Infrastructure-Foundations, 10SRE, and 2 others: Collect netflow data for internal traffic - https://phabricator.wikimedia.org/T263277 (10ayounsi) @BTullis thanks! Real-time, would be a nice plus, but a hard requirement (unlike netflow). @cmooney [[ https://gerrit.wikimedia... [19:28:23] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10SRE, 10ops-ulsfo: ulsfo: (2) mx80s to become temp cr[34]-drmrs - https://phabricator.wikimedia.org/T295819 (10RobH) [19:29:29] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10SRE, 10ops-ulsfo: ulsfo: (2) mx80s to become temp cr[34]-drmrs - https://phabricator.wikimedia.org/T295819 (10RobH) So both of the old mx80s are attached to mgmt, power, and scs. I've not bothered to document these connections in netbox, as they are un... [19:31:32] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10SRE, 10ops-ulsfo: ulsfo: (2) mx80s to become temp cr[34]-drmrs - https://phabricator.wikimedia.org/T295819 (10RobH) a:05RobH→03ayounsi I'm not sure if this should assign to @ayounsi or @cmooney, either can handle this! This is ready for software up... [20:42:14] 10HTTPS, 10Continuous-Integration-Infrastructure, 10SRE, 10Traffic-Icebox: contint.wikimedia.org: add TLS termination - https://phabricator.wikimedia.org/T263830 (10Dzahn) Wow @Majavah thanks for closing this! :) Just a bit sad that it was still not triaged in CI infra and people probably won't notice. [21:31:55] 10Traffic, 10DNS, 10SRE, 10WMF-Communications, 10Patch-For-Review: Setup subdomain for Foundation messaging site - https://phabricator.wikimedia.org/T296570 (10ssingh) 05Open→03Resolved a:03ssingh ` $ dig one.wikimedia.org CNAME +short messaging-wikimedia-org.go-vip.net. ` [22:14:12] 10Traffic, 10DNS, 10SRE, 10WMF-Communications: Setup subdomain for Foundation messaging site - https://phabricator.wikimedia.org/T296570 (10Varnent) Thank you so much @ssingh!