[06:48:56] (EdgeTrafficDrop) firing: 69% request drop in text@codfw during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=codfw&var-cache_type=text - https://alerts.wikimedia.org [06:53:56] (EdgeTrafficDrop) resolved: 69% request drop in text@codfw during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=codfw&var-cache_type=text - https://alerts.wikimedia.org [06:57:56] (EdgeTrafficDrop) firing: 69% request drop in text@codfw during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=codfw&var-cache_type=text - https://alerts.wikimedia.org [07:02:56] (EdgeTrafficDrop) resolved: 69% request drop in text@codfw during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=codfw&var-cache_type=text - https://alerts.wikimedia.org [09:31:15] hello traffic! Just to know, when do you plan to add drmrs to the 'datacenters' hiera key in hieradata/common.yaml? [10:01:42] 10netops, 10Infrastructure-Foundations, 10ops-codfw, 10cloud-services-team (Kanban): conncet 2nd cloudcontrol200x-dev NIC to vlan 2105 - https://phabricator.wikimedia.org/T297588 (10aborrero) [10:18:21] 10netops, 10Infrastructure-Foundations, 10ops-codfw, 10cloud-services-team (Kanban): conncet 2nd cloudcontrol200x-dev NIC to vlan 2105 - https://phabricator.wikimedia.org/T297588 (10aborrero) [10:18:43] 10netops, 10Infrastructure-Foundations, 10SRE, 10cloud-services-team (Kanban): cloud: decide on general idea for having cloud-dedicated hardware provide service in the cloud realm & the internet - https://phabricator.wikimedia.org/T296411 (10aborrero) p:05Triage→03Medium [10:19:18] 10netops, 10Infrastructure-Foundations, 10ops-codfw, 10cloud-services-team (Kanban): conncet 2nd cloudcontrol200x-dev NIC to vlan 2105 - https://phabricator.wikimedia.org/T297588 (10aborrero) p:05Triage→03Medium a:03Papaul [12:24:40] 10netops, 10Infrastructure-Foundations: Upgrade netflow VMs to Bullseye - https://phabricator.wikimedia.org/T297595 (10ayounsi) p:05Triage→03Medium [12:24:55] 10netops, 10Infrastructure-Foundations: Upgrade netflow VMs to Bullseye - https://phabricator.wikimedia.org/T297595 (10ayounsi) [12:24:58] 10netops, 10Data-Engineering, 10Infrastructure-Foundations, 10SRE, 10Traffic-Icebox: Collect netflow data for internal traffic - https://phabricator.wikimedia.org/T263277 (10ayounsi) [12:39:55] 10netops, 10Infrastructure-Foundations, 10SRE, 10cloud-services-team (Kanban): cloud: decide on general idea for having cloud-dedicated hardware provide service in the cloud realm & the internet - https://phabricator.wikimedia.org/T296411 (10aborrero) [14:39:49] 10netops, 10Infrastructure-Foundations, 10SRE: Increase in prefix announcements from AS15169 - https://phabricator.wikimedia.org/T297609 (10MatthewVernon) [14:41:39] ahoyhoy traffic, any chance I could get review on https://gerrit.wikimedia.org/r/c/operations/puppet/+/745560 please? [15:00:32] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-codfw, 10cloud-services-team (Kanban): conncet 2nd cloudcontrol200x-dev NIC to vlan 2105 - https://phabricator.wikimedia.org/T297588 (10Papaul) @aborrero can we test this for now on only one server to see if it works before moving it to the other serve... [15:01:07] 10netops, 10Infrastructure-Foundations, 10SRE: Increase in prefix announcements from AS15169 - https://phabricator.wikimedia.org/T297609 (10jbond) i don't see either of those route servers in our config as such i think there is no action. but pinging @ayounsi @cmooney to confirm [15:07:40] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-codfw, 10cloud-services-team (Kanban): conncet 2nd cloudcontrol200x-dev NIC to vlan 2105 - https://phabricator.wikimedia.org/T297588 (10Papaul) [15:07:59] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-codfw, 10cloud-services-team (Kanban): conncet 2nd cloudcontrol200x-dev NIC to vlan 2105 - https://phabricator.wikimedia.org/T297588 (10Papaul) [15:51:51] 10netops, 10Infrastructure-Foundations, 10SRE: Increase in prefix announcements from AS15169 - https://phabricator.wikimedia.org/T297609 (10ayounsi) 05Open→03Resolved a:03ayounsi Thanks, we're already at 250000 for those. We usually set a high limit from the get go for route servers. [16:08:18] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-codfw, 10cloud-services-team (Kanban): conncet 2nd cloudcontrol200x-dev NIC to vlan 2105 - https://phabricator.wikimedia.org/T297588 (10ayounsi) Could we trunk the new vlan instead of using a 2nd physical port? [16:18:26] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-codfw, 10cloud-services-team (Kanban): conncet 2nd cloudcontrol200x-dev NIC to vlan 2105 - https://phabricator.wikimedia.org/T297588 (10aborrero) >>! In T297588#7566526, @Papaul wrote: > @aborrero can we test this for now on only one server to see if i... [17:37:46] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-codfw, 10cloud-services-team (Kanban): conncet 2nd cloudcontrol200x-dev NIC to vlan 2105 - https://phabricator.wikimedia.org/T297588 (10aborrero) [18:22:32] 10Traffic, 10Foundational Technology Requests, 10SRE, 10Wikimedia Enterprise, and 2 others: Allow-Listing for Enterprise IPs - https://phabricator.wikimedia.org/T294798 (10hnowlan) This has been merged and will come into effect over the next 25 minutes or so. [19:30:10] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-codfw, 10cloud-services-team (Kanban): connect 2nd cloudcontrol200x-dev NIC to vlan 2105 - https://phabricator.wikimedia.org/T297588 (10Dzahn) [22:21:39] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-codfw, 10cloud-services-team (Kanban): connect 2nd cloudcontrol200x-dev NIC to vlan 2105 - https://phabricator.wikimedia.org/T297588 (10Papaul) The 2nd NIC is connected to port ge-1/0/34 only thing left is to do the config in Netbox.