[03:48:52] 10Traffic, 10DC-Ops, 10SRE, 10Sustainability (Incident Followup): Audit eqiad & codfw LVS network links - https://phabricator.wikimedia.org/T286881 (10Papaul) |Host| Host iface| switch iface|switch name| change notes|iface on new siwtch |lvs2007|ens2f0np0|xe-2/0/45|asw-a2-codfw|no change| no change |lvs200... [03:53:35] 10Traffic, 10DC-Ops, 10SRE, 10Sustainability (Incident Followup): Audit eqiad & codfw LVS network links - https://phabricator.wikimedia.org/T286881 (10Papaul) >>! In T286881#7242985, @Vgutierrez wrote: > |Host |Row |Host iface |switch iface|switch name| > |lvs2007|**A**|ens2f0np0|xe-2/0/45|A2| > |lvs2008|A... [04:48:48] 10Traffic, 10netops, 10DBA, 10Infrastructure-Foundations, and 3 others: Switch buffer re-partition - Eqiad Row A - https://phabricator.wikimedia.org/T286032 (10Marostegui) m2-master failed over from dbproxy1013 to dbproxy1015. Once the maintenance is done we need to revert this. [04:49:16] 10Traffic, 10netops, 10DBA, 10Infrastructure-Foundations, and 3 others: Switch buffer re-partition - Eqiad Row A - https://phabricator.wikimedia.org/T286032 (10Marostegui) [05:07:45] 10Traffic, 10netops, 10DBA, 10Infrastructure-Foundations, and 3 others: Switch buffer re-partition - Eqiad Row A - https://phabricator.wikimedia.org/T286032 (10Bstorm) [05:08:32] 10Traffic, 10netops, 10DBA, 10Infrastructure-Foundations, and 3 others: Switch buffer re-partition - Eqiad Row A - https://phabricator.wikimedia.org/T286032 (10Bstorm) [07:20:16] 10Traffic, 10Maps, 10Product-Infrastructure-Team-Backlog, 10SRE, and 2 others: Support maps serving for affiliate sites via an allow list - https://phabricator.wikimedia.org/T261694 (10Aklapper) +1 to Legoktm's last comment. "Add a comment to this task" makes this a neverending open ticket, though tickets... [07:56:02] 10Traffic, 10netops, 10DBA, 10Infrastructure-Foundations, and 2 others: Switch buffer re-partition - Eqiad Row A - https://phabricator.wikimedia.org/T286032 (10MoritzMuehlenhoff) [08:26:32] 10Traffic, 10netops, 10DBA, 10Infrastructure-Foundations, and 2 others: Switch buffer re-partition - Eqiad Row A - https://phabricator.wikimedia.org/T286032 (10elukey) [12:08:49] 10Traffic, 10netops, 10DBA, 10Infrastructure-Foundations, and 2 others: Switch buffer re-partition - Eqiad Row A - https://phabricator.wikimedia.org/T286032 (10cmooney) [12:13:42] 10Traffic, 10netops, 10DBA, 10Infrastructure-Foundations, and 2 others: Switch buffer re-partition - Eqiad Row A - https://phabricator.wikimedia.org/T286032 (10cmooney) [12:14:52] 10Traffic, 10netops, 10DBA, 10Infrastructure-Foundations, and 2 others: Switch buffer re-partition - Eqiad Row A - https://phabricator.wikimedia.org/T286032 (10cmooney) [12:34:21] 10Traffic, 10DC-Ops, 10SRE, 10Sustainability (Incident Followup): Audit eqiad & codfw LVS network links - https://phabricator.wikimedia.org/T286881 (10Vgutierrez) @Papaul that's a mistake on my side, thanks for spotting it, the second NIC `ens2f1np1` is actually connected to `B7` [12:38:50] 10Traffic, 10netops, 10DBA, 10Infrastructure-Foundations, and 2 others: Switch buffer re-partition - Eqiad Row A - https://phabricator.wikimedia.org/T286032 (10MoritzMuehlenhoff) [12:54:02] 10Traffic, 10netops, 10DBA, 10Infrastructure-Foundations, and 2 others: Switch buffer re-partition - Eqiad Row A - https://phabricator.wikimedia.org/T286032 (10MoritzMuehlenhoff) [12:54:53] 10Traffic, 10netops, 10DBA, 10Infrastructure-Foundations, and 2 others: Switch buffer re-partition - Eqiad Row A - https://phabricator.wikimedia.org/T286032 (10MoritzMuehlenhoff) [13:19:09] 10Traffic, 10DC-Ops, 10SRE, 10Sustainability (Incident Followup): Audit eqiad & codfw LVS network links - https://phabricator.wikimedia.org/T286881 (10Papaul) @Vgutierrez thank you. What about lvs2007 ens3f1np1? Actually it is connected to d7 and you want it to be moved to C7 or lvs2007 ens3f0np0 is alread... [13:21:08] 10Traffic, 10DC-Ops, 10SRE, 10Sustainability (Incident Followup): Audit eqiad & codfw LVS network links - https://phabricator.wikimedia.org/T286881 (10Vgutierrez) @Papaul same thing.. lvs2007 ens3f1np1 is connected to D7, the only desired changes are the new links against A4, B4, C4 and D4 [13:29:05] 10Traffic, 10netops, 10DBA, 10Infrastructure-Foundations, and 2 others: Switch buffer re-partition - Eqiad Row A - https://phabricator.wikimedia.org/T286032 (10MoritzMuehlenhoff) [14:09:26] 10Traffic, 10DC-Ops, 10SRE, 10Sustainability (Incident Followup): Audit eqiad & codfw LVS network links - https://phabricator.wikimedia.org/T286881 (10Papaul) @Vgutierrez thank you I have all the information needed. I will do my site audit and get back with you next week to setup a day and time to start m... [14:30:30] vgutierrez: just a heads up I'll be performing that buffer change on eqiad row A in 30 mins. [14:30:37] ack [14:30:44] we will begin our tasks soon [14:30:51] actually mmandere is running a few seconds late [14:30:56] great thank you :) [14:31:08] but I guess we can cut him some slack 😈 [14:31:16] there is no rush, can hold off until everyone is ready. [14:31:26] it won't be necessary :) just kidding [14:31:32] lol [14:32:06] Sorry, I am now around :) [14:38:42] 10Traffic, 10netops, 10DBA, 10Infrastructure-Foundations, and 2 others: Switch buffer re-partition - Eqiad Row A - https://phabricator.wikimedia.org/T286032 (10ops-monitoring-bot) Icinga downtime set by mmandere@cumin1001 for 1:00:00 4 host(s) and their services with reason: Eqiad row A maintenance ` cp[10... [14:45:29] 10Traffic, 10netops, 10DBA, 10Infrastructure-Foundations, and 2 others: Switch buffer re-partition - Eqiad Row A - https://phabricator.wikimedia.org/T286032 (10ops-monitoring-bot) Icinga downtime set by mmandere@cumin1001 for 1:00:00 1 host(s) and their services with reason: Eqiad row A maintenance ` dns10... [14:47:38] 10Traffic, 10netops, 10DBA, 10Infrastructure-Foundations, and 2 others: Switch buffer re-partition - Eqiad Row A - https://phabricator.wikimedia.org/T286032 (10Vgutierrez) [14:48:58] 10Traffic, 10netops, 10DBA, 10Infrastructure-Foundations, and 2 others: Switch buffer re-partition - Eqiad Row A - https://phabricator.wikimedia.org/T286032 (10ops-monitoring-bot) Icinga downtime set by mmandere@cumin1001 for 1:00:00 1 host(s) and their services with reason: Eqiad row A maintenance ` lvs10... [14:50:09] 10Traffic, 10netops, 10DBA, 10Infrastructure-Foundations, and 2 others: Switch buffer re-partition - Eqiad Row A - https://phabricator.wikimedia.org/T286032 (10Vgutierrez) [14:50:31] topranks: everything ready in our side, mmandere handled it flawlessly [14:51:01] glad to hear :) [14:51:02] thanks! [15:05:20] 10Traffic, 10netops, 10DBA, 10Infrastructure-Foundations, and 2 others: Switch buffer re-partition - Eqiad Row A - https://phabricator.wikimedia.org/T286032 (10Marostegui) [15:07:26] 10Traffic, 10netops, 10DBA, 10Infrastructure-Foundations, and 2 others: Switch buffer re-partition - Eqiad Row A - https://phabricator.wikimedia.org/T286032 (10Marostegui) >>! In T286032#7245427, @Marostegui wrote: > m2-master failed over from dbproxy1013 to dbproxy1015. Once the maintenance is done we nee... [15:27:43] 10netops, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: Adjust egress buffer allocations on ToR switches - https://phabricator.wikimedia.org/T284592 (10cmooney) [15:28:57] 10netops, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: Adjust egress buffer allocations on ToR switches - https://phabricator.wikimedia.org/T284592 (10cmooney) [15:37:56] Hello traffic folks! Could I get someone to fix up the puppet runs on diffscan.traffic.eqiad1.wikimedia.cloud? This is https://phabricator.wikimedia.org/T287612 [15:45:32] jbond: I464840540d2e7982f6ae910d65d857e2787f7771 do you have some context on this change? [15:45:45] apparently that's what's breaking diffscan on cloud [15:56:45] thx vgutierrez [16:04:20] vgutierrez: looking [16:08:33] vgutierrez: andrewbogott: for contex on the contacts profile best to see https://phabricator.wikimedia.org/T216088#7119005 (the whole task has relevent information but this comment disccusses this profile) [16:09:16] te profile::contacts dosn't work in cloud so the fix is simple https://gerrit.wikimedia.org/r/c/operations/puppet/+/708790 [16:11:56] and its working again [16:20:50] thanks jbond ! [16:21:12] np andrewbogott [16:22:34] thanks! [16:22:57] 10Traffic, 10SRE, 10cloud-services-team (Kanban): Puppet broken on diffscan.traffic.eqiad1.wikimedia.cloud - https://phabricator.wikimedia.org/T287612 (10Andrew) 05Open→03Resolved looks fixed! [20:02:57] (VarnishTrafficDrop) firing: 69% GET drop in text@ during the past 30 minutes - https://grafana.wikimedia.org/d/000000180/varnish-http-requests?viewPanel=6 - https://alerts.wikimedia.org [20:07:57] (VarnishTrafficDrop) resolved: 69% GET drop in text@ during the past 30 minutes - https://grafana.wikimedia.org/d/000000180/varnish-http-requests?viewPanel=6 - https://alerts.wikimedia.org [21:34:22] 10netops, 10DC-Ops, 10SRE, 10ops-codfw, 10Wikimedia-Incident: asw-a2-codfw unresponsive - https://phabricator.wikimedia.org/T286787 (10Papaul) Dear Juniper Networks Customer, Thank you for returning your defective product in relation to your recently created RMA. This notification confirms that Juniper...