[06:34:31] hello folks, morning [06:34:38] going to merge https://gerrit.wikimedia.org/r/c/operations/puppet/+/924507, no op for caching nodes [06:40:10] disabled puppet to verify on a couple of nodes that it is indeed a no-op [06:41:18] ok done, re-enabling puppet [06:41:39] the next steps, finally, would be to deploy the new vk cert on cp4037 [06:41:40] https://gerrit.wikimedia.org/r/c/operations/puppet/+/924509 [06:41:56] but I'd need to depool it to safely test it [06:42:15] lemme know when you have a moment to do it :) [07:52:29] 10Traffic, 10Commons, 10Infrastructure-Foundations, 10WMF-General-or-Unknown: Upload to Commons fails with a common ADSL connection in Taiwan - https://phabricator.wikimedia.org/T205619 (10ayounsi) [07:57:41] 10netops, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: Netflow/pmacct: use forwardingStatus - https://phabricator.wikimedia.org/T331707 (10ayounsi) @JAllemandou, I was wondering if you had a time-frame for this or need anything from SRE. [08:02:05] 10netops, 10Infrastructure-Foundations, 10SRE: Peering: prefer primary IXP for direcly connected networks - https://phabricator.wikimedia.org/T338201 (10cmooney) Thanks for this one. While not ideal I think probably option 2 / adding DIRECT_PEER_PRIMARY is gonna be best. Is getting a little complex, but at... [09:46:00] 10netops, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: Netflow/pmacct: use forwardingStatus - https://phabricator.wikimedia.org/T331707 (10JAllemandou) Thank you for reminding me, I had forgotten about this task @ayounsi. We can prioritize the work, the details we'll need are: - precise names... [10:38:51] https://usercontent.irccloud-cdn.com/file/EPIiTDWO/image.png [11:25:08] 10netops, 10Cloud-VPS, 10Infrastructure-Foundations, 10SRE, and 2 others: Move cloud vps ns-recursor IPs to host/row-independent addressing - https://phabricator.wikimedia.org/T307357 (10aborrero) I just had a conversation with @cmooney about this, with result being: * we will move the setup to 1x BGP VIP... [11:28:34] 10netops, 10Cloud-VPS, 10Infrastructure-Foundations, 10SRE, and 2 others: Move cloud vps ns-recursor IPs to host/row-independent addressing - https://phabricator.wikimedia.org/T307357 (10cmooney) @aborrero that makes sense. For the auth dns service we need to patch the dns repo to update the IPs for 'ns0'... [11:39:07] 10netops, 10Cloud-VPS, 10Infrastructure-Foundations, 10SRE, and 2 others: Move cloud vps ns-recursor IPs to host/row-independent addressing - https://phabricator.wikimedia.org/T307357 (10aborrero) Created https://netbox.wikimedia.org/ipam/ip-addresses/13309/ [12:40:53] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10SRE, 10ops-eqiad: Q2:(Need By: TBD) Rows E/F network racking task - https://phabricator.wikimedia.org/T292095 (10cmooney) 05Open→03Resolved All links have now been migrated. Massive thanks to @Jclark-ctr for all the work on site! [12:41:51] 10netops, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: Migrate row E/F network aggregation to dedicated Spine switches - https://phabricator.wikimedia.org/T322937 (10cmooney) All links have now been successfully migrated. All row E/F connectivity is now flowing via Spine switches ssw1-e1-eqiad... [12:42:15] 10netops, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: Migrate row E/F network aggregation to dedicated Spine switches - https://phabricator.wikimedia.org/T322937 (10cmooney) 05Open→03Resolved [12:45:34] 10netops, 10Infrastructure-Foundations, 10SRE, 10SRE-tools: Setup zero touch provisioning (ZTP) for network devices - https://phabricator.wikimedia.org/T336485 (10cmooney) @Papaul I've merged that changed and pushed to the mr routers, so hopefully if you try again the ZTP cookbook will work. [14:14:47] 10Traffic, 10DC-Ops, 10SRE, 10ops-codfw, 10Patch-For-Review: Q4:rack/setup/install lvs2011, lvs2012, lvs2013, lvs2014 - https://phabricator.wikimedia.org/T326767 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host lvs2014.codfw.wmnet with OS bullseye completed... [14:28:17] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10SRE, and 2 others: Q1:(Need By: TBD) rack/setup/install cloudswift100[12] - https://phabricator.wikimedia.org/T289882 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1001 for host cloudswift1002.eqiad.wmnet with OS bulls... [14:39:52] 10netops, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: Netflow/pmacct: use forwardingStatus - https://phabricator.wikimedia.org/T331707 (10ayounsi) Sounds good, thanks! > precise names of the fields in the data (we can look for this in realtime in the data when it starts flowing) Sure, is it sa... [15:11:08] 10Traffic, 10DC-Ops, 10SRE, 10ops-codfw, 10Patch-For-Review: Q4:rack/setup/install lvs2011, lvs2012, lvs2013, lvs2014 - https://phabricator.wikimedia.org/T326767 (10Papaul) [15:20:11] 10Traffic, 10DC-Ops, 10SRE, 10ops-codfw, 10Patch-For-Review: Q4:rack/setup/install lvs2011, lvs2012, lvs2013, lvs2014 - https://phabricator.wikimedia.org/T326767 (10Papaul) 05Open→03Resolved Complete [15:23:52] 10Traffic, 10DC-Ops, 10SRE, 10ops-codfw, 10Patch-For-Review: Q4:rack/setup/install lvs2011, lvs2012, lvs2013, lvs2014 - https://phabricator.wikimedia.org/T326767 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin2002 for host lvs2014.codfw.wmnet with OS bullseye [15:58:20] 10Traffic, 10DC-Ops, 10SRE, 10ops-codfw, 10Patch-For-Review: Q4:rack/setup/install lvs2011, lvs2012, lvs2013, lvs2014 - https://phabricator.wikimedia.org/T326767 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin2002 for host lvs2014.codfw.wmnet with OS bullseye executed w... [15:58:56] 10Traffic, 10DC-Ops, 10SRE, 10ops-codfw, 10Patch-For-Review: Q4:rack/setup/install lvs2011, lvs2012, lvs2013, lvs2014 - https://phabricator.wikimedia.org/T326767 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin2002 for host lvs2014.codfw.wmnet with OS bullseye [16:06:30] 10Traffic, 10DC-Ops, 10SRE, 10ops-codfw, 10Patch-For-Review: Q4:rack/setup/install lvs2011, lvs2012, lvs2013, lvs2014 - https://phabricator.wikimedia.org/T326767 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin2002 for host lvs2014.codfw.wmnet with OS bullseye executed w... [16:20:37] 10netops, 10Data-Engineering, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: Netflow/pmacct: use forwardingStatus - https://phabricator.wikimedia.org/T331707 (10JArguello-WMF) [16:45:48] 10Traffic: Confusing error message when making plaintext HTTP POST requests to Wikimedia sites - https://phabricator.wikimedia.org/T338481 (10taavi) [16:46:08] 10Traffic: Let HAProxy handle port 80 - https://phabricator.wikimedia.org/T323557 (10taavi) {T338481} seems to have been caused by this change. [17:30:28] 10netops, 10Data-Engineering, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: Netflow/pmacct: use forwardingStatus - https://phabricator.wikimedia.org/T331707 (10JAllemandou) >> precise names of the fields in the data (we can look for this in realtime in the data when it starts flowing) > Sure,...