[06:42:25] 10Domains, 10Traffic, 10SRE: Mark Monitor administration panel (redirects for wikimedia.pl) - https://phabricator.wikimedia.org/T333827 (10Jacek_Broda_WMPL) Hi! Thanks very much, Dzahn! I will try to contact Legal team and ask them about it. Hope that will work. Best for you! [08:11:33] hello folks good morning :) [08:12:18] any merciful soul that could validate quickly https://gerrit.wikimedia.org/r/c/operations/puppet/+/920218 ? If everything looks right I'll also proceed with the lvs setup + pybal restarts (this time I'd need both eqiad and codfw) [08:13:20] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10SRE, and 2 others: Q1:(Need By: TBD) rack/setup/install cloudswift100[12] - https://phabricator.wikimedia.org/T289882 (10ayounsi) https://netbox.wikimedia.org/extras/reports/puppetdb.PhysicalHosts/ was alerting with: > cloudswift1001 (WMF5069) Device is i... [08:36:56] 10netops, 10Infrastructure-Foundations, 10SRE: Update network SSH keys to ssh-ed25519 - https://phabricator.wikimedia.org/T336769 (10ayounsi) [09:23:00] vgutierrez: o/ - sorry for the extra ping, just wanted to know if it is ok to roll restart some pybals :) [09:23:27] sure, go ahead [09:23:39] the guys on-call will handle any potential issue [09:23:44] }:) [09:26:26] ack :) [09:44:44] vgutierrez: since I am stupid, I'd need to fix the lvs config with https://gerrit.wikimedia.org/r/c/operations/puppet/+/920649 [09:45:01] I disabled puppet on all lvs affected, except lvs2010 that has the new config [09:45:08] I realized that it was wrong checking ipvs adm [09:45:21] sorry, just wanted to confirm that I can proceed with the fix [09:46:24] the diff looks good to me [09:50:20] elukey: :) [09:50:39] vgutierrez: I know I know sorry, all good now [09:50:51] so easy to miss a tiny bit [09:51:07] ok so lvs2010 looks good, will do other final checks and then proceed with 2009 [09:52:25] nice [09:52:46] looking good from my PoV as well [09:52:58] thanks for double checking <3 [09:55:37] 2009 done as well, all good [10:04:52] all done in eqiad! [11:12:25] 10Traffic, 10envoy, 10serviceops, 10Patch-For-Review: Upgrade Envoy to supported version - https://phabricator.wikimedia.org/T300324 (10JMeybohm) [11:13:08] 10Traffic, 10envoy, 10serviceops, 10Patch-For-Review: Refactor envoy access_log_path to access loggers - https://phabricator.wikimedia.org/T303231 (10JMeybohm) 05Open→03Resolved Done in deployment-charts and puppet [11:39:55] 10netops, 10Cloud-VPS, 10Infrastructure-Foundations, 10SRE, and 3 others: Upgrade cloudsw1-c8-eqiad and cloudsw1-d5-eqiad to Junos 20+ - https://phabricator.wikimedia.org/T316544 (10aborrero) [12:54:39] 10netops, 10Infrastructure-Foundations, 10SRE, 10cloud-services-team: Configure cloudsw1-b1-codfw and migrate cloud hosts in codfw B1 to it - https://phabricator.wikimedia.org/T327919 (10aborrero) [12:54:49] 10netops, 10Cloud-VPS, 10Infrastructure-Foundations, 10SRE, and 2 others: Move cloud vps ns-recursor IPs to host/row-independent addressing - https://phabricator.wikimedia.org/T307357 (10aborrero) [12:55:09] 10netops, 10Cloud-VPS, 10Infrastructure-Foundations, 10SRE, and 3 others: cloudservices[2004/2005]-dev & cloudweb2002-dev: connect them to cloudsw so they can have cloud-private vlan - https://phabricator.wikimedia.org/T336587 (10aborrero) 05Open→03Resolved thanks! [12:58:05] 10netops, 10Cloud-VPS, 10Infrastructure-Foundations, 10SRE, and 2 others: Move cloud vps ns-recursor IPs to host/row-independent addressing - https://phabricator.wikimedia.org/T307357 (10aborrero) >>! In T307357#8847884, @aborrero wrote: > > * hook the cloudservices boxes to the cloud-private vlan -- requ... [13:13:40] 10Traffic, 10SRE, 10envoy, 10serviceops, 10Patch-For-Review: Refactor envoy HTTP protocol options to new version - https://phabricator.wikimedia.org/T303230 (10JMeybohm) Done in deployment-charts and puppet [13:13:45] 10Traffic, 10envoy, 10serviceops, 10Patch-For-Review: Upgrade Envoy to supported version - https://phabricator.wikimedia.org/T300324 (10JMeybohm) [13:13:57] 10Traffic, 10SRE, 10envoy, 10serviceops, 10Patch-For-Review: Refactor envoy HTTP protocol options to new version - https://phabricator.wikimedia.org/T303230 (10JMeybohm) 05In progress→03Resolved [13:20:44] 10netops, 10Cloud-VPS, 10Infrastructure-Foundations, 10SRE, and 2 others: Move cloud vps ns-recursor IPs to host/row-independent addressing - https://phabricator.wikimedia.org/T307357 (10cmooney) >>! In T307357#8859005, @aborrero wrote: > With the remaining bits being: > * allocate a public IPv4 for the BG... [13:26:49] 10netops, 10Infrastructure-Foundations, 10SRE, 10cloud-services-team: Configure cloudsw1-b1-codfw and migrate cloud hosts in codfw B1 to it - https://phabricator.wikimedia.org/T327919 (10cmooney) [13:26:57] 10netops, 10Cloud-VPS, 10Infrastructure-Foundations, 10SRE, and 2 others: Move cloud vps ns-recursor IPs to host/row-independent addressing - https://phabricator.wikimedia.org/T307357 (10cmooney) [13:28:26] 10netops, 10Cloud-VPS, 10Infrastructure-Foundations, 10SRE, and 3 others: cloudservices[2004/2005]-dev & cloudweb2002-dev: connect them to cloudsw so they can have cloud-private vlan - https://phabricator.wikimedia.org/T336587 (10cmooney) 05Resolved→03Open @aborrero quick question on how best to set th... [13:29:11] 10netops, 10Cloud-VPS, 10Infrastructure-Foundations, 10SRE, and 3 others: cloudservices[2004/2005]-dev & cloudweb2002-dev: connect them to cloudsw so they can have cloud-private vlan - https://phabricator.wikimedia.org/T336587 (10cmooney) a:05Papaul→03cmooney [14:25:51] 10Traffic, 10SRE, 10ops-codfw, 10Patch-For-Review: Q4:rack/decom codfw unified decommission task - https://phabricator.wikimedia.org/T335777 (10Jhancock.wm) @ssingh is it safe for me to physically remove lvs2008 and the three dns servers from the racks and offline them? [14:26:59] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Codfw:row A/B: rack/cable new switches - https://phabricator.wikimedia.org/T332180 (10Jhancock.wm) [14:29:42] 10netops, 10Cloud-VPS, 10Infrastructure-Foundations, 10SRE, and 3 others: cloudservices[2004/2005]-dev & cloudweb2002-dev: connect them to cloudsw so they can have cloud-private vlan - https://phabricator.wikimedia.org/T336587 (10aborrero) >>! In T336587#8859125, @cmooney wrote: > @aborrero quick question... [14:39:30] 10netops, 10Cloud-VPS, 10Infrastructure-Foundations, 10SRE, and 3 others: cloudservices[2004/2005]-dev & cloudweb2002-dev: connect them to cloudsw so they can have cloud-private vlan - https://phabricator.wikimedia.org/T336587 (10cmooney) @aborrero in general that makes sense yes. > if we are annoyed by u... [14:50:05] 10Traffic, 10SRE, 10ops-codfw, 10Patch-For-Review: Q4:rack/decom codfw unified decommission task - https://phabricator.wikimedia.org/T335777 (10ssingh) >>! In T335777#8859472, @Jhancock.wm wrote: > @ssingh is it safe for me to physically remove lvs2008 and the three dns servers from the racks and offline t... [15:49:25] 10Traffic, 10SRE, 10ops-codfw, 10Patch-For-Review: Q4:rack/decom codfw unified decommission task - https://phabricator.wikimedia.org/T335777 (10Jhancock.wm) [17:25:12] 10netops, 10Cloud-VPS, 10Infrastructure-Foundations, 10SRE, and 3 others: cloudservices[2004/2005]-dev & cloudweb2002-dev: connect them to cloudsw so they can have cloud-private vlan - https://phabricator.wikimedia.org/T336587 (10cmooney) @aborrero set up now, let me know if this looks correct: [cloudserv... [18:25:39] 10Traffic, 10SRE, 10Wikidata, 10wdwb-tech, 10wmde-wikidata-tech: Wikidata seems to still be utilizing insecure HTTP URIs - https://phabricator.wikimedia.org/T331356 (10Ladsgroup) >>! In T331356#8853449, @OlafJanssen wrote: > The sidebar is just one place the concept URI appears, how about all the concept... [21:59:55] 10Traffic, 10Patch-For-Review: Write a cookbook to handle restarts of Wikimedia DNS - https://phabricator.wikimedia.org/T335533 (10BCornwall) I don't agree with abstracting the After=/BindsTo= and would rather just hard-code it. But since bird did it that way... [22:14:17] 10netops, 10Cloud-VPS, 10Infrastructure-Foundations, 10SRE, 10cloud-services-team: Move cloud vps ns-recursor IPs to host/row-independent addressing - https://phabricator.wikimedia.org/T307357 (10cmooney) @aborrero based on our conversation on irc we can use this range for private VIPs in codfw: https:/...