[01:23:42] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: codfw: Relocate servers to make space for new switches in rowA and rowB - https://phabricator.wikimedia.org/T326564 (10Papaul) [01:25:53] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: codfw: Relocate servers to make space for new switches in rowA and rowB - https://phabricator.wikimedia.org/T326564 (10Papaul) [01:32:56] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: codfw: Relocate servers to make space for new switches in rowA and rowB - https://phabricator.wikimedia.org/T326564 (10Papaul) [07:39:46] 10netops, 10Infrastructure-Foundations, 10SRE, 10conftool, and 2 others: Scap deploy failed to depool codfw servers - https://phabricator.wikimedia.org/T327041 (10Joe) 05Open→03Resolved This is now fully resolved. [08:51:48] 10netops, 10Infrastructure-Foundations, 10SRE, 10fundraising-tech-ops: Upgrade fasw to Junos 21 - https://phabricator.wikimedia.org/T316542 (10ayounsi) Some notes from {T316532} Make sure console access works. Before the upgrade, remove this configuration stanza, otherwise the `request system software add... [09:38:42] 10netops, 10Infrastructure-Foundations, 10SRE, 10Sustainability (Incident Followup): Cr1-eqiad comms problem when moving to 40G row D handoff - https://phabricator.wikimedia.org/T320566 (10ayounsi) Seeing what happened with codfw row B, it's safe to assume that only a reboot of the faulty switch member wil... [10:35:34] 10netops, 10Infrastructure-Foundations: eqiad/codfw virtual-chassis upgrades - https://phabricator.wikimedia.org/T327248 (10ayounsi) [16:16:51] 10SRE-tools, 10Infrastructure-Foundations: Cookbook for rack downtime - https://phabricator.wikimedia.org/T327300 (10ayounsi) [18:27:17] 10SRE-tools, 10Infrastructure-Foundations, 10cloud-services-team: Spicerack: Add CI step to test with wmcs cookbooks - https://phabricator.wikimedia.org/T325758 (10fnegri) [18:27:43] 10SRE-tools, 10Infrastructure-Foundations, 10cloud-services-team: Allow wmcs cookbooks running on cloudcuminXXXX to write to the SAL - https://phabricator.wikimedia.org/T325756 (10fnegri) [18:27:51] 10SRE-tools, 10Infrastructure-Foundations, 10cloud-services-team: Update Spicerack documentation - https://phabricator.wikimedia.org/T325754 (10fnegri) [18:28:38] 10SRE-tools, 10Cloud-Services, 10Infrastructure-Foundations, 10cloud-services-team, 10Patch-For-Review: Cumin/Openstack: multi-project commands are extremely slow - https://phabricator.wikimedia.org/T325773 (10fnegri) [18:30:24] 10SRE-tools, 10Infrastructure-Foundations, 10cloud-services-team: Decide sudoers rules for users without global root - https://phabricator.wikimedia.org/T325067 (10fnegri) [18:35:58] 10Puppet, 10Data-Services, 10Infrastructure-Foundations, 10SRE, 10cloud-services-team: clouddumps1002: ferm is being started on every puppet run - https://phabricator.wikimedia.org/T323324 (10fnegri) [18:40:42] 10SRE-tools, 10Infrastructure-Foundations, 10cloud-services-team (FY2022/2023-Q3): WMCS Cookbook Automation FY2022-23 Q2 tracking task - https://phabricator.wikimedia.org/T319401 (10fnegri) [18:41:36] 10netops, 10Infrastructure-Foundations, 10SRE, 10cloud-services-team: neutron: cloudnet nodes use VRRP over VXLAN to instrument HA and they require to be on the same subnet - https://phabricator.wikimedia.org/T319539 (10fnegri) [18:45:18] 10Puppet, 10Cloud-VPS, 10Infrastructure-Foundations, 10cloud-services-team: Remove prod-specific bits from cloud puppetmasters - https://phabricator.wikimedia.org/T309281 (10fnegri) [18:46:36] 10netops, 10Cloud-VPS, 10Infrastructure-Foundations, 10SRE, 10cloud-services-team: Move cloud vps ns-recursor IPs to host/row-independent addressing - https://phabricator.wikimedia.org/T307357 (10fnegri) [18:51:19] 10netbox, 10DNS, 10Infrastructure-Foundations, 10SRE, 10cloud-services-team: Move some of wikimediacloud.org 185.15.56.0/23 to Netbox - https://phabricator.wikimedia.org/T268621 (10fnegri) [18:51:33] 10netops, 10Infrastructure-Foundations, 10SRE, 10cloud-services-team: Join ARIN waiting list to request additional IPv4 resources. - https://phabricator.wikimedia.org/T288342 (10fnegri) [18:54:06] 10Puppet, 10Cloud-VPS, 10Infrastructure-Foundations, 10SRE, and 2 others: CPU scaling governor audit - https://phabricator.wikimedia.org/T225713 (10fnegri) [18:59:54] 10netops, 10Cloud-VPS, 10Infrastructure-Foundations, 10cloud-services-team, 10Epic: CloudVPS: network architecture - https://phabricator.wikimedia.org/T209460 (10fnegri) [19:02:21] 10Puppet, 10Cloud-VPS, 10Infrastructure-Foundations, 10cloud-services-team, 10User-jbond: Normalise hiera default values - https://phabricator.wikimedia.org/T289665 (10fnegri) [19:02:37] 10Puppet, 10Cloud-VPS, 10Infrastructure-Foundations, 10cloud-services-team, and 2 others: Add more rspec test to the puppet code - https://phabricator.wikimedia.org/T289668 (10fnegri) [19:02:57] 10Puppet, 10Cloud-VPS, 10Infrastructure-Foundations, 10cloud-services-team, 10User-jbond: Audit puppet usage in cloud hosts - https://phabricator.wikimedia.org/T289658 (10fnegri) [19:06:28] 10Puppet, 10Cloud Services Proposals, 10Cloud-VPS, 10Infrastructure-Foundations, and 3 others: Easing pain points caused by divergence between cloudservices and production puppet usecases - https://phabricator.wikimedia.org/T285539 (10fnegri) [19:11:17] 10SRE-tools, 10Infrastructure-Foundations, 10Spicerack, 10cloud-services-team: wmcs.spicerack: Setup a host to run cookbooks from prod network - https://phabricator.wikimedia.org/T276440 (10fnegri) [19:14:15] 10Puppet, 10Infrastructure-Foundations, 10SRE, 10cloud-services-team: Puppet class systemd needs to throw a more useful error - https://phabricator.wikimedia.org/T195553 (10fnegri) [19:15:10] 10Puppet, 10Infrastructure-Foundations, 10cloud-services-team: ops/puppet: generalize systemd resource control for users - https://phabricator.wikimedia.org/T215401 (10fnegri) [19:17:39] 10SRE-tools, 10Infrastructure-Foundations, 10cloud-services-team, 10IPv6: Some WMCS clusters apparently do not support IPv6 - https://phabricator.wikimedia.org/T271139 (10fnegri) [19:28:56] 10netops, 10Infrastructure-Foundations, 10SRE, 10cloud-services-team: ceph: test and decide 1 network interface setup - https://phabricator.wikimedia.org/T325531 (10fnegri) [19:29:54] 10netops, 10Infrastructure-Foundations, 10SRE, 10cloud-services-team, 10Patch-For-Review: Move WMCS servers to 1 NIC - https://phabricator.wikimedia.org/T319184 (10fnegri) [19:33:09] 10netbox, 10netops, 10DNS, 10Infrastructure-Foundations, and 2 others: Cloud: define relationship between wikimediacloud.org domain, CIDR prefixes and netbox automation - https://phabricator.wikimedia.org/T266331 (10fnegri) [19:34:15] 10netops, 10Infrastructure-Foundations, 10SRE, 10cloud-services-team: CloudVPS: IPv6 early PoC - https://phabricator.wikimedia.org/T245495 (10fnegri) [19:36:12] 10SRE-tools, 10Infrastructure-Foundations, 10cloud-services-team: spicerack: introduce GridEngine controller - https://phabricator.wikimedia.org/T300032 (10fnegri) [19:36:55] 10Puppet, 10Infrastructure-Foundations, 10cloud-services-team: Reduce the effects of puppet breakage on VPS - https://phabricator.wikimedia.org/T226270 (10fnegri) [19:37:01] 10Puppet, 10Infrastructure-Foundations, 10cloud-services-team, 10User-jbond: Prevent catalog breakage on cloud instances by decoupling core cloud puppetmaster from custom puppetmasters - https://phabricator.wikimedia.org/T227029 (10fnegri) [19:38:04] 10Puppet, 10Cloud-Services, 10Infrastructure-Foundations, 10SRE, and 2 others: Create a cron to clean clientbucket every day or hour - https://phabricator.wikimedia.org/T165885 (10fnegri) [19:42:16] 10netops, 10Cloud-VPS, 10Infrastructure-Foundations, 10SRE, 10cloud-services-team: Upgrade cloudsw1-c8-eqiad and cloudsw1-d5-eqiad to Junos 20+ - https://phabricator.wikimedia.org/T316544 (10fnegri) [19:44:44] 10netops, 10Infrastructure-Foundations, 10SRE, 10IPv6, and 2 others: Fix IPv6 autoconf issues once and for all, across the fleet. - https://phabricator.wikimedia.org/T102099 (10BBlack) Bump - these issues continue to affect us sometimes. There seem to be some cases where Juniper can mis-route an RA to an... [19:44:54] 10netops, 10Data-Services, 10Infrastructure-Foundations, 10Wikidata, and 5 others: Do not rate limit dumps from internal network - https://phabricator.wikimedia.org/T222349 (10fnegri) [19:47:58] 10netops, 10Infrastructure-Foundations, 10SRE, 10cloud-services-team: CloudVPS: enable BGP in the neutron transport network - https://phabricator.wikimedia.org/T245606 (10fnegri) [19:53:15] 10Puppet, 10Cloud-Services, 10Infrastructure-Foundations, 10cloud-services-team: Consider ways to make puppetmaster CA changes smoother on the puppet client end - https://phabricator.wikimedia.org/T220268 (10fnegri) [19:58:37] 10netops, 10Infrastructure-Foundations, 10SRE, 10IPv6, and 2 others: Fix IPv6 autoconf issues once and for all, across the fleet. - https://phabricator.wikimedia.org/T102099 (10BBlack) I fixed all these cases noted above for now. Note that in the lvs1017 case, this could've potentially caused a public ser... [21:44:53] 10Puppet, 10Cloud-VPS, 10Infrastructure-Foundations, 10cloud-services-team, 10User-dcaro: Investigate use of Puppet "environments" for per-project Puppet manifests - https://phabricator.wikimedia.org/T170370 (10fnegri) [21:49:11] 10Mail, 10Cloud-VPS, 10Infrastructure-Foundations, 10cloud-services-team: Set up SPF, DKIM, etc. for new cloud MX servers - https://phabricator.wikimedia.org/T208281 (10fnegri) [21:50:25] 10SRE-tools, 10Infrastructure-Foundations, 10cloud-services-team: Cumin: create external backend for WMCS Puppet API - https://phabricator.wikimedia.org/T179816 (10fnegri) [21:52:56] 10puppet-compiler, 10Infrastructure-Foundations, 10cloud-services-team, 10User-jbond: puppet-catalog-compiler: new feature to report hiera interaction - https://phabricator.wikimedia.org/T215507 (10fnegri)