[09:03:34] 10Puppet, 10Cloud-VPS, 10Infrastructure-Foundations, 10Patch-For-Review, and 3 others: Audit usages or the realm variable with a view to drop it - https://phabricator.wikimedia.org/T289661 (10dcaro) [09:04:46] 10Puppet, 10Cloud-VPS, 10Infrastructure-Foundations, 10User-dcaro, and 2 others: Add more rspec test to the puppet code - https://phabricator.wikimedia.org/T289668 (10dcaro) [09:37:11] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10SRE, and 2 others: Q1:(Need By: TBD) rack/setup/install cloudswift100[12] - https://phabricator.wikimedia.org/T289882 (10ayounsi) @aborrero is it possible to have more information on this new service? Design doc or similar. I can't find anything on Wikite... [09:40:29] 10netops, 10Infrastructure-Foundations: Rebuild Routinator (rpki) VMs with larger disk - https://phabricator.wikimedia.org/T292503 (10cmooney) [09:43:05] 10netops, 10Infrastructure-Foundations: Document our OOB - https://phabricator.wikimedia.org/T292504 (10ayounsi) p:05Triage→03Medium [09:52:02] 10netops, 10Infrastructure-Foundations, 10SRE: Rebuild Routinator (rpki) VMs with larger disk - https://phabricator.wikimedia.org/T292503 (10MoritzMuehlenhoff) https://packages.nlnetlabs.nl/ also provides the routinator debs for bullseye (plus it's a static Go binary anyway), so if we're recreating the VMs a... [09:54:55] 10netops, 10Infrastructure-Foundations, 10SRE: Rebuild Routinator (rpki) VMs with larger disk - https://phabricator.wikimedia.org/T292503 (10cmooney) @MoritzMuehlenhoff yes smart thinking we'll do that :) [10:07:25] 10netops, 10Infrastructure-Foundations, 10SRE: Rebuild Routinator (rpki) VMs with larger disk - https://phabricator.wikimedia.org/T292503 (10cmooney) @ayounsi Riccardo suggested maybe using a separate disk/partition for the routinator data? That was partly to just do a quick dirty job and not rebuild, but w... [10:10:21] 10netops, 10Infrastructure-Foundations, 10SRE: Rebuild Routinator (rpki) VMs with larger disk - https://phabricator.wikimedia.org/T292503 (10cmooney) p:05Triage→03Low [10:18:21] 10netops, 10Infrastructure-Foundations, 10SRE: Rebuild Routinator (rpki) VMs with larger disk - https://phabricator.wikimedia.org/T292503 (10MoritzMuehlenhoff) >>! In T292503#7401527, @cmooney wrote: > @ayounsi Riccardo suggested maybe using a separate disk/partition for the routinator data? That was partly... [10:28:00] 10Puppet, 10Cloud-VPS, 10Infrastructure-Foundations, 10Patch-For-Review, and 3 others: Refactor puppet:base module to reducs unneeded shared code paths - https://phabricator.wikimedia.org/T289661 (10jbond) [10:29:11] 10Puppet, 10Cloud-VPS, 10Infrastructure-Foundations, 10Patch-For-Review, and 3 others: Refactor puppet:base module to reducs unneeded shared code paths - https://phabricator.wikimedia.org/T289661 (10jbond) [10:33:49] 10Puppet, 10puppet-compiler, 10Cloud-VPS, 10Infrastructure-Foundations, and 2 others: Improve PCC support for cloud VPS environments - https://phabricator.wikimedia.org/T289666 (10jbond) [11:10:17] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10SRE, and 2 others: Q1:(Need By: TBD) rack/setup/install cloudswift100[12] - https://phabricator.wikimedia.org/T289882 (10aborrero) >>! In T289882#7401435, @ayounsi wrote: > @aborrero is it possible to have more information on this new service? Design doc... [12:23:18] 10netops, 10Infrastructure-Foundations, 10SRE, 10Documentation: Document our OOB - https://phabricator.wikimedia.org/T292504 (10Aklapper) [12:50:34] 10Puppet, 10Infrastructure-Foundations, 10User-jbond: Upgrade puppetboard to the latest version - https://phabricator.wikimedia.org/T292522 (10jbond) 05Open→03In progress a:03jbond [12:51:59] 10Packaging, 10Puppet, 10Infrastructure-Foundations, 10User-jbond: create puppetboard debian package - https://phabricator.wikimedia.org/T292523 (10jbond) [12:52:14] 10Puppet, 10Infrastructure-Foundations, 10User-jbond: Upgrade puppetboard to the latest version - https://phabricator.wikimedia.org/T292522 (10jbond) [12:52:53] 10Packaging, 10Puppet, 10Infrastructure-Foundations, 10User-jbond: create puppetboard debian package - https://phabricator.wikimedia.org/T292523 (10jbond) The repo linked above now has a version of puppet board which builds correctly and has been tested in a labs instance. [12:54:26] 10Packaging, 10Puppet, 10Infrastructure-Foundations, 10User-jbond: Update python3-pypuppetdb package to 2.4.0 - https://phabricator.wikimedia.org/T292525 (10jbond) [12:54:33] 10Packaging, 10Puppet, 10Infrastructure-Foundations, 10User-jbond: Update python3-pypuppetdb package to 2.4.0 - https://phabricator.wikimedia.org/T292525 (10jbond) p:05Triage→03Medium [12:58:05] 10Packaging, 10Puppet, 10Infrastructure-Foundations, 10User-jbond: Update python3-pypuppetdb package to 2.4.0 - https://phabricator.wikimedia.org/T292525 (10jbond) I have created an update package for 2.4.0 https://salsa.debian.org/python-team/packages/pypuppetdb/-/merge_requests/4 [13:01:23] re-post from this morning as I think was lost in a netsplit [13:01:30] moritzm, jbond: I noticed that mx1001 has the wrong ip6 fact, it uses the wiki-mail-eqiad.w.o address (final :91) instead of the host's one (final :76) [13:01:48] ack taking a look now [13:02:10] thanks! :) [13:09:12] 10netops, 10Infrastructure-Foundations, 10SRE, 10Documentation: Document our OOB - https://phabricator.wikimedia.org/T292504 (10ayounsi) 05Open→03Resolved https://wikitech.wikimedia.org/wiki/OOB [13:11:33] mx1001 (and mx2001) has some special-casing for the IP address in site.pp, it's probably time to drop this? [13:43:27] volans: fyi https://gerrit.wikimedia.org/r/c/operations/puppet/+/726625 [14:06:09] 10SRE-tools, 10Infrastructure-Foundations, 10SRE, 10serviceops-radar: SVC DNS zonefiles and source of truth - https://phabricator.wikimedia.org/T270071 (10ayounsi) [14:07:14] 10SRE-tools, 10Infrastructure-Foundations, 10SRE, 10serviceops-radar: SVC DNS zonefiles and source of truth - https://phabricator.wikimedia.org/T270071 (10ayounsi) I pinged a few persons on the task description to know if those records are needed (and document their usecase if so). [14:08:59] jbond: ack I'll look in a few, in a rabbit hole myself [14:16:15] get out Alice, GET OUT [14:19:37] lol [14:21:59] 10SRE-tools, 10Infrastructure-Foundations, 10SRE, 10serviceops-radar: SVC DNS zonefiles and source of truth - https://phabricator.wikimedia.org/T270071 (10akosiaris) Thanks for chasing that info down! So for ` CNAMEs: swift -> ms-fe kubestagemaster -> kubestagemaster1001 / kubestagemaster2001 staging ->... [14:26:10] 10SRE-tools, 10Infrastructure-Foundations, 10SRE, 10serviceops-radar: SVC DNS zonefiles and source of truth - https://phabricator.wikimedia.org/T270071 (10Volans) >>! In T270071#7402357, @akosiaris wrote: > As far as > > ` > SVC records that point to host IPs outside of the SVC subnets. Are those really... [14:27:14] jbond: if you have a sec by any chance: https://gerrit.wikimedia.org/r/c/operations/puppet/+/726634 [14:28:13] * jbond looking [14:39:31] 10Puppet, 10Infrastructure-Foundations, 10User-jbond: Upgrade puppetboard to the latest version - https://phabricator.wikimedia.org/T292522 (10jbond) [14:39:44] 10Puppet, 10Infrastructure-Foundations, 10User-jbond: Upgrade puppetboard to the latest version - https://phabricator.wikimedia.org/T292522 (10jbond) [14:45:07] 10SRE-tools, 10Infrastructure-Foundations, 10SRE, 10serviceops-radar: SVC DNS zonefiles and source of truth - https://phabricator.wikimedia.org/T270071 (10aborrero) > * SVC records that point to host IPs outside of the SVC subnets. Are those really required or just tech debt that should be fixed? > ** nfs... [14:55:15] jbond: and ofc that solution worked without working... :/ [14:55:36] now part of the systemctl status output is considered performance data by icinga :/ [14:55:53] I'll put together another fix [14:58:29] lol [14:58:57] at least I've unblocked luca to run his cookbook [14:59:21] I was also wondering if we should read the status.dat file in icinga-status ignoring all unicode errors... thoughts? [15:00:42] passing errors='replace' to read_text() for example [15:02:43] yes i think thats probably fine [15:03:30] all teh icinga structure is guaranteed to be ascii anyway [15:03:46] at least it doesn't break the whole script and hence any cookbook that uses get_status [15:03:50] directly or indirectly [15:25:26] 10SRE-tools, 10Infrastructure-Foundations, 10SRE, 10serviceops-radar: SVC DNS zonefiles and source of truth - https://phabricator.wikimedia.org/T270071 (10akosiaris) >>! In T270071#7402401, @Volans wrote: >>>! In T270071#7402357, @akosiaris wrote: >> As far as >> >> ` >> SVC records that point to host IP... [16:35:53] 10netops, 10Infrastructure-Foundations, 10SRE: Rebuild Routinator (rpki) VMs with larger disk - https://phabricator.wikimedia.org/T292503 (10MoritzMuehlenhoff) I've added routinator to apt.wikimedia.org at "thirdparty/routinator" for bullseye-wikimedia and adapted the Puppet code, so that when the these get... [16:36:27] 10Puppet, 10Infrastructure-Foundations, 10GitLab (Infrastructure), 10Patch-For-Review, and 3 others: Puppetise gitlab-ansible playbook - https://phabricator.wikimedia.org/T283076 (10Jelto) I'd like to test https://gerrit.wikimedia.org/r/c/operations/puppet/+/724430 tomorrow on `gitlab2001` (replica). For t... [16:53:05] motitzm: thanks for adding the upstream nlnetlabs repo makes upgrading the rpki hosts easier :)