[00:14:46] (SystemdUnitFailed) firing: (3) uwsgi-puppetboard.service Failed on puppetboard1003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [04:14:46] (SystemdUnitFailed) firing: (3) uwsgi-puppetboard.service Failed on puppetboard1003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [08:14:46] (SystemdUnitFailed) firing: (3) uwsgi-puppetboard.service Failed on puppetboard1003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [08:56:04] Oh no... mediawiki uses http status 200 when telling me that I'm rate limited :-( [10:03:38] Good morning. I am trying to add a new external repository to reprepro: https://wikitech.wikimedia.org/wiki/Reprepro#Updating_external_repositories [10:04:34] I've added the config here: https://gerrit.wikimedia.org/r/c/operations/puppet/+/935071 and followed up with the GPG key here: https://gerrit.wikimedia.org/r/c/operations/puppet/+/935381 [10:05:37] But when I run `reprepro --verbose --component thirdparty/confluent7 update bullseye-wikimedia` reprepro complains of `Nothing to do found` - Have I missed a step, or done something wrong? [10:06:49] btullis: yes, you also need to add the name of the `updates` block to the `Update:` key of each repository in `distributions-wikimedia` [10:19:27] taavi: Thanks. I will check that now. [10:23:24] 10Puppet, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review, 10User-jbond: Shell/Python/other scripts should not be generated by ERB files; dynamic parts should be a simple ERB config file - https://phabricator.wikimedia.org/T254480 (10SLyngshede-WMF) [10:43:28] I think I made it worse. [10:43:32] https://www.irccloud.com/pastebin/DbvCA30A/ [10:43:47] https://gerrit.wikimedia.org/r/c/operations/puppet/+/935391 [10:45:39] Is it because it's not correctly ordered? [10:45:44] https://www.irccloud.com/pastebin/0LFSCmU4/ [10:51:02] btullis: un Updates: you need the name of the updates block, not the name of the component [10:51:07] but yes, you should keep it sorted [10:53:35] Oh, I see. Thanks. That's pretty confusing. Is there any common practice for when and when not to use the prefix `thirdparty/`in the updates block name? [10:55:07] not sure [10:55:31] and sometimes i've wondered whether it'd make sense to generate the reprepro deb822 files from hiera where we could add proper validation etc [10:58:01] 10CFSSL-PKI, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review, 10Puppet (Puppet 7.0): PKI: add the new puppet CA to the pki infrastructre - https://phabricator.wikimedia.org/T340557 (10jbond) > I will update the pki orchestration so that we create a cert file with made up of > ` > $ cat $(facte... [10:58:12] 10CFSSL-PKI, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review, 10Puppet (Puppet 7.0): PKI: add the new puppet CA to the pki infrastructre - https://phabricator.wikimedia.org/T340557 (10jbond) 05Open→03Resolved [11:19:46] (SystemdUnitFailed) firing: (4) uwsgi-puppetboard.service Failed on puppetboard1003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [13:03:56] 10SRE-tools, 10Infrastructure-Foundations, 10Puppet-Core, 10SRE, 10Puppet (Puppet 7.0): Creat cookbook to migrate serveres from the puppetmasters to puppetservers - https://phabricator.wikimedia.org/T340739 (10jbond) p:05Triage→03Medium [13:19:51] 10Mail, 10Cloud-VPS, 10Infrastructure-Foundations, 10cloud-services-team: Set up SPF, DKIM, etc. for new cloud MX servers - https://phabricator.wikimedia.org/T208281 (10taavi) 05Open→03Resolved a:03taavi I'm tentatively closing this. The mx-out servers now DKIM sign outgoing e-mails. In addition, the... [13:29:56] 10SRE-tools, 10Infrastructure-Foundations, 10Puppet-Core, 10SRE, and 2 others: Creat cookbook to migrate serveres from the puppetmasters to puppetservers - https://phabricator.wikimedia.org/T340739 (10jbond) Theses are the manual steps i made to migrate puppetboard1003 * Agent: [[ https://gerrit.wikimedi... [13:41:24] 10CFSSL-PKI, 10Infrastructure-Foundations, 10SRE, 10Puppet (Puppet 7.0): Create dynamic CRL - https://phabricator.wikimedia.org/T340543 (10jbond) I have added config to set the agent to `certificate_revocation = leaf` however we will need to move this config to the main section as we also need to to do a `... [13:52:23] 10CFSSL-PKI, 10Infrastructure-Foundations, 10SRE, 10Puppet (Puppet 7.0): puptet7: drop iunstances of :undef in erb files - https://phabricator.wikimedia.org/T341071 (10jbond) [13:52:59] 10CFSSL-PKI, 10Infrastructure-Foundations, 10SRE, 10Puppet (Puppet 7.0): puptet7: drop iunstances of :undef in erb files - https://phabricator.wikimedia.org/T341071 (10jbond) [13:53:02] 10CFSSL-PKI, 10Infrastructure-Foundations, 10SRE, 10Puppet (Puppet 7.0): Create dynamic CRL - https://phabricator.wikimedia.org/T340543 (10jbond) [14:00:37] 10SRE-tools, 10Infrastructure-Foundations, 10Puppet-Core, 10SRE, and 2 others: Creat cookbook to migrate serveres from the puppetmasters to puppetservers - https://phabricator.wikimedia.org/T340739 (10Volans) >>! In T340739#8988130, @jbond wrote: > Theses are the manual steps i made to migrate puppetboard1... [14:08:10] 10CFSSL-PKI, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review, 10Puppet (Puppet 7.0): puptet7: drop instances of :undef in erb files - https://phabricator.wikimedia.org/T341071 (10Aklapper) [14:08:59] 10SRE-tools, 10Infrastructure-Foundations, 10Puppet-Core, 10SRE, and 2 others: Create cookbook to migrate servers from the puppetmasters to puppetservers - https://phabricator.wikimedia.org/T340739 (10Aklapper) [14:28:03] 10CFSSL-PKI, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review, 10Puppet (Puppet 7.0): puppet7: drop instances of :undef in erb files - https://phabricator.wikimedia.org/T341071 (10Aklapper) [14:44:46] (SystemdUnitFailed) firing: (4) uwsgi-puppetboard.service Failed on puppetboard1003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [15:19:49] 10SRE-tools, 10Infrastructure-Foundations, 10Patch-For-Review, 10cloud-services-team (FY2022/2023-Q4): Allow wmcs cookbooks running on cloudcuminXXXX to write to the SAL - https://phabricator.wikimedia.org/T325756 (10fnegri) As you can see from the messages above, logging is working correctly from cloudcum... [15:22:01] 10CFSSL-PKI, 10Infrastructure-Foundations, 10SRE, 10Puppet (Puppet 7.0): puppet7: drop instances of :undef in erb files - https://phabricator.wikimedia.org/T341071 (10jbond) p:05Triage→03Medium [17:14:46] (SystemdUnitFailed) firing: (4) httpbb_hourly_appserver.service Failed on cumin2002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [18:14:46] (SystemdUnitFailed) firing: (4) httpbb_hourly_appserver.service Failed on cumin2002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [18:44:46] (SystemdUnitFailed) firing: (4) wmf_auto_restart_uwsgi-puppetboard.service Failed on puppetboard1003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [19:14:46] (SystemdUnitFailed) firing: (3) wmf_auto_restart_uwsgi-puppetboard.service Failed on puppetboard1003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [23:14:46] (SystemdUnitFailed) firing: (2) wmf_auto_restart_uwsgi-puppetboard.service Failed on puppetboard1003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed