[11:36:06] 10Puppet, 10Infrastructure-Foundations, 10Sustainability (Incident Followup): Fix the general problem of randomly-bad puppet agent cron timings within redundant clusters - https://phabricator.wikimedia.org/T161145 (10jbond) [11:51:46] 10Puppet, 10Cloud-Services, 10Data-Persistence, 10Infrastructure-Foundations, 10Thumbor: haproxy::site doesn't work as expected on the first puppet run - https://phabricator.wikimedia.org/T321684 (10jbond) >>! In T321684#8369618, @Vgutierrez wrote: >>>! In T321684#8369557, @jbond wrote: >>> cause we don'... [11:54:42] 10Puppet, 10Cloud-Services, 10Data-Persistence, 10Infrastructure-Foundations, 10Thumbor: haproxy::site doesn't work as expected on the first puppet run - https://phabricator.wikimedia.org/T321684 (10Vgutierrez) It's my understanding that right now the haproxy class is flexible enough for the current use... [12:40:07] 10Puppet, 10Cloud-Services, 10Data-Persistence, 10Infrastructure-Foundations, 10Thumbor: haproxy::site doesn't work as expected on the first puppet run - https://phabricator.wikimedia.org/T321684 (10fgiunchedi) Thank you for the investigation and the context -- appreciate it! >>! In T321684#8369634, @Vg... [13:27:19] 10SRE-tools, 10Discovery-Search, 10Infrastructure-Foundations, 10SRE, 10Spicerack: Create cookbook to reindex into elasticsearch / cirrus - https://phabricator.wikimedia.org/T219507 (10jbond) [13:30:55] 10Puppet, 10Infrastructure-Foundations, 10Wikidata, 10Wikidata-Query-Service, 10wdwb-tech: Migrate WDQS to profile::java - https://phabricator.wikimedia.org/T264181 (10jbond) [13:31:29] 10Puppet, 10Cloud-VPS, 10Infrastructure-Foundations, 10SRE, and 2 others: CPU scaling governor audit - https://phabricator.wikimedia.org/T225713 (10jbond) [13:32:35] 10Packaging, 10Infrastructure-Foundations, 10Wikidata, 10Wikidata-Query-Service, 10wdwb-tech: Migrate WDQS to Java 11 - https://phabricator.wikimedia.org/T316103 (10jbond) [13:36:13] 10Mail, 10Infrastructure-Foundations, 10MassMessage, 10WMF-JobQueue, 10Platform Team Workboards (Clinic Duty Team): Same MassMessage is being sent more than once - https://phabricator.wikimedia.org/T93049 (10jbond) [13:57:49] 10Packaging, 10Infrastructure-Foundations, 10SRE: Create an upload queue for reprepro - https://phabricator.wikimedia.org/T115349 (10jbond) [13:58:18] 10Packaging, 10Infrastructure-Foundations, 10SRE: Debian repository supporting multiple package versions - https://phabricator.wikimedia.org/T115758 (10jbond) [13:59:51] 10Puppet, 10Cloud-Services, 10Infrastructure-Foundations, 10cloud-services-team (Kanban): Untangle labs/production roles from labs/instance roles - https://phabricator.wikimedia.org/T119401 (10jbond) 05Open→03Resolved a:03jbond boldly closing but please re-open and update if its still actionable [14:07:03] 10Puppet, 10Cloud-Services, 10Data-Persistence, 10Infrastructure-Foundations, 10Thumbor: haproxy::site doesn't work as expected on the first puppet run - https://phabricator.wikimedia.org/T321684 (10Vgutierrez) >>! In T321684#8369754, @fgiunchedi wrote: > however if someone is introducing `haproxy::site`... [14:20:39] 10Puppet, 10netops, 10Cloud-Services, 10Infrastructure-Foundations, 10SRE: Moving network::external to hiera broke much of labs - https://phabricator.wikimedia.org/T141959 (10jbond) 05Open→03Resolved a:03jbond Im going to close this ticket assuming that the issues has been resolved in the mean time... [14:24:05] 10CAS-SSO, 10Infrastructure-Foundations, 10SRE: create notifications about user accounts that have not been used for a long time - https://phabricator.wikimedia.org/T146657 (10jbond) By used do you mean to accessing to some service via ldap auth/cas sso, ssh, both or something else? Further what would the... [14:26:13] 10Puppet, 10Infrastructure-Foundations, 10SRE: Puppet fails only once when restarting ferm is not successful - https://phabricator.wikimedia.org/T157972 (10jbond) 05Open→03Resolved a:03jbond [14:29:05] 10Packaging, 10Infrastructure-Foundations, 10SRE: make apt.wikimedia.org HA - https://phabricator.wikimedia.org/T158022 (10jbond) 05Open→03Resolved a:03jbond apt is [[ https://gerrit.wikimedia.org/r/c/operations/puppet/+/730523 | now configured ]] as a dns discover services in active/passive [14:35:27] 10Puppet, 10Infrastructure-Foundations, 10SRE: Fix UIDs for deployment server users - https://phabricator.wikimedia.org/T163667 (10jbond) Is this still an issue, we can now create damon users via admin.yaml with persistent uid's see reprepro as an example [14:59:12] 10Packaging, 10Infrastructure-Foundations, 10SRE, 10User-Joe: Sync internal nutcracker package with Debian package - https://phabricator.wikimedia.org/T166038 (10jbond) 05Open→03Resolved a:03jbond Closing, from what i can see we are now using the debian packages. please reopen if this is not the cas... [15:11:02] 10Puppet, 10Cloud-Services, 10Data-Persistence, 10Infrastructure-Foundations, 10Thumbor: haproxy::site doesn't work as expected on the first puppet run - https://phabricator.wikimedia.org/T321684 (10taavi) Looking at the logs, it seems like that haproxy.service is started when the package is installed, a... [15:13:16] 10Puppet, 10Infrastructure-Foundations, 10SRE, 10observability: run-no-puppet leave puppet disabled on kill/crash - https://phabricator.wikimedia.org/T182228 (10jbond) what exactly dose this script do? [15:28:16] 10Puppet, 10Infrastructure-Foundations, 10SRE: Usual git mechanism for aborting commit does not work on the private puppet repo - https://phabricator.wikimedia.org/T211121 (10jbond) Can you confirm is this is still an issue i just tried to recreate and it looks to work as expected now [15:32:41] 10Packaging, 10Infrastructure-Foundations, 10SRE: Add support for temporary chroots to boron - https://phabricator.wikimedia.org/T219977 (10jbond) @MoritzMuehlenhoff is this still worth exploring or has it perhaps been superseded ? [15:37:22] 10Puppet, 10Infrastructure-Foundations, 10SRE, 10observability: run-no-puppet leave puppet disabled on kill/crash - https://phabricator.wikimedia.org/T182228 (10jbond) [17:44:19] 10puppet-compiler, 10Infrastructure-Foundations, 10User-Joe: puppet-compiler should not link to not existing change.*.pson file - https://phabricator.wikimedia.org/T126796 (10jbond) i wonder if we should just remove the catalog files all together. They take up about 99% of the space for each pcc report and... [18:05:26] 10puppet-compiler, 10Infrastructure-Foundations, 10User-jbond, 10cloud-services-team (Kanban): puppet-catalog-compiler: new feature to report hiera interaction - https://phabricator.wikimedia.org/T215507 (10jbond) i did a quick and dirty run ro see what this might look like and here is the first pass ` $... [18:18:35] 10puppet-compiler, 10Infrastructure-Foundations, 10User-jbond, 10cloud-services-team (Kanban): puppet-catalog-compiler: new feature to report hiera interaction - https://phabricator.wikimedia.org/T215507 (10jbond) im also curious if you find the catlog files useful, i was thinking about proposing dropping... [18:34:57] 10puppet-compiler, 10Infrastructure-Foundations, 10User-jbond, 10cloud-services-team (Kanban): puppet-catalog-compiler: new feature to report hiera interaction - https://phabricator.wikimedia.org/T215507 (10jbond) sorted an uniqued, i think this is close to what you where wanting for the keys file? (of cou... [18:50:29] 10puppet-compiler, 10Infrastructure-Foundations: Create simple web endpoing for pcc - https://phabricator.wikimedia.org/T322437 (10jbond) p:05Triage→03Low [19:21:35] 10puppet-compiler, 10Infrastructure-Foundations: Create simple web endpoing for pcc - https://phabricator.wikimedia.org/T322437 (10jbond) it could be useful to expose the differ functionality as an api endpoint so we can submit two pson catalogues and produce a diff report. decoupling the report diffing and r... [19:40:49] 10puppet-compiler, 10Infrastructure-Foundations, 10User-Joe: puppet-compiler should not link to not existing change.*.pson file - https://phabricator.wikimedia.org/T126796 (10Dzahn) fwiw, I look at catalog files regularly. Imho the space issue just comes from the compiler running on * every once in a while.... [19:46:02] 10Mail, 10Infrastructure-Foundations, 10SRE: 2022-05-09 Exim BDAT Errors incident - https://phabricator.wikimedia.org/T309238 (10Krinkle) [21:20:11] 10puppet-compiler, 10Infrastructure-Foundations, 10Patch-For-Review: Create simple web endpoint for pcc - https://phabricator.wikimedia.org/T322437 (10Reedy)