[03:24:00] 10Puppet, 10SRE: Upgrade Puppet to 5.5.21 - https://phabricator.wikimedia.org/T248168 (10MacFan4000) I will note that Puppet 5.5 has been declared end of life as of November 2020 [05:33:58] 10SRE-tools, 10netops, 10DC-Ops, 10SRE, 10Patch-For-Review: Allow idrac tftp fetching of firmware updates (either to existing tftp or new solution) - https://phabricator.wikimedia.org/T283771 (10Papaul) With the same lab environment, the same command will upgrade the BIOS just by changing the file. ` sud... [08:42:46] 10SRE-tools, 10netops, 10DC-Ops, 10SRE, 10Patch-For-Review: Allow idrac tftp fetching of firmware updates (either to existing tftp or new solution) - https://phabricator.wikimedia.org/T283771 (10jbond) > could be run from cumin servers which already have access to the mgmt network so that seems a good o... [08:55:25] 10Mail, 10Znuny, 10User-jbond: Clean up OTRS/Znuny addresses handles by gsuite - https://phabricator.wikimedia.org/T284145 (10jbond) @DC thanks for the update, do you know what database attribute i could use to exclude deactivated emails from my query. im curently using the following SQL statment to get all... [09:22:23] 10netops, 10Continuous-Integration-Infrastructure, 10DC-Ops, 10serviceops: Flapping codfw management alarm ( contint2001.mgmt/SSH is CRITICAL ) - https://phabricator.wikimedia.org/T283582 (10hashar) Looking at the alarm history since midnight UTC at https://icinga.wikimedia.org/cgi-bin/icinga/history.cgi?h... [10:04:25] 10netbox: Request to add Custom fields in Netbox - https://phabricator.wikimedia.org/T284614 (10cmooney) > It's conceivable we may have configs span across multiple years, or even multiple configs within the same year -- so something incremental like "Config A (Gen1)" or "Config A01" may be a better idea? I thi... [10:06:03] 10Puppet, 10SRE-tools, 10SRE, 10Patch-For-Review, 10User-jbond: Private puppet commit hook checks current state of folder, not what is staged - https://phabricator.wikimedia.org/T278187 (10jbond) 05Open→03Resolved a:03jbond I have deployed a fix to this, closing but please re-open if you still see... [10:13:27] 10Puppet, 10SRE-tools, 10SRE, 10User-jbond: Private puppet commit hook checks current state of folder, not what is staged - https://phabricator.wikimedia.org/T278187 (10jbond) 05Resolved→03Open reopening, the new hook prevents file deletions [10:59:55] 10Puppet, 10SRE-tools, 10SRE, 10User-jbond: Private puppet commit hook checks current state of folder, not what is staged - https://phabricator.wikimedia.org/T278187 (10jbond) 05Open→03Resolved Have updated the hook to only work on added or modified files [11:36:08] 10netops, 10SRE, 10Traffic: BGP Policy on aggregate routes prevents them being created in some circumstances. - https://phabricator.wikimedia.org/T283163 (10cmooney) This configuration has been rolled out now across all CR routers. All looks ok, some slight increase in traffic in via eqord, and slight decre... [11:40:12] 10netbox: Request to add Custom fields in Netbox - https://phabricator.wikimedia.org/T284614 (10Volans) After having chatted a bit with Papaul and then Cathal last week I was about to write my thoughts but then I was carried away by other more urgent things. So here I am late to the party in the reply :) Thanks... [11:42:06] 10CFSSL-PKI, 10Packaging, 10User-jbond: Package Latest version go-cfssl - https://phabricator.wikimedia.org/T283840 (10jbond) [12:15:24] 10CFSSL-PKI, 10Packaging, 10User-jbond: Package Latest version go-cfssl - https://phabricator.wikimedia.org/T283840 (10jbond) [12:55:18] 10netops, 10SRE, 10Patch-For-Review, 10Sustainability (Incident Followup): ospf link-protection - https://phabricator.wikimedia.org/T167306 (10ayounsi) Deployed to cr3 and cr4-ulsfo. Some interesting (and expected) finding: No brainer, backup from cr3-ulsfo to mr1-ulsfo is cr4-ulsfo `name=mr1-ulsfo cr3-ul... [13:08:02] 10SRE-tools, 10netops, 10DC-Ops, 10SRE, 10Patch-For-Review: Allow idrac tftp fetching of firmware updates (either to existing tftp or new solution) - https://phabricator.wikimedia.org/T283771 (10Papaul) @jbond Yes uploading to IDRAC interfaces is very slow. [15:16:57] jbond, volans, should the profile::contact stuff go in the SRE meeting doc or it's too soon? [15:19:39] 10CFSSL-PKI, 10Packaging, 10User-jbond: Package Latest version go-cfssl - https://phabricator.wikimedia.org/T283840 (10jbond) [15:21:59] XioNoX: right now its is juts a file on the system, i think it would be nice if we had some tooling which shows its usefullness before presenting. but im not against it [15:22:22] XioNoX: I didn't had yet a chance to look at how it looks in real life, I've your CR in the to-read [15:22:28] and to add myself some stuff [15:23:38] lets leave it untill next time let some people here have more of a look and see if we can come up with a use case (possiblt a script for dc ops to use some how inform the correct team/user about a restart/reboot) [15:24:04] ok! [15:24:41] 10Puppet, 10observability, 10User-fgiunchedi: Puppet failing on the alert hosts should alert - https://phabricator.wikimedia.org/T283151 (10fgiunchedi) [15:24:47] 10Puppet, 10observability, 10User-fgiunchedi: Puppet failing on the alert hosts should alert - https://phabricator.wikimedia.org/T283151 (10lmata) a:03fgiunchedi [15:25:46] 10Puppet, 10observability, 10User-jbond: Add additional prometheus metrics to puppet runs - https://phabricator.wikimedia.org/T283585 (10lmata) @jbond moving to radar for visibility, let me know if you would like our help . [15:29:04] 10Puppet, 10observability, 10User-jbond: Add additional prometheus metrics to puppet runs - https://phabricator.wikimedia.org/T283585 (10jbond) @lmata, happy to help but definitely good to get review from someone in observability. some of this is already in place so may not bee needed [15:54:26] 10CFSSL-PKI, 10Packaging, 10User-jbond: Package Latest version go-cfssl - https://phabricator.wikimedia.org/T283840 (10jbond) Every thing builds now but the PATH issue, for now and while i wait for a gerrit repo i have added the changes to [[ https://github.com/b4ldr/cfssl/tree/debian/sid | github ]]. Note... [23:36:45] 10Mail, 10GitLab (Initialization): Outgoing mail for Gitlab - https://phabricator.wikimedia.org/T276459 (10brennen) [23:37:02] 10Mail, 10GitLab (Initialization): Outgoing mail for Gitlab - https://phabricator.wikimedia.org/T276459 (10brennen) 05Open→03Resolved