[08:31:33] 10netbox, 10Infrastructure-Foundations: Make more extensive use of Netbox custom fields - https://phabricator.wikimedia.org/T305126 (10Volans) @ayounsi some nits in the code/results: * https://netbox-next.wikimedia.org/dcim/inventory-items/62/ has a description of `,RMA:2019-08-08,RMA Task:T226422` * it's the... [08:41:58] 10netbox, 10Infrastructure-Foundations: Netbox: Add license keys as inventory items - https://phabricator.wikimedia.org/T311008 (10Volans) I agree that the description is the only field "usable" at the moment for it. How sensitive are those licences? Are we ok trusting Netbox to host them? This will also proba... [09:34:43] 10Puppet, 10Infrastructure-Foundations, 10Patch-For-Review, 10User-jbond: puppetdb seems to be slow on host reimage - https://phabricator.wikimedia.org/T263578 (10MoritzMuehlenhoff) This seems to happen again, today's reimages of ganeti2012 and ganeti2028 failed since the host key change didn't get properl... [09:35:16] excellent news from Debian: https://tracker.debian.org/news/1346226/accepted-puppetdb-7101-1-source-into-experimental/ [09:35:37] yay [09:37:13] whooop whoop [09:37:52] 10netops, 10Infrastructure-Foundations, 10SRE, 10Traffic-Icebox, 10User-jbond: fetch_external_clouds_vendors_nets.py fails to update DigitalOcean network ranges - https://phabricator.wikimedia.org/T313206 (10Vgutierrez) [09:38:39] 10netops, 10Infrastructure-Foundations, 10SRE, 10Traffic, 10User-jbond: fetch_external_clouds_vendors_nets.py fails to update DigitalOcean network ranges - https://phabricator.wikimedia.org/T313206 (10Vgutierrez) p:05Triage→03Medium [09:38:39] jbond: moritz pinged me earlier for some issue related to the reimage cookbook that lead us towards the possibility that puppetdb in codfw has started to be slow again sometimes [09:39:09] any magic you might have in your pockets to "fix" it? I don't recall if you did cleanup some temporary data as part of your efforts last time [09:40:09] volans: i dont think there is anything. last time it ended up being related to relations so i guess is someone has added some global relation sohmwhere then it may have caused an issue [09:40:22] or as moritz suggests a new fact [09:40:48] ill take a look to see if i9 can spot anything obvious; however i also plan to take a look at puppetdb7 next q [09:41:03] which you never no may just fix everything :P [09:41:56] eheheh [09:42:53] yeah relaions might be the culprit here, I don't thin we added new heavy facts [09:43:22] no i dont think so, which is annoying they are much easier to track down then relations :( [09:44:07] if you need a hand let me know, I think I put in the task all the queries I used back then [09:44:25] yes i think everything is in the task so should be fine thanks [09:44:28] did we setup auto-vacuum? [09:44:46] where we==you :) [09:44:58] 10netbox, 10Infrastructure-Foundations: Netbox: Add license keys as inventory items - https://phabricator.wikimedia.org/T311008 (10ayounsi) > How sensitive are those licences? The existing ones are not sensitive as they're generated based on the device's serial number (can't be used anywhere else) > Are we o... [09:45:11] i think so but will need to double check [09:45:17] ack, thx [09:46:27] 10netbox, 10netops, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: Represent sub-interface and bridge device assocations in Netbox - https://phabricator.wikimedia.org/T296832 (10Volans) >>! In T296832#8065318, @cmooney wrote: > @volans could you point me at any existing custom_facts and the cod... [09:50:35] 10netbox, 10Infrastructure-Foundations: Make more extensive use of Netbox custom fields - https://phabricator.wikimedia.org/T305126 (10ayounsi) > * https://netbox-next.wikimedia.org/dcim/inventory-items/62/ has a description of ,RMA:2019-08-08,RMA Task:T226422 Yep, that's expected (and the only reason why I pu... [10:04:44] 10netbox, 10Infrastructure-Foundations: Make more extensive use of Netbox custom fields - https://phabricator.wikimedia.org/T305126 (10ayounsi) [10:14:15] 10netbox, 10Infrastructure-Foundations: Netbox: use Custom Model Validation - https://phabricator.wikimedia.org/T310590 (10Volans) I did a quick pass in the UI and found some things, ofc to be integrated with all the checks in the current reports that can be converted to field validators: * site.slug: lower c... [10:51:41] XioNoX: all the outstanding diffs for homer are "expected"? [10:52:43] volans: in some way, it's due to the work being done in https://phabricator.wikimedia.org/T304710 fixes are https://gerrit.wikimedia.org/r/c/operations/software/homer/+/813604 and https://gerrit.wikimedia.org/r/c/operations/software/homer/deploy/+/813589 [10:53:32] ack, getting there, I'm spooling my inbox from things I left to check last week for inspiration [10:53:54] yeah, I'm seeing that in my inbox :) [10:54:26] 10netbox, 10Infrastructure-Foundations: Netbox: Add license keys as inventory items - https://phabricator.wikimedia.org/T311008 (10Volans) ACK, then SGTM. [10:56:06] 10Puppet, 10Infrastructure-Foundations, 10Patch-For-Review, 10User-jbond: puppetdb seems to be slow on host reimage - https://phabricator.wikimedia.org/T263578 (10jbond) >>! In T263578#8083691, @MoritzMuehlenhoff wrote: > This seems to happen again, today's reimages of ganeti2012 and ganeti2028 failed sinc... [10:56:47] getting there :) [11:00:34] 10Puppet, 10Infrastructure-Foundations, 10Patch-For-Review, 10User-jbond: puppetdb seems to be slow on host reimage - https://phabricator.wikimedia.org/T263578 (10Volans) For the record, the last state change of the Icinga alert that is alerting since then is `Last State Change: 2022-07-12 16:11:29`, re... [11:13:58] gentle reminder to update the etherpad for today's meeting, we'll self-manage (see email from fai.don) [11:39:10] 10Puppet, 10Infrastructure-Foundations, 10Patch-For-Review, 10User-jbond: puppetdb seems to be slow on host reimage - https://phabricator.wikimedia.org/T263578 (10jbond) i have re-synced puppetdb, however we need to prevent this from happening again. It seems we can increase the wal_keep_size but it may b... [11:58:27] 10Puppet, 10Infrastructure-Foundations, 10User-jbond: puppetdb postgress: Improve postgress standby server - https://phabricator.wikimedia.org/T313217 (10jbond) [11:58:34] 10Puppet, 10Infrastructure-Foundations, 10User-jbond: puppetdb postgress: Improve postgress standby server - https://phabricator.wikimedia.org/T313217 (10jbond) p:05Triage→03Medium [12:13:00] 10Puppet, 10Infrastructure-Foundations, 10Patch-For-Review, 10User-jbond: puppetdb seems to be slow on host reimage - https://phabricator.wikimedia.org/T263578 (10MoritzMuehlenhoff) I retried the ganeti2028 reimage and everything works fine again, thanks! [13:25:01] 10Puppet, 10Infrastructure-Foundations, 10User-jbond: puppetdb postgress: Improve postgress standby server - https://phabricator.wikimedia.org/T313217 (10Volans) Replication slots seems more interesting and tailored on what we need here as far as I can tell from a quick look. Thanks for opening this. [13:25:58] 10Puppet, 10Infrastructure-Foundations, 10User-jbond: puppetdb postgress: Improve postgress standby server - https://phabricator.wikimedia.org/T313217 (10jbond) [13:49:10] volans: are we doing async today or are we meeting? [13:49:58] as the group prefers, I don't mind meeting, we could share anthing interesting done last week, but can be done also async for me. No strong preference [13:50:41] cath.al and simon are out too [13:55:23] cdanis, moritzm, jbond, XioNoX, jhathaway: any preference for meeting vs async? [13:55:38] soft pref for async [13:55:40] happy to meet can always just end early [13:56:10] * jbond async is also fine with me [13:56:14] I'm fine either way, I guess if we meet it'll be quick anyway given some people are out [13:57:02] sure, either way is fine with me [13:57:31] either way is fine with me [13:58:09] let's take the meeting to discuss whether want to handle it async then :-) [13:58:15] ha! [13:58:45] lol [13:59:14] lol [13:59:35] alright, joining [13:59:38] quick meet it is then Id say