[01:44:54] 10SRE-tools, 10Infrastructure-Foundations: Netbox accounting report: exclude removed hosts - https://phabricator.wikimedia.org/T320955 (10wiki_willy) Hi @Volans - no problem, we can scrap the idea of having a "recycled status" in Netbox. For everything that gets deleted in Netbox, is there any feature or anyt... [08:40:45] 10Puppet, 10Infrastructure-Foundations: Consider migrating alternative configuration managment tooling - https://phabricator.wikimedia.org/T321874 (10jbond) p:05Triage→03Low [08:44:07] 10Puppet, 10Infrastructure-Foundations: Consider migrating alternative configuration managment tooling - https://phabricator.wikimedia.org/T321874 (10jbond) >>! @jhathaway > Thanks Brian for bringing up some alternative ideas! >>>! @bking >> I wonder if our energies might be better spent searching for >> alt... [08:47:06] 10Puppet, 10Infrastructure-Foundations: Consider migrating alternative configuration managment tooling - https://phabricator.wikimedia.org/T321874 (10jbond) >>! @bking > Thanks! Your perspective as a both a Puppet expert and relative n00b like me is very much appreciated. I hope you (and everyone else) will... [08:49:28] 10Puppet, 10Infrastructure-Foundations: Consider migrating alternative configuration managment tooling - https://phabricator.wikimedia.org/T321874 (10jbond) >>>>! @bking >> I agree, it will be very time-consuming and painful to move off Puppet. But the current situation also seems painful and untenable. >>... [09:18:05] 10Puppet, 10Infrastructure-Foundations: Consider migrating alternative configuration managment tooling - https://phabricator.wikimedia.org/T321874 (10MatthewVernon) [I'm not saying we should move to Ansible necessarily, but wanted to respond to something said up-thread :)] I've used Ansible a fair amount in p... [11:54:02] 10Puppet, 10Infrastructure-Foundations: Consider migrating alternative configuration managment tooling - https://phabricator.wikimedia.org/T321874 (10jbond) > but we're unable to migrate off a version that has been EOL for nearly 2 years without external help. Let me first start by saying that if there was so... [13:08:14] jbond: _joe_: we were thinking we should probably just increase the disk reservation of all the pcc workers, right? [13:10:03] cdanis: yes indeed at least double [13:10:40] ok cool -- just saw we had another user report yesterday of running out of disk https://wikimedia.slack.com/archives/C0153LQ5G82/p1666892421370739 [13:12:50] yes andrew.bogott was running some big pcc jobs yesterday so sort of expected but not great. ill do them on monday it probably make senses to move it all to some nfs storage which may need some minor tweaks to pcc [13:30:03] 10Puppet, 10Infrastructure-Foundations: Consider alternative configuration managment tooling - https://phabricator.wikimedia.org/T321874 (10jbond) [14:53:24] very generous of NL-IX :) [22:13:08] 10Mail, 10Data-Engineering-Operations, 10Data-Engineering-Planning, 10SRE: Change the analytics-alerts email alias to a mailman distribution list - https://phabricator.wikimedia.org/T315486 (10Dzahn) nice, works for me. thanks @BTullis [22:23:05] 10Puppet, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: Duplicate monitoring for systemd::timer::job - https://phabricator.wikimedia.org/T303253 (10Dzahn) @fgiunchedi I think both would be fine, either just don't worry about the duplicate part. I don't see it as a big problem. Or follow the sugg... [22:58:56] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10SRE, and 3 others: Q3:(Need By: TBD) rack/setup/install 2 new labstore hosts - https://phabricator.wikimedia.org/T302981 (10Dzahn)