[07:03:41] hey folks, deploy1003's /srv is almost full [07:07:27] <_joe_> i will remove some older docker images [07:10:15] we have a timer to do image prune, maybe I can also try a system prune with the usual filter until etc..? [07:10:22] Not sure if it will get something more [07:11:16] <_joe_> I've removed a few images, for a total of 20 gb in theory [07:11:24] ah no ok I used image prune with 120h and it removed 18GB [07:11:45] <_joe_> I removed old images left dangling [07:12:01] <_joe_> because there were failed containers stopped but not removed based on them [07:12:18] all right we are at ~70% now, I'd personally lower down the image prune to 120h instead of 168h [07:12:38] <_joe_> elukey: yesterday was a particularly intense day of releases it seems [07:12:46] <_joe_> but also, why is mediawiki-staging 30 GB [07:13:26] <_joe_> wow every php* directory is almost 10 GB [07:16:08] (relocating, bbiab) [09:36:22] topranks: welcome back! :), do you know if there's a task for the ceph ha/switches expansion? [09:38:30] there is - T388617 [09:39:15] we had to stall it until next FY to look at it though, with the amount of projects we have on the go the switches would just be sitting idle until then anyway as we'd not have time to get everything set up [09:50:20] thanks! I'll link it too [13:48:47] btullis: ok to merge your puppet changes? [13:48:56] Yes please. [16:45:35] jhathaway: hey, unfortunately I think https://gerrit.wikimedia.org/r/1141963 has broken something, seeing https://phabricator.wikimedia.org/P75885 on cloud-vps hosted puppetdb hosts [16:46:14] ah shoot [16:47:11] i think it's trying to create things even with ensure => absent [16:51:33] ah, that could be true [16:52:20] taavi: let me take a quick look, if it is not obvious we can revert [17:01:53] taavi: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1143621 [17:06:59] jhathaway: that seems fine (and I think that's restoring previous behaviour? there doesn't seem to have been any explicit cleanup beforehand) [17:10:34] heads-up, I may have goofed up a confctl pool command. Fixing it now [17:10:49] I don't think it's gonna hurt anything, but might set off validation alerts [17:12:25] taavi: yeah I don't believe there was [17:13:40] OK, config-master looks happy now, if there are any alerts LMK [18:53:37] Any of y'all running a WMF-provided macbook? I just got a new one and it has jamf remote management on it. I asked OIT to remove it and they say they've switched me to a 'bare minimum' profile which still feels like a rootkit to me. [18:53:42] Are my expectations out of date and everyone has ssh keys on remote-managed hardware? [18:54:05] er, weren't SREs supposed to be expemt from that? [18:54:07] *exempt [18:54:17] I thought so [18:54:19] I don't have a Macbook but that is my knowledge? [19:03:21] andrewbogott I do! I had a bit of a freak-out the other day when BitDefender suddenly showed up on my desktop and started connecting to its "BitDefendr DCI" VPN [19:03:35] OIT says that's normal though [19:03:50] inflatador: ok, noted! Can you say that in #mediawiki-security? [19:03:51] O.o [19:04:42] Sure. FWiW that is normal and expected behavior