[08:06:29] good morning folks [08:06:33] https://gerrit.wikimedia.org/r/c/operations/puppet/+/770879/ for kubernetes2006 :) [08:06:37] (another VM) [08:21:47] elukey: groundhog day +1 :-) [08:22:33] I'll be off in a bit for yet another "new" mainboard [08:22:55] ahahahah okok I wasn't getting the reference [09:13:43] kubernetes2006 ready for a check before uncordon [09:13:48] (lgtm afaics) [11:27:36] elukey: looks good to me as well [11:27:56] thanks, uncordoning :) [11:29:33] I wonder if I should allow ingressgateway to shedule on the kask nodes as well as pybal will see them as down now all the time [11:36:11] is there a specific reason to keep dedicated vms for kask? [11:36:40] the initial reason was more seperation for the PII [11:37:27] guess that's a thing still :-) [12:58:41] 10serviceops, 10Security-Team, 10GitLab (CI & Job Runners), 10Patch-For-Review, and 2 others: Setup GitLab Runner in trusted environment - https://phabricator.wikimedia.org/T295481 (10Jelto) [13:28:25] jayme: I created https://gerrit.wikimedia.org/r/c/operations/puppet/+/770912 as alternative proposal for the VMs, I just realized that the LVM config for the single virtual disk use case doesn't really make a lot of sense [13:28:34] lemme know what you think about it [13:28:46] (the change moves us more closer to the standard flat + virtual cfgs) [13:44:20] (and yeah if people like the new recipe I'll reimage..) [13:59:24] we should get you an "I reimaged wikipedia" shirt [14:00:55] * elukey cries in a corner [14:01:13] I was so happy this year that I didn't need to reimage the hadoop cluster [14:01:54] I'm obviously totally fine with dropping LVM but in that case it might really make sense to generate something totaly generic (like flat-noswap) [14:02:58] this is my plan, I kept a super similar flat.cfg recipe to wait for Filippo's opinion [14:04:37] I thought you would get away without changing the recipe tbh...but now that that needs manual intervention because of increased block storage it does not really make sense to keep LVS [14:06:31] jayme: the recipe can be modified to circumvent that in theory, there may be an option, but having an LVM with a single virtual disk seems not worth it [14:06:44] we can keep it if people prefer, no strong opinions [14:07:28] nono [14:07:55] btw. if it is more easy for you to do all nodes at the same time, we could depool sessionstore and echostore in codfw - they are only relly used in eqiad anyways AIUI [14:08:13] nono one at the time is ok, so I can modify disks etc.. [14:08:21] ok [14:09:16] it is all time billed that I'll present to your manager at the end [14:35:42] that's fine :) [14:36:36] he has an unlimited supply of friendly looks to pay you in [14:50:01] friendly looks? Your manager? [14:50:04] * elukey runs away [14:50:05] :D :D :D [16:46:29] 10serviceops, 10Release-Engineering-Team, 10SRE, 10SRE-Access-Requests: Add some users to the docker group on deployment servers - https://phabricator.wikimedia.org/T303450 (10Joe) a:03Joe [17:11:20] 10serviceops, 10Release-Engineering-Team, 10SRE, 10SRE-Access-Requests: Add some users to the docker group on deployment servers - https://phabricator.wikimedia.org/T303450 (10Joe) I would tentatively grant full docker access to the people listed above, but to be revisited at a later time, maybe creating a... [17:58:17] 10serviceops, 10Release-Engineering-Team, 10SRE, 10SRE-Access-Requests, 10Patch-For-Review: Add some users to the docker group on deployment servers - https://phabricator.wikimedia.org/T303450 (10Majavah) Is there a reason this is limited to the releng team? T297673 says that all scap syncs will be build... [18:11:55] 10serviceops, 10SRE, 10Thumbor: Migrate thumbor to Kubernetes - https://phabricator.wikimedia.org/T233196 (10WDoranWMF) [18:12:48] 10serviceops, 10SRE, 10Thumbor, 10User-jijiki: Upgrade Thumbor to Buster - https://phabricator.wikimedia.org/T216815 (10WDoranWMF) [18:14:32] 10serviceops, 10Release-Engineering-Team, 10SRE, 10SRE-Access-Requests, 10Patch-For-Review: Add some users to the docker group on deployment servers - https://phabricator.wikimedia.org/T303450 (10dancy) @Majavah The container build stuff happens under `sudo -u mwbuilder`, and `mwbuilder` does have permis... [18:14:43] 10serviceops, 10SRE, 10Thumbor, 10User-jijiki: Upgrade Thumbor to Bullseye - https://phabricator.wikimedia.org/T216815 (10WDoranWMF) [18:15:27] 10serviceops, 10SRE, 10Thumbor, 10User-jijiki: Upgrade Thumbor to Bullseye - https://phabricator.wikimedia.org/T216815 (10WDoranWMF) [18:15:34] 10serviceops, 10Release-Engineering-Team, 10SRE, 10SRE-Access-Requests, 10Patch-For-Review: Add some users to the docker group on deployment servers - https://phabricator.wikimedia.org/T303450 (10dancy) To clarify further, releng needs direct access to dockerd (without having to sudo) while we're debuggi... [18:16:33] 10serviceops, 10MW-on-K8s, 10Patch-For-Review, 10Release-Engineering-Team (Done by Feb 23 🧟): Build MediaWiki images for kubernetes on the deployment servers - https://phabricator.wikimedia.org/T297673 (10dancy) [23:40:31] 10serviceops, 10Prod-Kubernetes: Helmfile !log messages do not indicate failed deployments - https://phabricator.wikimedia.org/T303900 (10bd808)