[06:12:14] greetings [06:21:22] morning! [06:22:26] hey [07:04:18] hmm... I think I killed tools prometheus xd [07:04:30] ah, back up [07:15:42] morning [07:15:52] welcome back taavi \o/ [07:18:51] anything I should be aware of / looking at first? [07:21:22] Nothing urgent that I'm aware of :), thil.p joined [08:01:52] welcome back taavi ! [08:36:50] morning and welcome back taavi! [08:43:44] hmm.... I've seen a few harbor errors lately, in lima-kilo and now in toolforge, I think it might be misbehaving/having issues [09:07:06] are there any minutes from the backup meeting or is it just the recording? [09:33:14] taavi: no meeting notes IIRC [09:42:26] hello! who owns cloudbackup2003 ? [09:42:39] (phab tag or persons) [09:46:54] XioNoX: tools-infrastructure + cloud-services-team (tags) [09:47:49] godog: great, thanks, it's for https://phabricator.wikimedia.org/T430928 [09:48:58] XioNoX: ack thank you, right off the bat I'd say no depool needed for 20m loss [09:49:07] sweet [09:49:08] worst case there are backups running which will fail [09:49:16] though let's coordinate/agree on the task [09:49:27] it's in 3 weeks so no rush [09:49:42] ok! [10:28:16] godog: ha, saw T430651 and thought that looked familiar and then found https://gerrit.wikimedia.org/r/c/operations/puppet/+/1268942 [10:28:17] T430651: LOAD_BALANCER_HEALTH_CHECKS firewall set and asymmetry lvs1018 / lvs1020 - https://phabricator.wikimedia.org/T430651 [10:29:01] taavi: heheh indeed, I was only slightly confused [10:29:37] tbh the hiera key being named 'haproxy_allowed_healthcheck_sources' certainly doesn't help with the confusion of what it actually is supposed to contain [10:29:52] 100% agreed [10:32:08] taavi: re: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1306289 does nova self-recover on cloudvirt reimage when nova-compute comes back with a different ID (or does it? that's what I understood was the problem) [10:33:02] https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/refs/heads/production/modules/openstack/manifests/nova/compute/service.pp#132 means we get a static ID for new hosts [10:33:41] the problem was that older hosts had different IDs than what fqdn_uuid() would generate, so the hiera key was meant to be a transition thing for those older hosts, with new hosts using the automatically generated one [10:34:21] taavi: ah! thank you, now I get it [10:34:36] what cloudvirt was the cutoff? I'll remove the IDs for the newer ones [10:36:20] hmm if we have set a different one via the hiera key already, I'm not sure if we can remove it anymore without causing the issue this mechanism was meant to prevent [10:38:28] ok so for the cloudvirts I provisioned initially the key wasn't set, and the uuid I copied was on the filesystem, which means that will match fqdn_uuid [10:38:51] i.e. I added the hiera key only after the hypervisor was known to nova [10:39:21] to be verified of course before we remove the key [10:39:33] that seems trivial enough to verify with pcc [10:40:07] ok thank you, I'll open a task to do the cleanup [10:40:37] and nuke the nova id bits from https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Maintenance#new_cloudvirt_install [10:41:01] thanks [10:41:41] which I can't edit right now because vpn )o) [10:43:02] * dcaro lunch [10:45:00] godog: now you can [10:48:18] taavi: thank you <3 out of curiosity what did you do ? [10:48:41] https://wikitech.wikimedia.org/w/index.php?title=Special:Log&logid=1014587 [10:49:13] ah! that makes sense [10:49:39] mhh should that be part of sre onboarding ? [10:49:58] or wmcs onboarding ? [10:50:27] maybe? [10:52:04] some folks need the buttons which tend to be a bit sensitive (it also lets you delete pages, protect them, edit protected pages, block people, etc), but not everyone, and it's easy enough to grant later so maybe not? [10:52:50] yeah fair enough, easy to do ad-hoc when/if needed [11:14:55] godog: can https://gerrit.wikimedia.org/r/c/1306672 be reverted now that https://gerrit.wikimedia.org/r/c/1306690 is merged? [12:53:48] XioNoX: yes 100%, I'll revert [12:59:09] taavi: re: depool strategy, what do you think re: cloudbackup2003 in T430928 ? [12:59:09] T430928: codfw: rack B7 maintenance - https://phabricator.wikimedia.org/T430928 [13:32:44] godog: in an ideal world I'd say that it's fine and the system would automatically recover, we can deal with the downtime as backups are not relevant to the live traffic path [13:32:57] we haven't tested if that is exactly the case, but this seems like a good opportunity? [13:35:22] indeed, +1 to no depool action [13:35:51] best work is having no work to do at all [13:46:26] ok that's https://gerrit.wikimedia.org/r/c/operations/puppet/+/1307149 then [13:51:58] neat, LGTM [15:25:52] :q [15:27:17] XX [15:40:17] hi I could use a hand :-] Dan Duvall created an application credential named "zuul-swift-upload". It is intended to upload some logs to an "artifacts" object storage [15:40:53] it looks he has created it with access rule and I ma not sure those rules are working, and of course I don't have access to the credential he has created to verify / amend the rule :] [15:41:06] the only clue I have is the rules listed in the comment he wrote https://gerrit.wikimedia.org/g/integration/config/+/refs/heads/master/zuul.d/secrets.yaml :) [15:41:10] hashar, we're in a meeting but I will catch up in 15-20. Ping me if I forget :) [15:41:14] sure! [15:41:45] an alternative is to introduce a new openstack rule (object-storage-uploader) that would have the proper access rules maintained in Puppet (I guess?) and exposed to all tenant. [15:42:24] so if one needs to create an application credential that can only upload they could use that role. Then of course we can't restrict it to a single bucket :] [15:47:25] andrewbogott: https://phabricator.wikimedia.org/P94715 [15:47:50] uuu, /me clinic duty [15:48:08] :) [15:48:19] thx taavi [15:49:08] that explains why barbican is failing but not why there's a cron trying to do barbican things... [15:52:22] thanks volans for the rotation script xd [15:55:08] hashar: can we wait and ask Dan if it ever worked for him? I haven't ever tried that particular form of restricted roles so would have to iterate a bit. [15:55:31] it never worked for him, the software stack was broken in sveral regard and I spent the week fixing it :] [15:56:02] I am pretty sure I ended up dropping the access rule last year cause I could not figure out the proper ones to use [15:56:35] then when I read https://wikitech.wikimedia.org/wiki/Help:Object_storage_user_guide#Swift_API , it mentions a role `object_storage` which only grants access to storage and I think that will suit it [15:57:19] but when creating a new application credential I can only grant "member" and/or "reader" (possibly because I am not an admin on the zuul project) [15:58:09] huh, those are the only roles it offers me too. Let's see why that is... [15:58:17] quick review https://gitlab.wikimedia.org/repos/cloud/wmcs/utils/-/merge_requests/4 [15:58:49] maybe the role is not grantable despite what the wiki doc state. But at least I can see an "object_storage" in puppet :) [16:00:40] ah, it's because you can only delegate roles that you have, of course. [16:00:43] that's easy to fix :) [16:00:53] we never built an user interface to give that role (or any of the other specialized roles out there) out [16:01:13] yeah, because it's pretty much only used internally for service accounts... [16:01:23] hashar: try now? [16:01:53] the credential dan has created has some access rules as well, nto sure whether that will work [16:01:57] * hashar triggers a new job to verify [16:02:24] also, just to double check, have you seen the blue box on top of https://wikitech.wikimedia.org/wiki/Help:Using_OpenStack_APIs#Application_Credentials ? [16:02:29] hashar: what I /want/ you to do is to actually investigate how to make those access rules work properly... but I understand that you might have other things you need to do [16:03:16] taavi: Did you write "at least add restrictions to limit which APIs the credentials can access" and if so does that mean you've actually done that ever? [16:04:10] andrewbogott: oh it is never ending, yesterday during our team meeting I bitched I had reached the 6th level of inception depth. I was debugging ruby :b [16:04:44] if that one fail, I'll give up. Revisit with Dan next week and come back to your team with a formalized problem statement [16:04:45] andrewbogott: I wrote that there to stop people from just using their own accounts directly, but I still think a dedicated service account is the only solution that does not scream "I am an easy target for privilege escalation attacks!!" [16:04:57] fair [16:06:08] HttpException: 401: Client Error for url: https://object.eqiad1.wikimediacloud.org/swift/v1/AUTH_c26d9d326bdf464fa1025939ded7e5a2/artifacts, Unauthorized\n [16:06:08] :-] [16:06:29] hashar: did you create a new cred with the objectstorage role? [16:06:46] that was what I meant by 'try now', I was not at all clear [16:06:49] OH [16:06:52] sorry [16:07:19] I granted you personally the role. [16:07:30] If that works then the next step is for you to create a service account and then I'll adjust its roles [16:08:00] yeah I can now grant `object_storage` and `k8s_admin` to an app credential [16:08:12] and of course I dont have access to Dan created one ( named zuul-swift-upload ):b [16:08:54] then I guess I can create a new one, grant it `object_storage` rule without any access rule , export & encrypt that in Zuul config and see what happens. [16:08:54] Or do you have the ability to grant the role to Dan created credential? [16:08:56] It's just amazing that we've gone 20+ years without a reliable way to share credentials among staff :) [16:09:02] ahaha [16:09:25] I don't think I can adjust roles assigned to credentials (that sounds scary!), you'd need to make a new one. [16:09:29] the day someone solves the password/credentials problem for good, it will be a revolution larger than the current AI boom! [16:09:36] Is zuul-swift-upload a service account, or an app credential? [16:09:49] pretty sure that is an app credential. Aka it is not in LDAP [16:10:10] ok, so it probably is attached to Dan the person, which is what Taavi is trying to discourage [16:10:17] oh [16:10:58] so we would need to create a new account flagged as a service, add it to the `zuul` tenant and grant it the `object_storage` role. Then generate an app credential for that service account and use that? [16:11:13] Yep [16:11:33] Is it easy for you to test with a hashar-owned credential just to make sure that object_storage actually solves your problem? [16:11:57] yeah I need to create it, refigure out how to encrypt the credential, update zuul conf and check :] [16:12:06] that is not the end of the world. At least that will validate the role works [16:15:34] * andrewbogott wishes hashar good health on his journey [16:19:39] zuulclient.api.ZuulRESTException: Insufficient privileges to perform the action. [16:19:39] ERROR - Command encrypt completed with error(s) [16:19:41] too bad :b [16:26:12] I wonder... is that job using swift APIs or S3? Seems like the endpoints would be different... [16:26:22] Oh, but... [16:26:42] um, I have just remembered that radosgw uses a different policy engine from other openstack services. [16:26:52] So probably nothing works like we expect :( [16:29:05] * andrewbogott makes T430999 [16:32:13] [16:41:06] * dcaro approves of the swearing [16:53:34] good it only took me half an hour to encrypt a secret [16:53:40] https://gerrit.wikimedia.org/r/c/integration/config/+/1307204 zuul: change OpenStack secret [16:53:45] I'll clearly never qualify for staff :b [16:57:28] bd808: do you have ownership of https://lists.wikimedia.org/postorius/lists/toolhub-dev.lists.wikimedia.org/ ? Can you add thibaut? (maybe add aputhin as admin too) [16:58:41] andrewbogott: HOLY !!!!! IT WORKED AHHHHHHHHHHH [16:59:05] echo https://zuul.wikimedia.org/t/wikimedia/build/e0524118e6a647959ee23f7df16910df : SUCCESS in 12s [16:59:05] hello-world-container https://zuul.wikimedia.org/t/wikimedia/build/c5e51ddca1764c52a2f22dfeba581c95 : SUCCESS in 23s [16:59:13] great! So next steps are straightforward right? [16:59:34] just lmk when you have an account for me to grant that role on [16:59:36] yeah migrate to GitHub actions and see builds randomly failing as their AI is overhauling their infra :b [16:59:58] also that [16:59:58] but yea I took note of switching to a service account, Iwill check that with dan next week [17:00:04] huge thanks andrewbogott! [17:00:55] It's still not an ideal solution (should be a UI for it at least) but I'll try to keep you unblocked at least. [17:01:26] well https://horizon.wikimedia.org/identity/roles is not happy :] it is missing roles I guess [17:03:40] dcaro: I accepted the subscription request from tlepage, removed 2 folks from the list who don't work here anymore, and added aputhin@wikimedia.org as a list owner. [17:03:54] thanks! [17:05:11] * aputhin feeling thankful and empowered [17:05:29] andrewbogott: again huge thank you! [17:05:32] I am off [17:05:42] * thilp is unused to that speed after years of corporate [17:06:07] aputhin: if folks start using that list again I imaging a number of people will want off of it. :) [17:06:19] xd [17:12:52] okok, I'm kinda exhausted, cya on monday! [17:12:54] * dcaro off