[11:21:13] neat :), got project creation from cookbook creating an MR on tofu-infra [11:21:13] https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/104 [11:21:39] (uses python yaml to parse the contents... so it does not like empty lines :/) [11:33:28] there is an incoming refactor happening that will render the code that produces that MR obsolete [11:38:10] what's the change? the only dependency there is the yaml snippet for the project stuff [11:38:20] (should be easy to adapt to a different file/yaml dict) [11:39:24] I don't know yet. I can't tell what layout we will have in, lets say 2 months. In the last 2 months it changed 2 or 3 times already [11:40:26] that's ok, as I say, unless the refactor needs the project definition to be split into different files and merge in different MRs or something like that, should be easy to adapt [11:40:56] this one should simplify it even more https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/93/diffs#2483d4abc8d1f2222bd05250c015a05a04637215 [11:41:56] it won't be hard to update the code... but I'm not a big fan of auto-generating patches from cookbooks, also because I prefer to run cookbooks from cloudcumin [11:42:15] yeah that, I don't think automating a MR is the right abstraction ehre [11:42:18] here* [11:43:30] my hope is to make the tofu MR super-simple, like as simple as running a cookbook, but we're not there yet [11:43:35] I think we shared a similar opinion yesterday already [11:44:09] I think it's ok to improve the cookbook in the meantime though, my only request is that I can still run the cookbook from cloudcumin [11:44:13] dhinus: the cookbook can run from cloudcumin, there's no problem there [11:44:18] ok then :) [11:44:29] but how can it create the patch from there, without my ssh key for gitlab? [11:44:44] it creates it with the api token (uses the api) [11:45:14] ah I see! [11:45:39] but it says "Cookbook and David Caro committed" [11:45:47] because it used my token [11:45:48] xd [11:45:53] (ran it from my laptop) [11:48:19] dhinus: do you need any help with that latest refactor? the MR draft has been opened for a while, and if you are not finding the time, I might be able to take over in the next couple of days [11:48:53] sure, I have other things that are more urgent, feel free to work on it [11:49:18] ok, I'll let you know if I start working on it [11:50:19] btw. what's the flow to deploy a tofu mr? Create MR, plan, review apply MR, merge? [11:50:40] we have a meeting later today about Q2 planning, I think some opentofu work could maybe fit in the priorities [11:51:10] dcaro: I think we should write it somewhere if we don't have it, because it's easy to get confused [11:51:23] it's not in the wiki (or at least I don't see it) [11:51:32] https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/OpenTofu [11:52:25] let me write the steps in there [11:52:53] there's also https://wikitech.wikimedia.org/wiki/Help:Using_OpenTofu_on_Cloud_VPS, though no steps there either [11:53:26] the latter is for cloud vps users, in their own project [11:53:44] and they're free to choose their workflow, I think. but we should define the workflow for the tofu-infra repo [11:54:14] ack [11:54:59] dcaro: I just updated https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/OpenTofu#Automated_workflow_via_cookbook [11:55:46] thanks [11:56:38] when you apply tofu, the projects are created right away? (as in, they are not queued for creation of some sort no?) [11:57:17] they are created right away [11:58:41] if actions on dependent resources happen in the same tofu run, it should figure out the right order [11:59:12] ie. at the moment when a project is created, a security group is created as well. Tofu is "smart" to create the security group after the project has been created [12:01:33] ack, I think I'll have a fully working cookbook today that will allow us to not have to do any other step than run the cookbook to create a project with certain users and maybe specific quotas [12:02:21] dcaro: I will insist on that idea being the wrong way to go [12:03:20] thanks for your input, but unless you have any other reasons you have not shared, I'll go forward with it and simplify my life for at least the next 2 months [12:03:41] (and potentially others) [12:07:42] I believe you will better simplify yours and everyone's life if you invest your time adding support for users and quotas to tofu-infra [12:08:13] * dhinus lunch [12:08:40] arturo: thanks again, but I disagree [12:10:52] * dcaro lunch [13:20:36] Any approvers? https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/107 I'll merge myself if nobody disagrees (it's a test project, but I want to test the whole flow) [13:21:59] hmm... from the plan, this looks wrong maybe? [13:22:02] https://www.irccloud.com/pastebin/uBSsk2Pi/ [13:22:14] the project name was not replaced? [13:22:56] that's correct, it cannot compute it at plan time [13:23:09] I think [13:23:29] would be nice to have a primer of "what does a good plan look like?" [13:23:36] for us at least [13:23:54] shouldn't it say 'known after apply' like the others? [13:24:22] having the plan in the MR comments is very useful so you can ask for a review [13:24:58] but yeah we could write a checklist of things you want to verify in a plan output [13:25:07] agree, though I would try to avoid needing someone else to review it, self-review should be enough in most cases [13:25:24] that's a literal string [13:25:37] I disagree, to me the review process is maybe the biggest advantage of tofu vs spicerack :) [13:26:02] the `` string is a placeholder for a neutron API internal template. It is expected like that [13:26:05] you'll need two people then for a process that already had review [13:26:09] in this case I was also confused by the thing [13:26:30] so you need three people to get a project created [13:27:01] that template is from an earlier tests of mine, I will drop it, it is no longer needed [13:27:32] dcaro: I think the MR review might eventually remove the need for the +1 in the task [13:27:51] +1 for doing that [13:28:12] I would still try to remove the need of specific knowledge to know if the plan is right for simple cases [13:28:35] (that means now for example, that only you or artruro, in this case actually only arturo can approve this MR) [13:28:57] without the extra effort of having to learn everything you have learnt in the last half a year [13:29:15] that's OK and expected: we are in the middle of a transition [13:29:25] that's not ideal but I see it as part of the learning curve of introducing a new technology... being slower for a while, for hopefully being faster in the long run [13:31:43] I agree we should be careful before adding extra complexity to existing processes, but there was a tradeoff of adding some complexity here in exchange of simplifying the work arturo is doing on networks and other things [13:31:46] the goal should be to make it simpler than it was [13:32:22] I understand this is an interim stage, but if we have to teach everyone we want adding projects to be able to interpret custom tofu plans, I'm not sure that's simpler than before [13:32:43] I don't think it is simpler right now, no, but hopefully it will be in the end [13:34:29] I tried to simplify it a bit with my refactor, but as always we're juggling a lot of things at the same time, and with very few people [13:34:41] it's hard to decide which things to simplify first :) [13:34:43] heh, a cookbook output is usually filled with lots of useless information. A tofu plan is arguably easier to understand [13:35:47] is the process harder than before? at the moment I think yes. is it worth trying to track projects in opentofu? also yes, IMHO [13:36:52] you only need the cookbook output when it fails, not in the "good path", you need the tofu plan all the time [13:37:21] dhinus: I agree, I would add, is it adding it to the cookbooks simplifying the current state? yes, big time [13:38:01] (it being generating the tofu patch + plan + merge + create user + add quotas + check name validation + check user validation + ....) [13:38:27] I'm in favour of your cookbook patch, until we can do the same things in tofu [13:38:52] I think I have a better alternative in mind, but your one is ready and my one is only in my mind :) [13:40:15] I agree that most of the checks + resources can be moved eventually to tofu, and that'd be good, I don't think we should stop using the cookbook as the entry point, mainly because there's things we would never be able to do with tofu (ex. managing nodes, drainages, etc.), and having the cookbook gives you a single point of entry, that tofu can't give you [13:40:32] (I don't care if we move from cookbooks to whichever else, but tofu as it is today can't do that stuff) [13:40:35] I don't think tofu will ever replace 100% of cookbooks [13:41:01] but I think it could replace project creation [13:41:09] but it needs more work [13:41:11] and many of the processes can be "self-documented" as cookbooks, instead of long wiki pages with steps [13:41:37] can you create users from openstack? [13:41:57] you can add users to projects [13:42:39] what do you mean "create users"? [13:42:41] I'm just wondering, I think you can't right? it's all ldap/bitu nowadays no? [13:42:59] yep I think it's correct [13:43:04] (it always was ldap, but bitu is new) [13:43:22] yes [13:50:34] arturo: there's a bunch of stuff now in the plan [13:51:00] dcaro: what plan? [13:51:05] https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/109 [13:51:42] that's the handle default sec group option? [13:52:17] yes [13:52:28] that is expected [13:52:45] you need to run the plan for eqiad1 too [13:52:55] okok, let me add that to the code [13:53:42] actually, you need to run the tofu cookbook, which does it by default [13:54:05] I'm doing that, but I was passing the cluster because it's mandatory in python (fixing the types in the tofu cookbook) [13:55:42] dcaro: you will need to fix the unwanted line changes. I wont accept that patch as it is [13:56:25] arturo: why not, there's no way for the parsing library to generate anything else, and the change is just style (and does not impact it much imo) [13:56:30] ? [13:57:33] I think you know the reasons, no need for me to write a bible here [13:57:46] I really don't know them [13:58:14] so I'd appreciate it if you wrote them somewhere [14:00:58] you have been ignoring the hints and guidelines, we have communicated the roadmap, the current transition status, asked for your understanding, and asked that you dedicate your engineering time to help in this transition and roadmap [14:01:02] you are making it very difficult to collaborate with you [14:03:19] * arturo food time [14:42:48] none of that (even if all of it were true) has nothing to do with the change I proposed, that does not affect at all the current roadmap, except easing the current transition period (with minimal effort!) [14:44:13] I'm sorry, but you are being unreasonable regarding this change (empty lines or not in a yaml does not affect in any way the current tofu setup, roadmap, or ability to change anythingt) [14:59:26] ok [14:59:38] middle ground, put the empty line cleanup in a different MR [14:59:50] so your automated MR don't change it every time [14:59:54] does that sound good? [15:00:34] sure, weird middle ground [15:25:25] arturo: do you know what's up with the 'cloudinfra-codfw1dev.codfw1dev.wikimedia.cloud.' zone that was created last thursday? that's blocking the actual records in the 'codfw1dev.wikimedia.cloud.' zone from resolving :/ [15:26:13] taavi: oh :-( makes sense [15:26:16] let me delete it [15:30:30] taavi: https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/111 [15:34:02] +1 [15:43:20] taavi: tofu was run, could you please verify if you still see the problem? [15:43:43] let me see [15:43:50] it might still be cached somewhere or something [15:44:10] seems better now [15:44:12] thank you! [15:44:18] great, thanks [16:28:14] https://wmcs-proxy-test.taavivaananen.fi/ 🎉 [16:28:45] w00t! [16:38:02] can I get a review of this email draft? https://etherpad.wikimedia.org/p/replicas-temp-accounts [16:41:21] dhinus: maybe add fist who the email concerns, so people don't need to read much [16:42:23] how would you phrase it? "if you manage one or more tools"? [16:42:29] might also use a `[N]` kind of replacement for the task (as it's used for the other links) [16:42:42] let me try to write in the etherpad [16:44:48] not sure, I'm not a great writer, something like [16:44:56] "If your tool does not read user information for anonymous users from wikireplicas, feel free to ignore this email, if it does, keep reading." comes to mind [16:44:56] yeah I think it's not bad [16:45:25] maybe wikireplicas is too specific and some people will not understand, not sure [16:45:42] "replica databases" or similar? [16:46:23] adding something like '[toolforge,wikireplicas] ...` in the subject might help too [16:46:40] Wiki Replicas is the official name of the product that all of our documentation uses, let's be consistent please [16:46:57] there's already the automatic [cloud-announce] tag I don't like having too many tags :) [16:47:31] +1 to consistent naming [16:48:03] if I were a user, I'd appreciate having 'Wiki Replicas' (or whatever) in the subject so I don't even need to open the email if I don't use them [16:48:11] just my 2c though [16:58:07] I think having it in the first line is a good compromise :) I don't feel like this is really a wikireplicas change, it might affect tool logins or other things [16:58:28] I will wait tomorrow before sending, because I don't think this is live yet, but it's apparently scheduled for today [17:00:36] * arturo offline [17:39:13] dhinus: sounds good, if it not only affects replicas yep, makes total sense [17:39:16] * dcaro off