[04:35:36] [1/2] CA's wiki has the second most visits last week after more than 4 months of inactivity. [04:35:37] [2/2] https://cdn.discordapp.com/attachments/615786602454581249/1516662575784398968/image.png?ex=6a337598&is=6a322418&hm=fb3593feeb39e7c676619578bc7730ae63378e22047898c11662a9c12a37583c& [05:37:06] We had students writing things with AI and lecturers marking with AI [05:37:58] what's the dark angel wiki? [05:39:17] Wiki about a TV show. It's also the only local wiki that CA has worked on extensively. [05:39:41] I can't seem to find it on Miraheze [05:39:44] only the Fandom version [06:07:32] It's https://darkangel.wikitide.org/ [06:34:42] oh [06:34:43] cool [06:34:51] thanks [06:42:10] ah we love bots [14:38:44] Question, anyone here have experience with running LLMs on a self hosted server? [14:44:13] i ran qwen2.5 coder 14b on my home pc before and it sucked so bad [14:59:04] Did you use gpu acceleration and did your gpu have enough vram? [15:01:29] for that model it was enough, im using a 7800XT so by no means ideal for running an LLM [15:02:06] Oh cool [15:02:23] i wouldnt say its worth the effort if you dont have the GPUs required for a good model [15:02:36] unless you want to do something small like spell check or auto complete [15:04:48] Ah right. Well my current plan is to get my own server with adequate GPUs to self host LLMs (maybe agents when I learn enough about them) so that I can get rid of ChatGPT from my workflows for idea expansion, image generation and reviewing processes. [15:05:26] I don't let it write articles for me as that got very boring very fast and took a lot of fun out of writing the world's content. [15:06:53] I should also mention that images generated are then either recreated by me in Krita or are used as references and recreated by my friends who specialise in digital art or 3D modelling and rendering and I pay them for their services. [15:07:03] I haven't done enough with it to be a useful asset in actual troubleshooting or recommendations I would say [15:07:38] but I know it varies a lot not only on your gpu resources but models, weighs, quantization, etc, something wrong in any of those categories will screw you [15:08:02] qwen 2.5 can express very differently by the particulars of which variation it is and how it is being applied in weights [15:08:28] my fun task is making it work on 2gb of vram despite an abundance of system ram [15:08:44] and stumbling into something that is coherent for an actual workflow [15:10:25] Yeah I'm currently researching that. I think my best option would be https://github.com/open-webui/open-webui for the UI, Ollama for the models and either one or two NVIDIA RTX 3090s (used/refurbished) for the server. I'm looking at electricity providers and trying to select the one with the most renewable energies as well. Basically I want this to be as ethical as possible [15:11:17] I know theres a lot of things I don't yet understand though [15:11:40] So I've got a Trilium Notes instance with all of the info I've got so far for planning [15:16:27] open webui is pretty neat as they go [15:17:00] there are other models with a more ethical foundation but you may find them less versatile, makes sense when they didn't have the luxury of stealing the entire internet I suppose [15:17:13] Yeah true [15:24:46] not even just the internet but I digress [15:42:49] Have you ever heard of or used PROXMOX before? [15:43:05] I might use that too. [15:46:34] Heard of it, looked into it, but never used it [15:56:12] My logic is that the server could do other things, not just LLMs and related stuff... PROXMOX is like a Virtua Machine manager and container platform [17:32:44] You could do anything server-related with Proxmox [17:36:34] Definitely using it then lol [19:42:35] Miraheze uses it for its servers [19:42:56] Depending on your usecase may be a bit overkill compared to say containers via docker or podman [20:12:11] Oh nice [20:13:38] Maybe at first but I want to learn it [20:13:57] Fair! Totally respect that, learning is awesome and rocks [20:16:06] Yeah I want to be able to put it on my CV as a skill, just like Docker so I'll probably end up using both [20:19:57] Same with Opnsense although thats a whole different beast [20:22:25] Don’t know if I e heard of that one [20:42:56] Its a fork of PFsense [20:43:27] https://opnsense.org/ [20:43:53] opnsense is epic [20:44:45] if you mainly want to use for experience go for it, but podmon is probably the middle I'd recommend for production use if you get tired of tinkering with the environment [20:45:22] Ok cool, Ive added that to my notes [20:46:13] openwrt is also cool [20:46:26] I kind of suck at them but I've used both as cores of my home networking [20:47:22] Ive heard of openwrt but never used it [20:47:45] Don't you have to flash it to a router or something? [21:16:35] pretty much its gist yeah, I've put it on a number of otherwise ancient worthless routers [21:17:29] still a chance the hardware itself is not entirely trustworthy but at least at that point on the software end you're giving an old router something to live for on software with updates within the decade [21:22:26] [1/2] Alright I;ve done some writing and this is now my main wiki page for my project [21:22:26] [2/2] https://cdn.discordapp.com/attachments/615786602454581249/1516915951038365868/image.png?ex=6a346191&is=6a331011&hm=852a0927ac0b459ea8f3fa12a9f3583fa8be63f4ce1493b821e1262952a82a6c& [22:04:21] its interesting material, feel free to publish findings at your convenience [22:26:37] Thanks. I think I'll upload it to https://grimtown.miraheze.org/wiki/project-atlas/ eventually when it's all complete but for now, it's going to stay on my Trilium Notes instance# [22:28:20] You can find Trilium Notes at https://triliumnotes.org/ which is an awesome note and knowledge base building application which can also be self hosted through Docker. It's like Obsidian Notes but IMO better as you don't have to deal with markdown and can just add images, do formatting, link to other notes, put notes into folders and so on. [22:29:55] its neat software, I have no trouble with markdown though [22:30:08] its a modest upgrade over having used raw text since forever and wordpad for a little while [22:30:44] Yeah, I also use LibreOffice Writer for docx files and PDFs for projects which will be released at some point [22:42:58] I like looking at https://meta.miraheze.org/wiki/Tech:Server_usage . That's quite interesting